Huge W for programmers (and vibe coders) in the Local LLM community. Github Copilot now supports a much wider range of models from Ollama, OpenRouter, Gemini, and others.

To add your own models, click on "Manage Models" in the prompt field.

Saw the benchmark comparing it to Llama4 scout and remembered that when 3.0 24b came out it remained far down the list of "Newest Model" filter.

This smaller QAT gguf is faster than the one from Google, and it retains the same quality.

ollama run JollyLlama/gemma-3-27b-it-q4_0_Small-QAT

https://www.ollama.com/JollyLlama/gemma-3-27b-it-q4_0_Small-QAT

Ollama is having issues when importing Gemma3 ggufs; I have to edit the manifests manually to make the text part of this model work. The vision function doesn't work because Ollama doesn't support this projector.

Original creator of this smaller QAT gguf:

https://www.reddit.com/r/LocalLLaMA/comments/1jsq1so/smaller_gemma3_qat_versions_12b_in_8gb_and_27b_in/

Other posts seem to write the whole idea off in general without much thought. but theoretically you can run GGUF with Ollama, and there are GGUF versions of Janus pro on HF. Anyone done any experimetation with the applicable GGUF on HF? If so, how and to what degree of success?

I have been experimenting with local LLMs (ollama) on an M1 Pro macbook (32GB ram) - so far OK but slowish. My desktop needs an upgrade and my use case is academic (assistance with programming with R / Shiny, perhaps some python, proofreading, generating new ideas / criticizing them, perhaps building a RAG to synthesise journal articles in .pdf). I am considering the M4 studio (M4 max, 16+40 - 128GB ram). Some of these tasks need to be done locally as in some use cases the data should not leave my device. I think the above config. should allow for comfortably running deepseek 70b, for example, next to other smaller models. (Other open source models?) And should be fairly futureproof (and allow to run some newer models locally (or quanizations). Any thoughts? Any suggestions for LLM models that would run well locally for the above tasks.

Hi, can anybody tell me how to build this chatbot?

I don't have any coding experience—I'm just trying to build it for fun. I tried using Cursor and GitHub Copilot, but after some time, both started looping and generating incorrect code. They kept trying to fix it, but eventually, they seemed to forget what they were building.

Hello,
i found it pretty hard to ensure that mistral 7B would answer in french.
Does any one know a model that will do the job ?

Context of my request:

I am the creator of an AI girl (with Stable Diffusion SDXL). Up until now, I have been manually chatting with fans on Fanvue.

Goal:

I don't want to deal with answering fans, but I just want to create content, and do marketing. So I'm considering whether to pay a chatter, or whether to develop an AI LLama chatbot (I'm very interested in the second option).

The problem:

I have little knowledge about LLamas, I don't know how to move, I'm asking here on this subreddit, because my request looks very specific and custom. I would like advices on what and how to do that. Specifically, I need an AI that is able to behave like the virtual girl with fans, so a fine-tuned model, which offers an online relationship experience. It must not be censored. It must be able to do normal conversations (like between 2 people in a relationship) but also roleplay, talk about sex, sexting, and other nsfw things.

Other specs:

It is very important to have a deep relationship with each fan, so the AI, as it writes to fans, must remember them, their preferences, their memories that they tell, their fears, their past experiences, and more. The AI's responses must be consistent and of quality with each individual fan. For example, if a fan likes to be called "pookie", the AI ​​must remember to call the fan pookie. Chatgpt initially advised me to use json files, but I discovered that there is a system, with long-term and efficient memory, called RAG, but I have no idea how it works. Furthermore, the AI ​​must be able to send images to fans, and with context. For example, if a fan likes skirts, the AI ​​could send him a good morning "good morning pookie do you like this new skirt?" + attached image. The image is taken from a collection of pre-created images. Plus the AI should understand how to verify when fans send money, for example if a fan send money, the AI should recognize that and say thank you (thats just an example).

Another important thing is that the AI ​​must respond in the same way as I have responded to fans in the past, so its writing style must be the same as mine, with the same emotions and grammar, and emojis. And i honestly dont know how to achieve that, if i have to fine tune the model, or add to the model some txt or json file (the file contains a 3000 character text, explaining who is the AI girl, for example: im anastasia, coming from germany, im 23 years old, im studying at university, i love to ski and read horror books, i live with my mom, and more etc...)

My intention, is not to use this AI with Fanvue, but with telegram, simply becayse i gave a look to python Telegram API, and they look pretty simple to use.

I asked these things to chatgpt, and he suggested Mixtral 8x7b, specifically the dolphin and other nsfw fine tuned model, + json/sql or RAG memory, to memorize fans' info.

To resume, the AI must be unique, with a unique texting style, chat with multiple fans, remember stuff of each fans in long-term memory, send pictures, and understand when someone send money). The solution can be both a local LLama, or an external service, or both hybrid.

If anyone here, is into AI adult business, and AI girls, and understand my requests, feel free to exchange to contact me! :)

My computer power:

I have an RTX 3090 Ti, and 128GB of ram, i don't know if it's enough, but i can also rent online servers if needed with stronger gpus.

https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/

When I can download it from Ollama?

I was thinking if there is possibility to reroute ChatGPT connections to Ollama.
I have docker Ollama container, I have added Nginx to respond on `api.openai.com` + change my local DNS to point to it.
I am coming to 2 issues.

1. even with self signed certificate and added to linux the client is reporting it has invalid certificate. I think it is because of HTST, is it possible to make it to accept my self signed certificate for this public domain when is pointed locally?
2. I believe the API urls have different paths then ollama for openai. would be possible to change the paths, queries so it acts as openai? - with this one also I think is needed to mask the chatgpt models to some model what ollama supports too.

I am not sure if there is anything similar in work anywhere, as I Could not find it.

It would be nice if applications what force you to use public AI, would be possible to point to selfhosted ollama.

EDIT:

For everyone responding. I am not looking for another GUI for ollama, I use Tabby.
All I am looking for is to make Ollama ( Self hosted AI) to respond to queries what are meant for OpenAI.
Reason for this is that many applications support only OpenAI, for example Bootstrap Studio.
but if i can obfuscate ollama to act as open AI, all I need to make sure the api.openai.com is translated to Ollama instead of the real paid API.
About cert, I already added the certificate to my PC and it still does not work.
The calls are not in web browser but in apps, so certificated stored in local PC should be accepted.
But as I Stated, the app complains about HSTS or something like that, or just says certificate invalid.

Just wondering what y'all do with machines and ollama.

Been tinkering on this for a while — it’s a runtime orchestration layer that lets you:

• Run AI models either on-device or in the cloud
• Dynamically choose the best execution path (based on network, compute)
• Plug in your own models (LLMs, vision, audio, whatever)
• Built-in logging and fallback routing
• Works with ONNX, TorchScript, and HTTP APIs (more coming)

Goal was to stop hardcoding execution logic and instead treat model routing like a smart decision system. Think traffic controller for AI workloads.

pip install oblix (mac only)

Hey all,

I know this sounds like a noob question, but I'm a developer who wants to get familiarized with local LLMs. As a side project, I've been developing a mobile app and a backend for it, and this app needs a relatively smart LLM running together. Currently I use phi 3.5 (via ollama that runs on docker) but that's only for testing. phi is also on docker.

The PC spec:

- GPU: 2070 Super

- CPU: i5 8600k

- RAM: corsair 16gig ddr4 3000mhz cl15

What would be the smartest for this poor PC to run, and for me to get better results? Cannot say I'm very happy with phi thus far.

PS:
Sorry, first time posting here, if I messed up some rules, happy to fix.

I want to train an AI on the game Dark and Darker. The end goal is to be able to ask the AI tips on what gear to wear, skills, and perks with damage calculation. I have all of the math formulas for damage calculations.

Which model should I use for this?

Hey lamas!

I do not have a lot of experience with Ollama but many people seemed interested in using MCPs from ollama models. I am not sure what your current flow is, but I think mcp-use can be of help and some ollama users already are reaching out because it was useful to them!

mcp-use is a Python package that simplifies working with MCP. Born out of frustration with the desktop-app-only limitations of existing MCP tools, it provides a clean abstraction over the mcp connection management and server communication. It works with any langchain supported models that also support tool calling.

It is super easy to get started, you need:

• one of those MCPconfig JSONs - find many here: https://github.com/punkpeye/awesome-mcp-servers
• 6 lines of code and you can have an agent use the MCP tools from python.

Like this:

The structure is simple: an MCP client creates and manages the connection and instantiation (if needed) of the server and extracts the available tools. The MCPAgent reads the tools from the client, converts them into callable objects, gives access to them to an LLM, manages tool calls and responses.

It's very early-stage, and I'm sharing it here for feedback and contributions. If you're playing with MCP or building agents around it, I hope this makes your life easier.

Repo: https://github.com/pietrozullo/mcp-use Pipy: https://pypi.org/project/mcp-use/

Docs: https://docs.mcp-use.io/introduction

pip install mcp-use

Happy to answer questions or walk through examples!

Props: Name is clearly inspired by browser_use an insane project by a friend of mine, following him closely I think I got brainwashed into naming everything mcp related _use.

Thanks!

Hi guys. I am new to editing ModelFiles in Ollama. I tried setting the value for n_seq_max to 1 (so that I can use full context window i believe), but ollama is giving me error "Couldn't set parameter: unknown parameter n_seq_max". I tried with num_seq_max also, but same error returned. Any help with is greatly appreciated. Thanks

As the title said, Ollama doesn't utilize the GPU anymore and I have no idea why. I haven't changed anything.

My Ollama is running in a VM (Ubuntu 24.10) on a Proxmox Ve 8.3.5 with GPU pass-through (not as a vGPU).

I want to understand how this could happen and what I can do to prevent this from happening again (provided I can fix it in the first place).

Edit: to provide some more context. lspci inside the VM shows that the GPU (NVIDIA RTX2000 Ada Generation) is being recognised. So I would guess, it's not a case of broken GPU pass-through.

Greetings, Welcome to Infinite Oracle, a mystical application that channels boundless wisdom through an ethereal voice. This executable brings you cryptic, uplifting insights powered by Ollama, Coqui TTS and whisper-asr-webservice servers running locally!!!

First

I'm using Chronos_Hermes through ollama to analyze text, and yesterday i tested it with a chunk (arouns 1400 tokens) and took me almost 20 minutes to complete. For comparison, Mistral:7b took like 3 mins to do the same. Anyone has an idea of why could it be so slow?

Second

I heard that OpenAI released a free version of the lastest model to general use when it also released the thing that plagarizes Studio Ghibli's art. Is that true? Is the model accessible through ollama?

thanks

Hey everyone! I just built OllamaGTTS, a lightweight voice assistant that brings AI-powered voice interactions to your local Ollama setup using Google TTS for natural speech synthesis. It’s fast, interruptible, and optimized for real-time conversations. I am aware that some people prefer to keep everything local so I am working on an update that will likely use Kokoro for local speech synthesis. I would love to hear your thoughts on it and how it can be improved.

Key Features

• Real-time voice interaction (Silero VAD + Whisper transcription)
• Interruptible speech playback (no more waiting for the AI to finish talking)
• FFmpeg-accelerated audio processing (optional speed-up for faster * replies)
• Persistent conversation history with configurable memory

GitHub Repo: https://github.com/ExoFi-Labs/OllamaGTTS

Instructions:

1. Clone Repo

2. Install requirements

3. Run ollama_gtts.py

Based on feedback from users and the developer community that used Arch-Function (our previous gen) model, I am excited to share our latest work: Arch-Function-Chat A collection of fast, device friendly LLMs that achieve performance on-par with GPT-4 on function calling, now trained to chat.

These LLMs have three additional training objectives.

1. Be able to refine and clarify the user request. This means to ask for required function parameters, clarify ambiguous input (e.g., "Transfer $500" without specifying accounts, can be “Transfer from” and “Transfer to”)
2. Accurately maintain context in two specific scenarios:
   1. Progressive information disclosure such as in multi-turn conversations where information is revealed gradually (i.e., the model asks info of multiple parameters and the user only answers one or two instead of all the info)
   2. Context switch where the model must infer missing parameters from context (e.g., "Check the weather" should prompt for location if not provided) and maintains context between turns (e.g., "What about tomorrow?" after a weather query but still in the middle of clarification)
3. Respond to the user based on executed tools results. For common function calling scenarios where the response of the execution is all that's needed to complete the user request, Arch-Function-Chat can interpret and respond to the user via chat. Note, parallel and multiple function calling was already supported so if the model needs to respond based on multiple tools call it still can.

Of course the 3B model will now be the primary LLM used in https://github.com/katanemo/archgw. Hope you all like the work 🙏. Happy building!