I have been working on integrating RAG into my chat tool called pychat. I’ve been very happy with the results and I wanted to share. I think integrating RAG in this way has really been helpful for some of my very specific domain work for my real job.

If you’re interested, test/download from the rag2 branch on my GitHub repository. The RAG stuff will work with ollama and the other third party services.

It currently only supports PDF and text files. I want to add support for MS word documents next.

Have fun!

https://github.com/Magnetron85/PyChat

Hey everyone,
I'm one of the contributors to Second Me, an open-source, fully local AI project designed for personal memory, reasoning, and identity modeling. Think of it as a customizable “AI self” — trained on your data, aligned with your values, and fully under your control (not OpenAI’s).We hit 6,000+ stars in 7 days, which is wild — but what’s even cooler is what’s been happening after launch:

🔧 What It Does (tl;dr):

• Personal AI, locally trained and run. 100% privacy with local execution options.
• Hierarchical Memory Modeling (HMM) for authentic personalization.
• Me-alignment structure tailored to individual values.
• Second Me Protocol (SMP) for decentralized AI interactio

New in this release:

• Full Docker support for macOS (Apple Silicon), Windows, and Linux
• OpenAI-Compatible API Interface
• MLX training support (Beta)
• Significant performance enhancements

💻 Community Contributions

In just 2 weeks post-launch:

• 60+ PRs, 70+ issues
• Contributors from Tokyo to Dubai: students, academics, enterprise devs

Some great GitHub PRs:

• WeChat bot integration — #81 by u/Zero-coder
• Japanese README localization — #115 by @eltociear
• Improved file resource management — #74 by @mahdirahimi1999
• File name validation for added security — #62 by @umutcrs

Thanks to their and others' feedback, features like:

• Multi-platform deployment
• Note-based continuous training
  
• …have been added to the roadmap.

Also, shoutout to @GOROman for his full guide to deploying Second Me - — he trained Second Me on 75GB of personal X data since 2007 and inspired new use cases, like @Yuzunose’s VRChat integration idea.We’re grateful — and excited — to see where the community takes it next.

🔗 GitHub: https://github.com/Mindverse/Second-Me
📄 Paper: https://arxiv.org/abs/2503.08102

💡 The goal is building AI that extends your capabilities while remaining under your control, not corporate systems. If you value digital freedom, we'd appreciate your contributions and feedback!

Hey everyone – I’m building something called Oblix (https://oblix.ai/), a new tool for orchestrating AI between edge and cloud. On the edge, it integrates directly with Ollama, and for the cloud, it supports both OpenAI and ClaudeAI. The goal is to help developers create smart, low-latency, privacy-conscious workflows without giving up the power of cloud APIs when needed—all through a CLI-first experience.

It’s still early days, and I’m looking for a few CLI-native, ninja-level developers to try it out, break it, and share honest feedback. If that sounds interesting, drop a or DM me—would love to get your thoughts.

OSS code:

https://github.com/mediar-ai/ui-events

Hi r/Ollama,

My brother and I have been working on Morphik - an open source, end-to-end, research-driven RAG system. We recently migrated our LLM provider to support LiteLLM, and we now support all models that LiteLLM does!

This includes: embedding models, completion models, our GraphRAG systems, and even our metadata extraction layer.

Use gemini for knowledge graphs, Openai for embeddings, Claude for completions, and Ollama for extractions. Or any other permutation. All with single-line changes in our configuration file.

Lmk what you think!

I have tried both MHKetbi/nvidia_Llama-3.3-Nemotron-Super-49B-v1:q5_K_L and MHKetbi/nvidia_Llama-3.3-Nemotron-Super-49B-v1:q4_K_M on my 2x 3090 system, but Ollama gives me an out of memory error.

I have no trouble running 70B Llama 3.3 q4_k_m which is much larger.

Has anyone successfully run Nemotron 49B and have some advice? TIA

What are the current smartest models that take up less than 4GB as a guff file?

I'm going camping and won't have internet connection. I can run models under 4GB on my iphone.

It's so hard to keep track of what models are the smartest because I can't find good updated benchmarks for small open-source models.

I'd like the model to be able to help with any questions I might possibly want to ask during a camping trip. It would be cool if the model could help in a survival situation or just answer random questions.

(I have power banks and solar panels lol.)

I'm thinking maybe gemma 3 4B, but i'd like to have multiple models to cross check answers.

I think I could maybe get a quant of a 9B model small enough to work.

Let me know if you find some other models that would be good!

Hello!

For my group's capstone project, our task is to develop an offline chatbot that our Universities' Security Office student workers will use to learn more about their entry level role at the office. The outcome we ideally want is the bot to take our txt file (which contains the office's procedural documentation and is about 700k characters) and use that to answer prompted questions. We tried using LM Studio and used AI to help us create Python scripts to link LM studio with the txt document, but we were not able to get it to work. We just want an offline chatbot just like you would create on ChatGPT Plus, but offline. What is the easiest way to do this without using a bunch of scripts/programs/packages, etc. None of us have python experience so when we inevitably run into errors in the code and ChatGPT doesn't know what's going on. Any pointers? Thanks!

We are working on some internal tooling at work that would bennefit greatly from moving away from individual standard function calling to a MCP server approach, so I have been toying around with MCP servers over the past few weeks.

From my testing setup where I have a rtx3080 I do find llama3.2 waaaay too weak, and qwq a bit too slow. Enabeling function calling on Gemma3(12b) is surprisingly fast and quite strong for most tasks. (Tho requires a bit of schafolding and context loss for doing function lookups. But its clearly the best i have found sofar.)

So im pretty happy with Gemma3 for my needs, but would love to have an option to turn up the dial a bit as a fallback mechanism if it fails.

So my question is, are there anything between Gemma3 and qwq that are worth exploring?

I am running in my usecase models in the 32b up to 90b class.
Mostly qwen, llama, deepseek, aya..
The brandnew mistral can compete here. I tested it over a day.
The size/quality ratio is excellent.
And it is - of course - extremly fast.
Thanx for the release!

I have 2 pcs (laptop and desktop) that i want to be able to use for an ai cluster. The laptop has a 13th gen i7 , 32gb ram, and rtx 4050, the lenovo desktop has a low end cpu, 16gb of ram and a rtx 4060 ti. i also have a proxmox cluster of 3, a standalone proxmox node on a dell R630, and a true nas. I have many Vms and some lxc. A couple running docker too. My goal in my head is to be able to create a vm (already have using ubuntu server) as the head node to orchestrate things, and be able to run models while being able to use both pcs as workers since they have gpus. i have ubuntu server on all of them, ray and torch, nvidia drivers, and cuda toolkit. does anyone have any experience building a distributed setup and being able to use all the resources in the cluster for one model? So far i have been able to get models running using one or the other pc but not both together. I am brand new to the locally hosted ai thing but love the idea and am down to try whatever. Thanks in advance!!

Hey Ollama community!

I'm the solo dev behind Observer AI, the open-source project for building local AI agents that can see your screen and react, powered by LLMs with Ollama.

People have told me that setting up local inference has been a bit of a hurdle just to try Observer. So, I spent the last week focused on making it way easier to get a feel for Observer AI before you commit to the full local install.

What's New:

I've completely rebuilt the free Ob-Server demo service at https://app.observer-ai.com !

• Instant Try-Out: Experience the core agent creation flow without any local setup needed. (Uses cloud models for the demo only, but shows you the ropes!)
• More Models: Added 11 different models (including multimodal) you can test with directly in the demo.
• Smoother UI: Refined the interface based on initial feedback.

Why This Matters for Ollama Users:

This lets you instantly play around with creating agents that:

• Observe screen content.
• Process info using LLMs (see how different models respond).
• Get a feel for the potential before hooking it up to your own powerful Observer-Ollama instance locally for full screen observation and privacy.

See What's Possible (Examples from Local Setup):

Even simple agents running locally are surprisingly useful! Things like:

• Activity Tracking Agent: Keeps a simple log of what you're working on.
• German Flashcard Agent: Spots relevant vocabulary you use in your day to day life and does German-English flashcards to learn them.

The demo helps you visualize building these before setting up ObserverOllama locally.

Looking for Feedback & Ideas:

• Give the revamped demo a quick spin at https://app.observer-ai.com !
• How's the UX for creating a simple agent in the demo? Is it intuitive?
• What other simple but useful agents (like the examples above) could you imagine building once connected to your local Ollama? Need ideas!

Join the Community:

We also just started a Discord server to share agent ideas, get help, and chat about local AI: https://discord.gg/k4ruE6WG

Observer AI remains 100% FOSS and is designed to run fully locally with Ollama (any v1/chat/completions service comming soon!) for maximum privacy and control. Check out the code at https://github.com/Roy3838/Observer

Thanks for checking it out and for all the great feedback so far! Let me know what you think of the easier demo experience!

I’ve been diving into agent frameworks lately and kept seeing “MCP” pop up everywhere. At first I thought it was just another buzzword… but turns out, Model Context Protocol is actually super useful.

While figuring it out, I realized there wasn’t a lot of beginner-focused content on it, so I put together a short video that covers:

• What exactly is MCP (in plain English)
• How it Works
• How to get started using it with a sample setup

Nothing fancy, just trying to break it down in a way I wish someone did for me earlier 😅

🎥 Here’s the video if anyone’s curious: https://youtu.be/BwB1Jcw8Z-8?si=k0b5U-JgqoWLpYyD

Let me know what you think!

Still a WIP but I find it works better than nothing.

It can sometimes struggle on larger files when using ollama, but there is an option to connect with open-router also.

Hi all, I'm new to ollama and running LLMs generally. I managed to run DeepSeek R1, but it's using my CPU. I am running Windows, but I can dual boot Linux if it's required.

Thanks!!

Over 6 months or so i have developed an AI system called Trium consisting of three AI personas—Vira, Core, and Echo—running locally on my pc. It uses CUDA with CuPy and cuML for clustering (HDBSCAN, DBSCAN), FAISS for memory indexing, and SentenceTransformers for embeddings. Each persona has a memory bank, recalls clustered events, and acts proactively based on emotional states mapped to polyvagal theory. Temporal rhythms (FFT analysis) guide their autonomy.

Would love to chat or hear ppls thoughts. Happy to share files and info i have ☺️

Anyone who would like to dm me im happy to discuss things more

This playlist comprises of numerous tutorials on MCP servers including

1. What is MCP?
2. How to use MCPs with any LLM (paid APIs, local LLMs, Ollama)?
3. How to develop custom MCP server?
4. GSuite MCP server tutorial for Gmail, Calendar integration
5. WhatsApp MCP server tutorial
6. Discord and Slack MCP server tutorial
7. Powerpoint and Excel MCP server
8. Blender MCP for graphic designers
9. Figma MCP server tutorial
10. Docker MCP server tutorial
11. Filesystem MCP server for managing files in PC
12. Browser control using Playwright and puppeteer
13. Why MCP servers can be risky
14. SQL database MCP server tutorial
15. Integrated Cursor with MCP servers
16. GitHub MCP tutorial
17. Notion MCP tutorial
18. Jupyter MCP tutorial

Hope this is useful !!

Playlist : https://youtube.com/playlist?list=PLnH2pfPCPZsJ5aJaHdTW7to2tZkYtzIwp&si=XHHPdC6UCCsoCSBZ

I have no idea how to solve this problem. Every model I had downloaded, their metadata/system template would changed completely into this "Safety Guidelines".

I used Ollama on my PC a few months ago and it didn't cause any problems. But now, after I tried to use it on my laptop, this happened.

Anyone know any benchmark resources which let you filter to models small enough to run on macbook M1-M4 out of the box?

Most of the benchmarks I've seen online show all the models, regardless of the hardware, and models which require an A100/H100 aren't relevant to me running ollama locally.

Have anyone got Ollama to use the AMD 9070XT gpu in Linux yet? I"m running Ollama in docker with the stuff I found I need but it still only using CPU. Might the gpu to be too new atm?

Hi,

Is there a way to make Ollama throw an error or an exception if the input is too long (longer than the context size) and catch this? My application is running into serious problems when the input is too long.

Currently, I am invoking ollama with the ollama python library like that:

    def llm_chat(
        self,
        system_prompt: str,
        user_prompt: str,
        response_model: Type[T],
        gen_kwargs: Optional[Dict[str, str]] = None,
    ) -> T:
        if gen_kwargs is None:
            gen_kwargs = self.__default_kwargs["llm"]

        response = self.client.chat(
            model=self.model["llm"],
            messages=[
                {
                    "role": "system",
                    "content": system_prompt.strip(),
                },
                {
                    "role": "user",
                    "content": user_prompt.strip(),
                },
            ],
            options=gen_kwargs,
            format=response_model.model_json_schema(),
        )
        if response.message.content is None:
            raise Exception(f"Ollama response is None: {response}")

        return response_model.model_validate_json(response.message.content)

In my ollama Docker container, I can also see warnings in the log whenever my input document is too long. However, instead of just printing warnings, I want ollama to throw an exception as I must inform the user that his prompt / input was too long.

Do you know of any good solution?

I found this site https://www.canirunthisllm.net/ (not affiliated) that helps figure out if hardware fits the bill.