How I tore every cloud dependency out of my AI companion and rebuilt it on a fully local stack of faster-whisper, Ollama, ChatterBox, and ChromaDB.
When I started O.L.I.V.I.A. in January, the first prototype did what every AI companion prototype does: it called OpenAI. That was the fastest path to a working conversation loop, and for early R&D it was the right call. It was also a contradiction I knew I would have to resolve. An AI companion is a system you talk to about your life, continuously, over months. It accumulates the most intimate context of any software you run. Routing that through a third-party API meant the most personal system I had ever built was also the leakiest.
So I rebuilt it. Every cloud dependency in the pipeline is gone. Here is what the architecture looks like now, and what I learned tearing out the old one.
The system runs as a FastAPI backend orchestrating four local components. Speech comes in through faster-whisper for transcription. The brain is a fine-tuned 8B model served through Ollama on my own GPU. Responses go out through ChatterBox Turbo for text-to-speech. And memory lives in ChromaDB, structured as a tiered system rather than one flat vector store.
The memory design is the part I am most attached to. Short-term context flows into longer-term storage, and a consolidation process I call dreaming periodically reorganizes what has accumulated, distilling raw conversation into more durable memories. The name is a little whimsical, but the mechanism is practical: without consolidation, a companion's memory becomes an ever-growing pile of transcripts, and retrieval quality degrades as the pile grows. Dreaming keeps the memory dense instead of just large.
The hardest engineering lesson of the rebuild had nothing to do with infrastructure. It was about how to get a consistent persona out of an open model.
My first instinct was model merging. The ecosystem is full of TIES-family merges promising the best of multiple fine-tunes, and on paper that sounds like exactly what a companion needs: personality from one parent, capability from another. In practice, merging produced a model that was inconsistent in precisely the ways a persona cannot afford. Tool-calling reliability suffered, and the voice drifted between the merged parents depending on context. A companion that sounds like a different person every few turns is not a companion.
What worked was less clever and more disciplined: QLoRA supervised fine-tuning on a clean base model, followed by DPO to sharpen preferences. One base, one training pipeline, one consistent voice, and tool calls that behave the same way every time. The takeaway I would offer anyone building a persona-driven system is that merging optimizes for benchmark variety while fine-tuning optimizes for behavioral coherence, and a companion lives or dies on coherence.
Running everything locally costs real money in hardware and real time in engineering. The reason it is worth it is the same reason Voice Journal ships with no INTERNET permission: a privacy promise you enforce with architecture is categorically different from one you make in a policy.
O.L.I.V.I.A.'s memory of every conversation exists on one machine that I control. No provider can train on it, no breach at a company I have never heard of can expose it, and no terms-of-service update can change the deal. For a system whose entire purpose is accumulating personal context, I no longer think that is a nice-to-have. I think it is the only defensible design.
The repository is public, the architecture is locked, and development is active. There is a lot still ahead, but the foundation is finally the one this project deserved from the start.
Enjoyed this one? Say hi and let’s talk shop.
Get in Touch