The Problem: Three Separate Memory Systems That Didn’t Talk to Each Other
A few weeks ago, I realized my AI agent had a serious blind spot. Every time I asked a question that touched on something we’d discussed before, the agent would either find the answer in its short-term memory — or miss it entirely. There was no middle ground. No “I think I’ve seen this before, let me check.” It was either instant recall or complete amnesia.
The reason? My agent had three separate memory systems that didn’t talk to each other:
- Mem0 — a vector database (Qdrant) that stores facts as mathematical embeddings. Great for finding similar information, but it can’t do keyword search.
- Session DB — a full-text search (FTS5) over every conversation I’ve ever had with the agent. Great for finding exact phrases, but it can’t handle synonyms or related concepts.
- Context Graph — a knowledge graph of entities (people, projects, servers, tools) and their relationships. Great for understanding connections, but it’s built from regex patterns, not meaning.
Each system was powerful on its own. But together? They were three islands with no bridge. And when the agent needed to answer a complex question, it would pick one at random — and often pick the wrong one.
The Solution: Hybrid Recall
So I built something I call Hybrid Recall — a unified memory search that queries all three systems in parallel and synthesizes the results into one coherent answer.
The idea is simple: instead of asking the agent to guess which memory system to use, I run all three searches simultaneously and let the results speak for themselves.
- Mem0 finds semantically similar facts (cosine similarity in Qdrant, top-5 results)
- Session DB finds exact keyword matches in past conversations (FTS5, top-3 sessions)
- Context Graph finds known entities and relationships (SQLite, top-10 nodes)
Then I combine everything into one structured answer: “Here’s what Mem0 found, here’s what past sessions say, and here’s how everything connects.”
What I Learned
The biggest surprise? Each system catches things the others miss. Mem0 finds the semantic connection that no keyword would ever match. Session DB finds the exact phrase that the vector search blurred. The context graph finds the relationship that neither text search would spot.
For example, when I asked “What do we know about the genealogy app?”, Hybrid Recall returned:
- Mem0: “Flask+SQLite+Topola, 915 persons, port 5000”
- Session DB: “Sprint 3 discussions about media viewer, merge logic, vis-network family tree”
- Context Graph: “genealogie-app project → contains app.py, runs on openclaw:5000”
No single system would have given me the full picture. Together, they painted a complete story.
The Takeaway
If you’re building an AI agent — or even just thinking about how to organize your own digital knowledge — don’t put everything in one basket. Different types of information need different types of storage. The magic happens when you build bridges between them.
My agent now has a search hierarchy: memory → vector search → vault → session history → context graph → its own knowledge. Each layer catches what the previous one missed. And the result is an agent that actually remembers — not just retrieves.
Have you ever run into the “three islands” problem with your own tools? I’d love to hear how you solved it.
Wat vond je van dit bericht?
