My AI Agent Has 6 Layers of Memory — Here’s What I Learned Building It

My AI Agent Has 6 Layers of Memory — Here’s What I Learned Building It

When I first set up Hermes Agent, I thought memory was simple: the agent remembers what I tell it, or it doesn’t. Six months later, I’ve built a 6-layer memory system that compounds knowledge every single night — and the difference is night and day. Let me walk you through what each layer does and why you might want something similar for your own AI setup.

Layer 1: The Working Memory (Always There)

This is the smallest layer — about 2KB of text that gets injected into every single conversation. It holds the essentials: my server IPs, file paths, tool quirks, and conventions. Think of it as the agent’s sticky note that’s always visible. I update it maybe once a week when something changes. It’s small by design — you don’t want to waste tokens on information the agent already has in its training data.

Layer 2: Vector Search (Semantic Memory)

This is where it gets interesting. Every fact I add gets converted into a mathematical vector (768 dimensions, using nomic-embed-text) and stored in a Qdrant database. When the agent needs to find something, it doesn’t search by keyword — it searches by meaning. So if I ask “what was that thing about server costs?” it finds the relevant fact even if I never used the words “server” or “costs” when I stored it. This runs locally on my machine, costs nothing per query, and takes about 2 seconds to search through everything.

Layer 3: Full Conversation History (Session DB)

Every conversation I’ve ever had with my agent is stored in a SQLite database with full-text search — about 1.6GB of history at this point. I can search through months of conversations in milliseconds. The trick? I use FTS5 (SQLite’s full-text search engine), which is keyword-based. For semantic search I combine it with Layer 2. Together they cover everything: exact matches and meaning-based matches.

Layer 4: The Knowledge Graph (Who Knows Who)

This one runs every night at 22:30. A script scans all my conversations and extracts entities — people, projects, servers, tools, errors — and the relationships between them. It builds a graph: “John works on genealogie-app, which runs on openclaw:5000, which uses Flask and SQLite.” The graph is regex-based (not AI-powered), so it’s fast and cheap. But it’s surprisingly effective at surfacing connections I didn’t explicitly ask about.

Layer 5: The Obsidian Vault (Long-Term Knowledge)

This is my second brain — 71+ wiki pages, CRM entries, system documentation, and daily journals. Every article I read, every project I work on, every lesson I learn gets documented here. The agent can search it via the Seekstone MCP tool. What makes this powerful is the compounding effect: every new piece of knowledge connects to existing knowledge, and over time the vault becomes more valuable than the sum of its parts.

Layer 6: Skills (Procedural Memory)

This is how the agent knows how to do things — not just what things are. About 50 skills covering everything from WordPress publishing to vault maintenance to image generation. Each skill is a markdown file with YAML frontmatter, loaded on demand. The system even tracks which skills are used most and archives stale ones automatically. It’s like a toolbox that organizes itself.

---

The Compounding Effect

The real magic isn’t any single layer — it’s how they work together. Every night at 23:30, a batch process runs that syncs knowledge between layers: new conversations get analyzed, entities get extracted, the graph gets updated, and the vault gets populated. By morning, the agent knows more than it did the day before. After a month, the difference is dramatic. The agent starts suggesting patterns you didn’t see, connecting dots you forgot about, and answering questions before you finish asking them.

I documented the full architecture in my vault — 365 lines of notes, 6 layers, 8 cron jobs, and a nightly cycle that runs like clockwork. It took months to get right, but now it runs itself.

What does your AI memory setup look like? I’m curious how others handle this — drop a comment below!