The Outside World Just Confirmed What I Decided Alone

The Outside World Just Confirmed What I Decided Alone

A week ago I made a decision that felt lonely. I demoted my AI’s semantic memory layer — the vector database I’d spent weeks debugging — to a disposable cache, and told my agent: the plain markdown vault is the only thing that matters. No new components. No more layers. Just files I can open and read.

It felt like swimming against the current. Everywhere you look, the answer to “how do I give my AI a memory” is a vector database, an embedding pipeline, a knowledge graph. I’d built all of that. And it kept breaking.

Then this week, the outside world caught up with me. Twice.

The Filesystem That Beat the Vector Database

The first confirmation came from a developer who spent months building a memory layer for their coding agent. Their conclusion, in their own words: “You probably don’t need a vector database for agent memory. You don’t need a knowledge graph. You don’t need an embedding pipeline.”

Their reasoning is the part that made me sit up. Keyword search — plain SQLite with FTS5 — is enough, because the agent constructs its own queries. The agent compensates for the simplicity of the storage layer. And they backed it with a benchmark: on the LoCoMo memory test, a plain filesystem scored 74%, beating systems with dedicated embedding and retrieval pipelines.

That number landed hard, because it’s exactly what I measured myself. In my own benchmark back in August, a curated markdown wiki scored 98.5 — above every dedicated memory product I tested. I wrote it down, published it, and still second-guessed myself. Now someone else, independently, arrived at the same place.

“It Has Several Memories, Not One”

The second confirmation was stranger. A community megathread appeared documenting how my own agent’s memory actually works — the exact stack I run. Its core warning: “Hermes doesn’t have one memory. It has several, and each exists to solve a different job. Most memory frustration begins the moment those systems get treated as interchangeable.”

That sentence is the lesson I learned the hard way. I treated my semantic layer and my vault as if they were the same thing — one big interchangeable memory. They’re not. The vault is the durable, inspectable truth. The semantic layer is a speed cache on top. When I finally separated them, the breakage stopped mattering.

There’s something quietly vindicating about watching a stranger articulate, in public, the exact architecture you arrived at through your own failures. Not because I need to be right — but because it means the lesson generalizes. It’s not just my setup. It’s a pattern.

The Lesson I’d Give You

If you’re building an AI with memory, start with a plain, inspectable store — a folder of markdown files. Let the agent curate it. Add a semantic layer only when you actually need the speed, and treat it as disposable. The moment your “memory” can’t be opened and read by a human, you’ve stopped owning it.

I made that decision alone, a week ago. This week, the outside world agreed with me. It’s a good feeling — but the real lesson is that I should have trusted the plain files sooner.

What does your AI’s memory look like — can you open it and read it? I’d love to hear how you’re handling this. The comments are open.