The Benchmark I Couldn’t Ignore
Someone ran 8 AI agent memory systems through 2,176 scored tasks and a plain markdown wiki beat every single product on the market. Not a fancy vector database. Not a hosted memory service. A folder of .md files that the agent curates itself. It scored 98.5 out of 100 — above every commercial system tested.
When I read that, I laughed out loud. Because that folder is basically what sits inside my own vault. The same pattern Karpathy sketched in his llm-wiki idea, the same thing I have been running for months: plain markdown, git-versioned, and curated by the agent itself instead of by some clever ingestion pipeline.
I wrote a few weeks ago about how my agent’s memory kept growing without ever really learning. This is the other side of that coin: a hard, external benchmark that says the simple approach isn’t a compromise — it is the ceiling. The test ran 200 factual questions about simulated multi-week relationships plus 72 questions about facts that were never stored, to catch invented memories. A calibrated judge scored the answers. My kind of markdown approach won.
What This Tells Us About the Vault
Here is what the numbers actually mean for the way I work. First, the anti-hallucination part matters more than people think: a memory system that confidently answers questions about things it was never told is worse than useless, and the markdown wiki was honest there too. Second, the winner was a system where the agent itself decides what to write down — not a clever retrieval layer, but a simple curation habit. That is exactly the discipline I try to keep in my vault: keep the state, throw away the noise, let the agent own what it records.
But the benchmark had honest caveats, and I want to keep them in view. Markdown is terrible at collaboration — great for a single owner, useless if ten agents need to edit the same page. And it is the least token-efficient approach of everything tested, roughly twice as costly as the most efficient product on long-horizon recall. Neither of those bites me right now: my vault is mine, and my biggest lesson from September was that a sharp, current state beats a giant pile of history every time.
The Lesson I Keep Relearning
Every time I am tempted to buy the next memory product, a benchmark like this reminds me that the tool is never what makes a memory system good. The curation is. A folder of markdown that an agent updates with intent beats any database that just soaks up everything. The expensive systems solve the storage problem; the unglamorous folder solves the thinking problem — and thinking is the part I actually need.
Have you ever benchmarked your own memory setup, or are you running on trust and marketing? I’d love to hear what your agent actually remembers — and what it quietly forgets.
Wat vond je van dit bericht?
