
The best memory doesn’t help if the agent won’t use it
Yesterday I wrote that memory is a structure problem, not a search problem. Today I have to correct myself one step further: memory is not really a storage problem at all. It is a behaviour problem. I spent weeks tuning how well my agents remember. This week a single experiment showed me that how well they remember is almost beside the point.
A researcher at Sakana AI described a memory-harness experiment that ended in a negative result — and she called that the most useful part of it. On a literature-review task where every paper already fit inside the context window, adding a memory layer changed nothing. Same accuracy, higher cost. Memory only earns its keep on long-horizon work: the question asked at step 5 and answered at step 124, where nothing fits in the window any more.
Then came the part that actually stung. She kept the model fixed and varied only the recall policy across a ladder: no recall at all, vector RAG, a ranked decisions-ledger, and finally an oracle — a setup handed the exactly correct memory every turn. The ranked ledger won. And the oracle never hit the ceiling. Hand a model the right memory and it still does not necessarily use the right memory. That is the whole lesson in one sentence.
61% of memory failures are the agent giving up
A separate test across 12 memory systems and 1,800 tasks put a number on it. The single largest cause of failure — 61% of all memory errors — was an agent that refused to answer while it could have known. Not “the fact was never stored.” The agent either never consulted memory, or searched and gave up anyway. That is a retrieval decision, not a storage problem. It is also why bad memory is expensive: every wasted turn burns tokens and walks the agent in the wrong direction.
The same test found something I had been quietly avoiding: 3 of the 12 systems leaked facts they were supposed to forget. I do not have an eviction mechanism. I keep everything. So the honest question is whether I would be in that group of three — and I suspect I would.
My own stack showed up well in that same benchmark. A plain Markdown wiki and my Mnemosyne memory layer both scored near the top — the vault-first choice I made years ago, externally measured. That felt good for about ten seconds. Then the oracle result landed on top of it: a high score is not the same as the agent using what it scored well on.
The write side: supersede, don’t delete
The sharpest thing I read this week came from a thread about the write side of memory. A discovery call contains five different kinds of things at once: “they use Snowflake” is a fact; “the data team seems overloaded” is an observation; “we do not want another dashboard” is a preference; “we will run a six-week pilot” is a decision; “security approval is pending” is open work. The agent’s job is not to store the transcript — it is to file each piece at the right level on the way in.
And the fix for stale memory is not deletion. It is marking the old claim superseded and keeping it, so the agent can see that something changed and when. That is far cheaper than a decay algorithm, and it keeps the provenance. It fits the same principle I landed on for adversarial memory writes: never let a newer line silently erase an older one.
What I am going to measure
Three checks, because a metric beats a feeling. Does my agent give up while it knew the answer? That is the 61% category, and I have never counted it. Do I leak facts I archived? That is the 3-in-12 test. And can I replace “delete” with “supersede”? None of the three is a new component; all three are things I can measure this week.
If you have measured your own gave-up rate — or you simply trust your agent to try before it quits — I would like to hear how you handle it. Comments are open.
Wat vond je van dit bericht?