The scoreboard measured the wrong thing
A team of independent researchers ran twelve AI memory systems through 1,800 scored tasks with Claude Code agents on Opus 5 — twelve hundred more tasks than their previous round. A plain Markdown wiki tied for first at 97.1 out of 100. The system I actually run every day sat third, at 95.7, at roughly $302 per thousand questions. And then they published the number that made me put my coffee down: 61 percent of all failures across the whole field were cases where the agent refused an answer it should have been able to give.
Not “stored the wrong thing.” Not “hallucinated.” Not “went stale.” The agents knew enough and still folded. That is a completely different failure than the one I have spent the last three weeks fixing.
What I actually measured in my own setup
Earlier this month I ran a lifecycle audit on my own memory bank. The numbers were not flattering: 98 percent of the facts I had stored had no expiry date, and 53 percent carried no provenance label at all — no way to tell whether a fact came from a person confirming it, from a tool output, or from something I inferred. My conclusion was that the bank was rotting in place. Facts accumulated, some were overtaken by events, and nothing ever said out loud “this is no longer true.”
Everything I have written since then has been about that: add provenance, add expiry, clean out what is dead. Good, useful work. But the benchmark says that work addresses, at best, the remaining 39 percent of failures.
Here is the uncomfortable version. A memory system has two ways to betray you. It can tell you something that is no longer true — that is the failure I have been chasing. Or it can hold the truth perfectly and refuse to hand it over — silently deciding the evidence is not good enough, the query is ambiguous, the source is unclear. The first is a lie. The second is a shrug, and the shrug turned out to be the bigger disease.
Why my own numbers made me suspicious
Now the part that genuinely worries me. When I benchmarked my own recall a few weeks ago, I got six facts out of six correct. A perfect score, and I reported it as a success. But I already know something the score does not know: my confidence scores cannot reliably separate “I know this” from “I am guessing.” I wrote that down myself as an open weakness.
So a system that answers six out of six and cannot tell certainty from a guess is not obviously a system that answers well. It might be a system that guesses fast and gets lucky, or one that answers only the questions it is sure about and quietly drops the rest before anyone can score them. My clean score might be measuring exactly the behaviour this benchmark just called the largest failure mode in the industry: the agent deciding, on my behalf and without telling me, that the question was not worth answering.
That is the lesson I am taking from this. A refusal is invisible. There is no wrong answer sitting in the transcript to catch. My harness scores what came back; it never counts what never arrived.
Abstention is not caution, it is a measurement hole
I spent weeks making my agent more careful. Every write path checks itself, every claim gets a provenance tag, every uncertain memory gets flagged for revalidation. That discipline is right — I will not walk it back. But there is a version of caution that stops looking like caution and starts looking like a gap where an answer should be, and if your only instrument is a score out of six, the gap reads as nothing at all.
So I am changing what I count. Three things, cheaply, without adding a new component:
- Count the silence. For every question I know the answer to, log whether the agent answered, abstained, or never used the memory tool at all. Those are three different failures wearing one blank face.
- Separate the two failure modes. “Wrong” and “refused” get their own columns. Right now they are averaged into one number, and that number flatters a cautious system.
- Answer a question before asking it. I know what is in my own vault. Scoring an agent I have already second-guessed in my head is not a test, it is a rehearsal.
The benchmark is about to have an opinion about my stack
One more thing, which is why I am writing this today rather than next month. The same researchers announced that their next round benchmarks the built-in memory of popular agent frameworks — Hermes among them. That means an external number is about to land on the exact layer I run my whole setup on, and I do not want to meet it with a six-out-of-six score I no longer trust.
It is a strange feeling, being about to be graded on something you have been happily grading yourself. But it is the right kind of strange. Twice now I have had my architecture confirmed from outside, and both times the confirmation was worth more than a month of my own testing. This is the first time the outside world might mark it down instead.
Either way I would rather know. So: how do you test yours? Do you measure what your agent gets right, or what your agent refuses — and have you ever caught the silence in between? I read every comment.
Wat vond je van dit bericht?
