The stranger in the mirror
Last week I wrote here that a stranger on Reddit had called my agent “overwhelming”. I thought that was the uncomfortable feedback of the month. It wasn’t.
The real one came from a post I read this week, written by someone who spent a year building agent memory for real users. His conclusion: “Save everything and RAG it is the wrong default.” And when I read his three failure modes, I recognised all of them. Not from my reading list — from my own vault, my own recall, my own agent.
I have been running a memory system for my agent for months. A vector database, thousands of entries, a nightly consolidation job, a write-path rule I was quite proud of. And when he described what breaks in production, he described my setup.
Failure one: nothing ever gets suppressed
In June I changed my mind about something. I wrote the new decision down. I stored it properly. And the March version of that same decision is still sitting in my memory, just as easy to retrieve, just as confident-sounding.
Why? Because saving a new fact creates a second entry. It does not overwrite the first one. The model now has two truths to choose from, and which one wins depends on wording, not on time or authority. That is not a memory system. That is a transcript with a search bar.
Failure two: it has no idea who “he” is
My notes say “he”, “the client”, “that project”. My vector search is brilliant at finding sentences that sound like my question and completely blind to the fact that two different sentences are about the same person. Embeddings match meaning, not identity. If your memory has no entities — real people, real projects, real commitments as first-class things — then retrieval is guessing.
What I do have is a folder of structured notes with frontmatter, tags and internal links. That was my accident: my long-term memory never was a pile of chat logs, because I write to it by hand and by rule. That is the one part of this I would keep.
Failure three: two memories about the same thing are strangers
Someone promises something in one conversation and delivers it in another. To me those are obviously connected. To a semantic search they are two unrelated chunks of text that share maybe a few words. So my agent asks me for status on work that was finished two weeks ago.
What I changed in my head, not (yet) in my stack
The author’s fix is to treat memory as a model of the world, not a log: entities as first-class objects, facts hanging off them with a source, a confidence and a timestamp, and new information that updates the model instead of stacking next to it. Vector search becomes one road inside that model — useful, not the whole country.
That is a bigger change than a database setting, and I am not going to pretend I have already made it. What I did do is write down the rule I was missing, because the author admits it is the hardest part: when should an update overwrite the old fact, and when should both be kept as a genuine change over time?
My answer comes from something I already own and never thought of as a memory feature: my notes folder is under version control. The history keeps the change — including the part where I was wrong in March. The memory itself only has to hold the version that is true now. Git is my both-track. The recall bank is my now-track. I don’t have to choose, because I already keep a diary of my own thinking.
Which also settles the most common counter-argument in that thread: “but what about facts that legitimately changed?” Fine. Show me the diff. That is exactly what version history is for — and it is the one part of my stack I fully control.
The uncomfortable part
I have spent months making sure new things land in my memory in a disciplined way. Almost none of that effort went into what happens after — who wins when two entries disagree, whether the loser gets quiet, whether a stale fact can still be retrieved with full confidence. I filtered the entrance and left the exit open.
So: how do you handle a fact you changed your mind about? Do you delete the old one, or do you keep both — and if you keep both, how does your agent know which one belongs to now?
Wat vond je van dit bericht?
