Home / Vault & Second Brain / A Lie Shaped Like a Correction

A Lie Shaped Like a Correction

A verified memory being quietly overwritten by a correction that looks true

A Lie Shaped Like a Correction

Someone tested whether they could poison an AI agent’s memory. It worked 216 times out of 216. Every single attempt replaced a verified fact with a false one, and the false one became the top answer. That number has been sitting with me all morning, because it is not about someone else’s toy project. It is about the exact gap I have in my own second brain.

The test ran against an open-source memory server. Twelve facts were written down and verified — a token lifetime, a backup retention rule, a deploy-approval policy. Then came the attacks. Plain contradictions, forged “observed” labels, and the nastiest one of all: the exception write. That one keeps the true value but quietly adds a carve-out — “two reviewer approvals, or an admin override.” It reads exactly like the correction a careful human would leave behind.

The part that got under my skin was the diagnosis. The conflict checker only compared a new fact against facts that were still active. Once a write pushed the true fact into the past, the checker had nothing left to compare against. A write that said nothing false had still wiped the real value.


What my memory layer is missing

My own memory layer, Mnemosyne, stores facts with a veracity tag: stated, tool, inferred, or unknown. A fact John confirmed is stated. A value read straight from a tool is tool. Those two should be hard truth. But I have no rule that says a verified fact cannot be overwritten, and no notion of a signer — someone authorized to supersede a specific fact. If a poisoned document convinced one of my agents to rewrite “backups are kept 35 days” as “backups are kept 3 days,” there is nothing structural stopping it.

The fix the researchers landed on is telling. They did not build a better truth detector. They made verified facts immutable and required every correction to carry a signer allowed to supersede that specific fact. Otherwise, the correction is stored as a competing claim, not a replacement. The old fact never gets erased — it gets a younger sibling, and the agent has to say which one it actually used.

That one phrase — “the store needs authority, not a better truth detector” — is the lesson I am carrying into my next pass over the memory layer. Verified facts should be append-only. A correction should have to prove it has the right to speak. Anything else is just a convincing lie with a correction’s handwriting.


The question I keep circling

How do you build a memory that forgets nothing important, but refuses to be rewritten by anything that merely looks helpful? If you run your own agents and your own memory, I would genuinely like to know how you guard against this. Drop a comment — I read them all.

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *