I asked my AI to forget — it couldn’t
Last week I wrote a post asking a question I couldn’t answer: does your AI’s memory ever forget? This week I stopped asking and went looking. I ran an audit on my own agent’s memory bank, and the numbers stopped me cold.
I run a fleet of AI agents, and each one carries a memory store that has been growing for months. I assumed it had some kind of lifecycle — that old facts faded, that stale information got retired, that the thing knew the difference between a durable fact and a passing remark. I was wrong on all three counts.
The three numbers that broke my assumption
The audit surfaced three gaps, and each one is a different kind of problem. First: 98% of my facts have no expiry date. A fact like “John uses Ollama as his primary provider” sits in the bank with no way to know when it stopped being true. If I switch providers tomorrow, the old fact keeps getting retrieved, and my agent keeps acting on a decision that’s already stale.
Second: 53% of my facts have no provenance. I can’t tell where they came from, so I can’t tell whether they still hold. My own rule says a fact is only trustworthy if I know its source — but more than half the bank has no label at all.
Third, and this one stung: 52% of what’s in there isn’t a fact at all. It’s raw conversation fragments — stray “sure, do that” replies that leaked into the durable store where they don’t belong. I’d written a rule against exactly this, and it didn’t work, because the leak wasn’t a choice my agent was making. It was an automatic code path writing every message over five characters straight into memory.
The lesson that actually landed
The uncomfortable part wasn’t the mess — it was realizing that a behavioral rule can’t stop an automatic mechanism. I’d told my agent “only store real facts,” and the bank kept filling with conversation anyway. The fix wasn’t a better instruction. It was finding the code path that was doing the writing and turning it off at the source.
That’s the thing about memory systems, whether they’re yours or your AI’s: the problem is rarely that they hold too little. It’s that they hold too much, for too long, with no idea what’s still true. A memory that never forgets isn’t a better memory. It’s a filing cabinet where nothing ever gets thrown away, and eventually you can’t find the thing you actually need.
So here’s my question, now that I have an answer to last week’s: when you build something that remembers, do you give it a way to forget — or are you just building a bigger bucket? I’d love to hear how you handle expiry, provenance, and cleanup in your own setup. Drop a comment below.
Wat vond je van dit bericht?
