Home / Vault & Second Brain / Memory is what separates a chatbot from an agent

Memory is what separates a chatbot from an agent

Second brain: memory as structure, four rooms

Memory is what separates a chatbot from an agent

I run a stack of AI agents every day, and for the longest time I thought my memory problem was a search problem. If I just indexed everything well enough — tags, vector similarity, cross-links — the right fact would surface at the right moment. This week, three independent sources landed on my desk with the same correction: memory is a structure problem, and the structure has four rooms, not one.

IBM’s breakdown of the CoALA taxonomy (Cognitive Architectures for Language Agents, from Princeton) is the clearest map I’ve seen. Every agent’s memory splits into four types. Working memory is the context window — fast, volatile, bounded, the RAM of the thing. Semantic memory is the knowledge base, and here’s the part that made me smile: in production, they argue, this is something much simpler than vector databases — it’s just Markdown files. Procedural memory is the agent’s skills. And episodic memory is the distilled record of what actually happened.

That map fits my own setup exactly. Working memory is the context window I fight to keep lean. Semantic memory is my Obsidian vault — and I already learned, the hard way, that Markdown beat every dedicated memory product I tried. Procedural memory is my skills folder. Episodic memory is what Mnemosyne stores as distilled experience. I wasn’t missing the taxonomy; I just didn’t have a shared language for it, so every memory discussion started from scratch.


Forgetting is an engineering problem

But the map exposed two rooms I’d left empty. The first is decay. IBM put it in one sentence: humans are good at forgetting — for agents, forgetting is an engineering problem. My stack is excellent at remembering. I’ve tuned the intake. What I don’t have is an explicit mechanism for eviction: when does a stored fact age out of recall? Right now, nothing ever quietly fades. It all accumulates, and accumulation without decay is just slow context bloat.

The second empty room has a concrete number attached. Someone on Reddit described an orchestrator-worker setup where every orchestrator call replayed the entire project state — and measured that over 60% of the orchestrator’s tokens were re-reading context it already had. $580 a month of re-reading. The fix wasn’t a better model; it was a ranked summary of the current state instead of a full history replay. That single change halved the token spend.

I know that pain. My own prefill cost measures around 312K input tokens per turn on some jobs — and a chunk of that is history I’ve already paid to load once. The lesson isn’t “store less.” It’s that the orchestrator shouldn’t get the whole transcript back every time; it should get a distilled state summary — intent, changes, decisions, next steps. Remembering everything is cheap. Re-reading everything is what burns you.

Yesterday I wrote about the adversarial side of agent memory — a write that says nothing false can still wipe a verified fact. Today is the structural side of the same coin. Immutability and signer-authority protect what’s already stored. Decay and ranked summarization decide what’s still worth keeping in the window. You need both, and I only have the first half.

So here’s my question for you: does your agent forget anything on purpose? Or does it, like mine, just keep piling things up until the context window is a landfill of things it already knew? I’d genuinely like to know how you handle decay — leave a comment.

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *