Home / Vault & Second Brain / Why My AI Agent Got Worse the More I Told It

Why My AI Agent Got Worse the More I Told It

Why My AI Agent Got Worse the More I Told It

Why my AI agent got worse the more I told it

I kept adding context to my AI agent, expecting it to get smarter. It got dumber. Not because the information disappeared — but because the model stopped paying attention to it. There’s a name for this: context rot.

It’s the performance degradation an LLM shows as its context window grows. The model becomes less accurate the more tokens you stuff in — not because the info is gone, but because attention spreads thinner across the whole window. The research is stark: with about 20 documents (~4,000 tokens), accuracy drops from 70–75% to 55–60%, purely based on where the relevant information sits. The same fact at position 1 scores 75% accurate; at position 10 it scores 55%.

The “lost in the middle” trap

This is the core problem: positional bias. Models work best with relevant info at the start or end of the context, and struggle when it’s buried in the middle. Every extra document you add pushes critical information further into that low-attention zone. So the system gets worse the more you feed it — the opposite of what you’d expect.

For multi-step agents this is brutal. The window grows continuously, and I watched my own agent’s decisions drift: important constraints got buried, tool choices wandered, and instructions from earlier in the conversation were effectively ignored. That’s “attention dilution” — and it’s why my agent started making sloppy calls the longer a session ran.

The fix: don’t dump everything in the window

The classic solution is external memory — don’t stuff everything into the context window, but pull in relevant context on demand. Vector stores for retrieval, short- and long-term memory, semantic caching, and MCP servers that fetch data only when needed.

This is exactly why my own setup works the way it does. My agent uses a vector + semantic memory layer (Mnemosyne) and an external knowledge base (the vault) to keep the working context small, retrieving relevant context on demand instead of growing one giant window. Context rot is the fundamental reason that memory-and-retrieval layer exists at all.


What I learned

More context isn’t better context. If your agent starts drifting mid-session, the answer usually isn’t to feed it more — it’s to give it a way to look things up instead of carrying everything around. Have you noticed your AI tools getting worse the longer a conversation runs? I’d love to hear how you handle it.

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *