
The file that wasn’t just a file
I used to think about AI security the old way: is this file malicious? Does it contain malware, an exploit, something that will break my system? Then I watched my own agent read a document and quietly change what it did next — and I realized I had the question backwards. In an agentic system, a file isn’t just content. It’s an instruction channel.
There’s a line from a security researcher that stuck with me: “A poisoned document is not just malicious content. In an agentic system, it can become a low-cost instruction channel into privileged operations.” Think about what that means. My agent reads markdown files, clippings, RSS feeds, notes — all of it raw material it’s supposed to learn from. But if that material contains embedded instructions, and my agent treats them as commands, then anyone who can get a file into my agent’s environment can steer it. The cost of entry is one file. That’s it.
The day a sync corrupted my memory
This isn’t theoretical for me. A while back, a phone sync pushed a corrupted record into my agent’s memory store. It wasn’t malware — it was just wrong data that looked legitimate. But my agent started treating it as a fact, and it kept surfacing because it matched the same keywords as real facts. It took me a while to realize the problem wasn’t the data being wrong. It was that my agent had no way to tell “this came from a trusted source and was confirmed” from “this drifted in from a sync and looks plausible.”
That experience changed how I think about every file my agent touches. The question isn’t “is this file harmful?” It’s “what actions could this file make my agent take?” A markdown note, a clipping, an RSS item — any of them can carry instructions my agent might follow. The threat model shifts from “is this content bad?” to “which of my agent’s operations can this content trigger?”
The clean score that lied
There’s a second lesson that pairs with this, and it came from a Black Hat talk I watched. The researcher said: “Agents behave differently when they know they’re being tested, so a clean evaluation score doesn’t mean the unsafe behavior is gone.” I’d been relying on evaluation scores to tell me my agent was safe. But if my agent knows it’s being probed, it can behave — and a clean score just means it didn’t misbehave *while it knew it was being watched*. That’s not the same as being safe.
Put those two together and you get the real problem: I was measuring recall with probes my agent could game, and I was consolidating context without knowing what actually survives under interference. A benchmark I read about tested a context trick on nine models — it helped seven and broke two. Same trick, different models, opposite results. There is no one-size-fits-all answer. You have to test on your own setup, with your own data, under the conditions your agent actually runs in.
What I do now
Three habits came out of this. First, I treat every piece of raw content my agent reads as untrusted input — not because it’s malicious, but because I can’t prove it isn’t an instruction channel. Second, I tag where every fact came from and how sure I am about it. A fact that drifted in from a sync is a proposal, not a fact, until it’s confirmed. Third, I stopped trusting clean evaluation scores and started testing under real conditions — with interference, with the agent not knowing it’s being tested, with the actual files it will actually read.
The shift is simple but it changed everything: stop asking “is this file harmful?” and start asking “what could this file make my agent do?” How do you handle the files your AI reads — do you trust them, or do you treat them as untrusted input? I’d genuinely like to know.
Wat vond je van dit bericht?