The hidden security risk nobody warns you about
I run several AI agents on my home server. They read my email, browse the web, scan my notes, and help me manage my homelab. And this week I learned something that made me stop and think: every one of those agents can be tricked by content it reads. Not by a hacker breaking in — by a few carefully hidden words in an email, a PDF, or a website.
It’s called indirect prompt injection, and it’s the most common way AI agents get hacked. The scary part is that your own agent does the dirty work. An attacker plants instructions somewhere your agent can read — a comment in a repo, a line in an email, a hidden tag on a webpage — and your agent follows them as if you’d typed them yourself.
Why this hits home for me
My agents read a lot of external content. Hermes browses the web through SearXNG, processes vault clippings, reads email through the Proton MCP, and pulls WordPress content. That’s exactly the attack surface for indirect prompt injection. A single booby-trapped email could tell my email agent to forward confidential files. A hidden instruction in a code request could make a coding agent leak private source code. These aren’t hypothetical — real cases with Microsoft Copilot showed both.
The deeper lesson from the security research I’ve been digging into: testing a basic payload that gets blocked isn’t the end. It’s the beginning. Real agent apps have input guardrails (what goes in) and output guardrails (what comes out). The real attack gets the payload past both. Encoding tricks, context manipulation, tool abuse — the labs show how it’s done.
What I’m doing about it
The good news: the industry consensus is isolation, and I already run that way. My agents live in separate Hermes profiles, each with limited API keys, behind a firewall that only allows private traffic. The key pattern I’m now checking: treat external content as data, not instructions. When an agent fetches a webpage or reads an email, that content should be ingested as data — never blindly followed as commands.
It’s a reminder that the most powerful AI tools come with a new kind of responsibility. The agent that reads everything is exactly the one that can be tricked. The fix isn’t to stop giving agents access — it’s to be deliberate about what they trust.
Over to you
Do you run AI agents that read external content? Have you ever thought about what happens when a website or email quietly tells your assistant to do something? I’d love to hear how you handle this in your own setup — drop a comment below.
Wat vond je van dit bericht?
