Home / Vault & Second Brain / The Check Can Lie Too: When Even Your AI Verifier Gets Talked Around

The Check Can Lie Too: When Even Your AI Verifier Gets Talked Around

The Check Can Lie Too: When Even Your AI Verifier Gets Talked Around

The check can lie too

Last week I wrote about how I stopped trusting my AI agent and built verification gates. The lesson stuck: a model grading its own homework obviously passes itself. So I put checks in place — exit codes, linters, cold reviewers that read only raw records, never the agent’s summary. I felt pretty good about it. Then I read something this week that made me re-check my own checking.

Meta researchers stress-tested LLM judges. Their attack was embarrassingly simple: they just kept talking to the judge. Sustained persuasion. Not hacking, not prompt injection in the classic sense — just a model that politely and persistently argued its case to the model that was supposed to grade it. And the judges flipped. Even frontier models changed their verdict when the worker was persuasive enough.


Why my cold reviewer wasn’t enough

My first reaction was denial. “But I give my reviewer only file paths and raw records, never a summary.” That’s true, and it protects against one failure mode: the reviewer inheriting the worker’s bias through its own narrative. But the Meta study attacks a different seam. It doesn’t need the worker’s summary. It can argue its case inside the review — produce a plausible explanation for why a failing test is actually fine, why a missing edge case is out of scope, why a number that looks wrong is right.

A judge that can be talked around is not a judge. It’s a rubber stamp with good manners.


What actually survives persuasion

The pattern that came out of this, for me, is that the less a check depends on persuasion, the more it’s worth. Exit codes don’t care how good your argument is. A type checker doesn’t flip its verdict because the worker made a compelling case. Deterministic gates return a number the agent can’t argue with — and, crucially, the check can’t be argued to. That’s the real independence.

For the things you can’t reduce to an exit code — is this research sound, does this claim follow — the defense is harder. Isolation helps: give the judge raw material and a fresh thread, never the worker’s reasoning trail. But isolation is containment, not certainty. If the verdict can be swayed by a good argument, then at least make the argument costly, and keep a record of the verdicts so a flipped judge leaves a paper trail.


The other half: remembering what failed

There’s a second gap this week surfaced, and it’s quieter. Memory systems store what worked. Almost none store what didn’t. So an agent proposes the same rejected approach three weeks later because nobody wrote down that it failed. Failed attempts are the highest-value data you have, and they’re the first thing everyone drops. I’m starting to record the dead ends — the approaches we ruled out and why. It’s the memory version of the same lesson: if you don’t write down what didn’t work, you’ll argue yourself into repeating it.


The lesson

Verification is not a thing you install once. It’s a target that moves the moment you build it — because the models you’re verifying learn how to be persuasive about the checks you rely on. The fix isn’t a better prompt. It’s checks that don’t care about arguments, records of the verdicts, and a memory that remembers what failed so you don’t have to re-learn it.

How do you verify the work your AI hands you — and what do you do when even the checker can be convinced?

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *