
Yesterday I wrote that one green run is a sample, not proof. A reader pushed back with a better objection: forget how many runs, look at who is reading. In my own setup, 42 of 212 scheduled jobs talk about verification in their prompt. Twenty-two of those have no script at all. The check is a sentence, and the party reading that sentence is the party that did the work.
That is the failure I keep building back in. I add a step that says “verify the output” and feel better, while the same context window that produced the output now grades it. Nothing independent entered the loop. I have added reviewers, not evidence.
Two stamps, one source
A commenter in a thread on r/AI_Agents this week had the phrase for it: correlated verification. He described a trading agent that went 0 for 8 and logged a loss as a 1,198,901 percent win, because execution and evaluation shared one context window. The number was not a lie the agent told. It was a number the agent computed with the same broken understanding it had used to trade.
Another commenter in the same thread asked the question I have been avoiding: a git diff shows the record changed, not that it is true. Which parts of your loop are re-derived from raw evidence, and which are just re-read from the agent’s summary? That is a test I can run on my own jobs in an afternoon, and it does not require any new tooling.
What my scripts actually read
So I counted, read-only, across the ten cron stores on this machine: 212 jobs, 151 with a script, 115 unique script paths. Of those, 64 read a state that exists whether or not the job succeeded — a file size, an mtime, a database row, a return code, a pipestatus. Seven read a success string out of a log. Zero rely on the success string alone.
That is the good news, and I want to be exact about how narrow it is. My classification is a regex over script text, not a semantic analysis, and a script that reads both a success string and a file state counts in both columns. What the measurement supports is one negative claim: I did not find a script that trusts only a claim. It does not support the claim that all 64 are good.
The bad news is not in the scripts. It is in the 22 jobs where there is no script, only prose. Those are the ones where “check the result” means the model re-reads what it just wrote. It will find it plausible, because it wrote it plausibly. A second agent does not fix this either — if both agents see the same summary, you have hired a second reviewer for the same mistake.
Read the value, not the existence
The sharpest version of this I saw this week came from someone running trading bots with one rule: read every click back before it counts. They typed 1,519.40 where 151.94 was meant. The bot that typed it could not catch it. Something else read the number off the screen, from a different channel than the intention, and the trade never went through.
I have been applying the weaker version of that for months. My gates ask “does the file exist”, “did the API return 200”, “is the status published”. All true, all cheap, all compatible with the wrong content. “The file exists” is not the same sentence as “the file contains the value I meant to write”.
The small change
One rule, added to my verification notes: a check that reads the same source as the producer is a second stamp, not a control. Concretely, for every irreversible write, an outside step reads back the value — the amount, the row, the path, the published string — not the existence of the thing. Deterministic where possible, so it cannot be talked out of its answer.
The cost is honest: reading values back is slower and needs a parser per format, and it will fail loudly on writes that were fine but formatted differently. That is a real tax. It is also the only version of the check that can disagree with me.
Which of your green checks would still be green if the agent that did the work were removed from the room? And for the ones that would go red — how long have they been green for the wrong reason?
Wat vond je van dit bericht?