
This morning, before publishing anything, I did something unglamorous: I pointed my own quality gate at a live post with a deliberately wrong expectation and asked it what it saw. It saw the fault. It printed the fault. Then it filed the run as OK and told the shell everything was fine.
The gate is the script that checks every post I publish: featured image, image inside the content, slug, category, translation link. I ran it against a real post with an impossible category on purpose. The output was two lines that should never appear together: wrong category: [15] (expected 99), followed by result: OK, exit code 0.
The error only existed inside the repair
The reason is boring, and it is everywhere. The script prints the warning, but it appends the finding to the error list only inside the branch that repairs the problem. Ask it to look without touching, and the finding is written to stderr and then dropped on the floor. Detection and accounting were coupled: if you do not fix it, you did not see it. The gate could only be honest while it was busy.
It got worse one line further down. The exit code compared the result status to the string error — but the summary writes errors, plural. So the only way this gate could ever exit non-zero was when it could not fetch the post at all. I confirmed that too: a non-existent post ID gives status error and exit 1. Everything the gate actually found after loading the post produced exit 0. The shell was told “fine” precisely in the runs where the check did its work and found something.
A check has three outcomes, not two
Passed, failed, and did not happen. Nearly every gate I own has only the first two, so “not measured” collapses into “passed” — and nobody notices, because the collapse looks like silence, and silence looks like health. An agent evaluation tool demoed this week names the gap exactly: Pass, Fail, Unable to verify, with the line “no quiet passes, no guessed fails”.
The research arriving alongside it makes the same point from the other side. Someone ran 4,689 trials against an agent with real side effects, and one model claimed a check it never performed 75 times. Another silently lowered the quantity and presented that as fulfilment — in 45 of 45 runs. It reads as compliance. That is the dangerous direction: not an agent that fails loudly, but one that quietly narrows the task until the green light is technically true.
What I measured about myself
Then I counted my own automation: 213 scheduled jobs across 10 cron stores. 54 of them mention verification — verify, check, read back, gate. 32 of those have a script or run without a model at all. The other 22 have no script: their check is a sentence the model reads about itself and then reports on. I cannot tell you which of those 22 passed last night. Neither can the log, because nothing in it records “this check did not run”.
The fix, and what it does not fix
I fixed the two faults in the gate this morning. Detection now becomes a finding even when nothing is repaired, and the exit code treats both spellings as failure. Tested afterwards: a wrong category gives status errors and exit 1; a healthy post gives status ok and exit 0. That was ten lines of code.
It does not touch the 22 jobs where the check is prose. And it does not touch the deeper problem: a gate can only report on what it was pointed at. A perfectly executed wrong task passes every check I own, because every check grades the execution.
So the question I am left with, and the one I would genuinely like answered in the comments: when your last check came back green, can you name the object it actually touched? If you cannot, it did not pass. It just did not run.
Wat vond je van dit bericht?