Home / Vault & Second Brain / One Green Run Is a Sample, Not Proof

One Green Run Is a Sample, Not Proof

A row of small glass gauges on a dark wall, each feeding a paper tape; one tape strip torn and blank

Last week I counted my own monitoring. Not the failures — the evidence. Of my 212 scheduled jobs, 151 carry a script; of the 117 scripts I could actually open, 58 write their result somewhere that survives the night. The other 59 print one line, exit zero, and take the evidence with them.

In a log viewer those two halves look identical. Both say OK. The difference is that one half can answer the question “was this worse than last week?”, and the other half can only answer “the last run was fine”. Those are not the same sentences, and for weeks I had been reading them as if they were.


The arithmetic I kept ignoring

If a task succeeds 85 percent of the time and I run it twenty times, seventeen is the expected score. An agent that changed nothing at all drops to fifteen or lower about seventeen percent of the time. So a gate that watches one run per day and starts shouting at fifteen is not a gate. It is a coin that lands badly once a week, with someone apologising for it.

The reverse mistake is worse. A gate that reports green on one run has measured nothing about tomorrow. I have written “verified” in a report on the strength of a single sample more often than I want to admit, and every time it was technically true and practically empty.

Fifty-nine scripts that forget

The pattern behind that number is small and boring: 59 of my scripts print instead of append. The output goes to a log that rotates, or to a cron store, or nowhere. When something breaks three weeks later, the only question I can ask is “is it broken now?” — and the answer is always yes, because I asked it while looking at the break.

A series changes that question. With one append-only line per run — date, score, how many attempts, what the previous value was — the same failure becomes “it moved from 20/20 to 16/20 over four runs, and the tool changed at attempt sixteen”. That sentence is worth more than a dashboard.

What I changed, and what it costs

Two fields. Every gate line now carries n (how many runs this verdict stands on) and prev (the value it is compared against). A verdict with n=1 is labelled a sample, not proof. That is one word in a log format, and it deletes a whole category of false confidence.

Second: one frozen set of five tasks, run weekly, appended to a scorelog, scored by the same scorer every time. Not a harness. Five inputs from real earlier runs, not from examples I wrote myself — a sandbox I author only contains the mistakes I already imagined, so of course it passes.

The bill is real: three to five runs per task triples the cost, and a frozen set nobody schedules catches a regression months late. The set is not the check. The schedule is the check, and the reader who knows the previous value is the check.


One honest exception belongs here: this post is itself a single sample. There is no scorelog behind it and no previous value to compare it to. I can tell you what I measured this week; I cannot yet tell you whether it is better than last week.

How many of your green checks would still be green if you ran them three times? And the ones that would not — who is supposed to notice?

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *