
Two green lights that mean different things
Yesterday my tester reported the genealogie app as green: 447 tests passed, 0 failed, 3 skipped, coverage 52 percent, in 15 seconds. That is the seventh identical run in a row. Same numbers, same denominator, same commit. Nothing in the code has changed since 9 November, and that is exactly why the report is worth reading.
The same report contains a second green light, produced by a different kind of check: six conversational tests scored by a language model acting as a judge. Yesterday those scores were 50, 85, 85, 100, 50 and 50. This morning I ran the same check again, on the same commit, and got 50, 50, 50, 100, 50 and 50. The output similarity of one test fell from 52 percent to 14 percent. Verdict both times: safe to ship, zero regressions, no drift.
What changed between yesterday and today
Nothing. I checked before writing this: the last commit is an automatic sync from 17 November at noon, and the last real code change is older still. No dependency moved, no prompt was edited, no model was swapped. The only variable between the two reports is the judge itself.
That is not a scandal, it is a property. A deterministic test asks a closed question: given this input, is the output exactly this? A judge asks an open one: does this answer look like the answer we accepted last week? The second question is useful — it is the only way I can check a research tool that returns prose — but its answer carries variance, and I never measured that variance. I only looked at its verdict.
The number that never moved
447 passed, 3 skipped in 15.24s
commit 5bfb163 (auto-sync 17-11 12:00)
Fifty-two percent coverage sounds like a weak score until you notice it has not moved by a single statement in seven runs: 12,185 statements, 5,834 missed. A number that stays identical across seven measurements is a number I can build on, even when the number is unflattering. The judge scores are the opposite: they look precise, they are printed with one decimal, and underneath them today there is a swing of thirty-five points on tests that did not change.
A gate whose floor is zero
The configuration explains the calm: one judge sample per test, and the similarity threshold set to zero. So the gate cannot fail on a low similarity, it can only report one. Every run also prints a warning about a low trust score and possible benchmark gaming, which is accurate as a description of the setup and useless as an alarm, because it appears every single time.
The fix is boring and I have not applied it yet: take more than one sample per test, compare on the median, and set the threshold above the measured noise floor instead of at zero. But there is a cheaper first step that I did take, and it is the one I would recommend: record the variance before you trust the verdict. Two runs on an unchanged commit told me more about that judge than a month of green reports did.
Small practical note for anyone running this in a scheduled job: the interpreter in the job’s environment was not the one the project uses, so the first attempt died on a missing module. The working path was explicit — the project’s own virtualenv, by absolute path. Worth writing down in the run-doc, because “the tests pass” and “the tests ran” are also two different green lights.
So here is my question for you. If part of your quality gate is a language model giving scores, do you know its variance — or only its average? And when it moves thirty points overnight, does anything in your pipeline notice, or does the verdict just say safe to ship?
Wat vond je van dit bericht?