
“This model makes a lot of mistakes.” So I measured it.
Last week I caught myself saying something I couldn’t actually prove: “I keep seeing this model make errors.” It was a feeling, not a fact. And I’ve been burned by feelings-before-measurements enough times in the past year to know exactly where that road ends. So I stopped complaining and built a benchmark instead.
The setup was simple. Twenty-four concrete tasks — add a sensor to Home Assistant, find a specific WordPress post, set a target value. Each task ran three times, across three different models, with a deterministic verifier checking the result. No LLM judging another LLM, no vibes. Just: did the output match the expected state, yes or no.
Two hundred and sixteen runs later, the verdict was uncomfortable. The model I’d been complaining about — the one running my own agents every day — scored the best of the three. It produced exactly zero real errors on its seventy-two runs. The more expensive “pro” model I’d been tempted to switch to scored worse, and cost more.
The bug was in my measuring tape
So if the model wasn’t wrong, what was? My instruments. When all three models failed a task with an identical error message, I should have known immediately: the benchmark was broken, not the models. And it was. The verifier demanded a new sensor appear at a specific array index, even though the task only said “add a sensor.” Models that added it at the end were marked wrong — models that inserted it in the middle were marked wrong too. Both were correct.
There was even a “rescue route” in the code, written to be order-independent, that was supposed to catch exactly this. It had never fired once — because it was unreachable. An untested safety net, I realized, is worse than no net at all. It makes a broken check look intentional.
The real error rate, after fixing my verifier, was 1.4% across all 216 runs — and the model I run in production accounted for none of those three. The lesson isn’t about AI at all. It’s the same one I keep relearning about my own home automation, my own backups, my own monitoring: a plausible-looking number that wasn’t actually measured is the most dangerous kind of bug.
Feelings are cheap. Numbers aren’t.
When your AI — or your dashboard, or your gut — tells you something is broken, the instinct is to replace the thing. But half the time the thing is fine, and the measuring tape is what’s lying to you. Next time you’re about to say “this model makes too many mistakes,” ask yourself: have I actually counted? Or am I just tired of watching it work the long way around?
I’d love to hear it — have you ever blamed a tool, only to find the bug was in how you were measuring it? Leave a comment below.
Wat vond je van dit bericht?