
A self-improving skill without an eval is just a gut feeling
I read a line last week that stopped me cold: “A self-improving skill without an eval is nothing more than a gut feeling.” It hit home, because my AI agents have been improving themselves for months — patching their own skills after every task — and I had no idea whether any of it was actually making them better. I was flying on vibes.
So I decided to start measuring. Not a full evaluation suite — that would be overkill. Just a lightweight daily score-log on the few things that actually matter. For my news crons, that meant tracking four numbers: how many unique sources each top-10 used, how often a source failed, how relevant the articles were, and how fresh they were. Simple, automatic, one line a day.
The measurement caught a real problem on day one
Here’s the thing about gut feelings: they don’t catch regressions. The very first day the eval ran, it flagged that my AI news cron had selected only three unique sources for the top 10 — when the whole point of switching to an RSS aggregator was to get eight to eleven. I had been reading that top 10 every morning and never noticed the drift. The eval saw it in one run.
That one number changed how I work. I added a diversity rule to the cron prompt (at least six unique sources, max two per source), fixed a field-name drift in the eval script, and now the score-log tells me every day whether the aggregator route is actually better than the old search fallback. After a few weeks I’ll be able to say “the aggregator is three times better” instead of “it feels better.”
The lesson I keep coming back to
If you’re building agents that improve themselves, you need a way to know whether the improvement is real. Otherwise you’re just polishing a guess. It doesn’t have to be elaborate — a simple score-log on the two or three things that matter is enough. The point isn’t the tooling. The point is that “reported done” and “actually verified” are two different things, and only one of them should count.
What about you — do you measure whether your AI setup is actually getting better, or are you running on gut feeling too? I’d love to hear how you track it.
Wat vond je van dit bericht?