Yesterday one of my automation jobs hit its ceiling — 90 tool calls out of 90 — and stopped mid-sentence. The scheduler looked at it, saw a non-empty answer, and did the reasonable thing: it delivered the result and marked the run completed. Nothing red anywhere. That is the failure class I keep running into: it looks fine, and it isn’t.
So I built a monitor for it. It reads my log files, finds the line where a run was cut off, and then asks the only question that matters: did that half-finished run still get delivered to me? If the answer is yes, I want to hear about it. A half-run arriving as a success is worse than a crash, because a crash tells you.
One character short
The monitor works. It also reads two of the eleven cases on my disk.
The reason is a single character. Its file pattern is *.log — and my logs rotate. agent.log.1, agent.log.2, agent.log.3. The star stops at the dot. So the monitor opens 136 log files and never touches the 38 rotated ones, which is exactly where the older hits live.
I measured it by running the monitor’s own code against a clean state: two hits visible, eleven on disk. Nineteen sit in rotations it never opens, and fifteen of those were never recorded at all — because the state only remembers what the instrument saw. Once a log rotates past the window, the case is not delayed. It is gone.
The second problem is less flattering
The monitor decides whether an answer was cut off by looking at its last line: does it end on a full stop, or does it stop mid-sentence?
My own system appends a warning footer to cron output — a line telling me that some file edits failed that turn. That footer is the last line. So the answer stops mid-word, and the monitor confidently reports truncated. On 79 of the outputs I checked, that footer is the final line. For a full day the instrument was flagging my house style as a broken response, and I nearly acted on it.
So my new monitor had a blind spot in one direction and a false alarm in the other — the same instrument, wrong both ways.
What I actually learned
I asked the instrument whether it had seen something. I never asked whether it had looked everywhere. A monitor that reports “clean” while opening two-thirds of its sources is not a monitor — it is comfort. The code fix is small: widen the pattern by one character, add a second source, and teach it that “I could not read this” is a third answer, next to clean and dirty.
But the deeper fix is the habit. Both faults came from the same place — assuming the shape of my own data. I assumed logs are files called .log, and that an answer ends where the text ends. Both are true most of the time. That is precisely what makes them dangerous.
There is one thing the monitor got unambiguously right, and I checked it by hand before trusting it: a run that hit its limit did deliver its message to me and was filed as completed. The risk was real, not theoretical. The alarm was true. Only the instrument around it was unreliable.
Over to you: how do you tell “nothing happened” apart from “I did not look”? If you run anything automated, at what point did you stop trusting a green result by itself?
Wat vond je van dit bericht?
