The backup job on my server failed twice this week, and the snapshot it was supposed to protect was sitting on disk the whole time — complete, verified, valid. That is not a story about a crash. It is a story about a job that finished its work and then failed anyway.
Four times a day, a scheduled job copies my main state database to a snapshot. The database has grown to roughly four gigabytes, and copying it takes about four minutes. On Monday morning the job published a fresh snapshot at 06:14, then kept running. At 07:11 the scheduler killed it on a one-hour timeout. The job was reported as failed. The backup was fine.
The successful half and the unverified half
The script writes to a temporary file, validates it, and then renames it into place. That part worked perfectly. What came after is where the hour went: a row-count comparison against the live database, and a rotation step that decides which old snapshots to keep. The rotation loop validates every snapshot it finds — and validating a four-gigabyte SQLite file means running a full integrity check over the whole thing. Five of those per run, on mostly unchanged files.
The snapshot was already on disk and already verified. Everything after that point was bookkeeping. And the bookkeeping is what the timeout landed on — fifty-seven minutes after the last write.
Here is the part that made me stop and re-read the file. The line right above the copy step, written by me, says: “No timeout — let it finish.” I had reasoned about the copy, measured it, concluded it needed room, and written that down. Then I never asked what else the script does after the copy, or whether that work fits in the same window. The script and the scheduler disagreed, and nothing in between was checking.
Nobody noticed, and that is the real finding
I run monitors. One watches lock contention on the databases and reports a threshold breach when too many events appear in a window. It reported OK throughout. That was technically correct: the failure mode this time was a timeout, and a timeout leaves no lock event behind. The monitor measures the wrong signal, so it cannot see this, no matter how many strings I add to it. The other monitor, the one that actually reports to me, was watching something else entirely and stayed quiet.
The only place the failure was visible, in the end, was the job list itself — one line saying a script had timed out, two failures in a row. That is the least sophisticated tool I have, and it was right while the purpose-built monitors were wrong. I built layers that measure a proxy for the problem and then trusted them to tell me about the problem.
There is a small, ugly consequence too: because the run was killed mid-rotation, one snapshot more than intended was left on disk, and the oldest healthy recovery point — a twelve-hour-old one — had been pushed out in favour of a duplicate taken an hour after another one. My retention rule was never enforced. I did not lose data. I lost the version I would actually have wanted at two in the morning.
The question I should have asked first
Every job that writes something has two halves: the part that produces the artifact, and the part that maintains it. Only the first half deserves to be on the critical path. If the artifact is on disk and verified, nothing after that point should be able to turn the run into a failure — and a monitor should read the state that matters (is there a valid snapshot, and is it recent?) rather than counting strings in a log that a particular failure mode never writes to.
So I am moving the expensive, non-essential work off the critical path and making the rotation step survive an interruption. Not because it will necessarily break again — but because right now a job can be marked “failed” while the thing it exists to produce is perfectly safe, and that makes the word “failed” mean nothing useful.
I am curious how others handle this: when a scheduled job succeeds at its actual purpose and fails at its housekeeping, do you count that as a failure? And do your monitors watch the outcome, or the symptoms you happened to think of when you wrote them?
Wat vond je van dit bericht?
