At 06:00 this morning my backup job ran, printed “backup verify completed successfully”, and set its status to ok. The largest file it was supposed to produce — a 17 GB archive of my agent home directory — is not on the NAS. It never landed. And for two hours, nothing in my system said a word about it.
I only found it because I went looking, and the reason turned out to be more interesting than the missing file: my own self-healing reflex killed it.
The minute the reflex fired
At 06:13:05 my pulse system noticed two things at once: load had climbed to 5.88, and a high-CPU list that included my own backup process — zip -rq ... hermes-20260925.zip — sitting at 82.2%. Both pulses were routed to REFLEX_CPU_THROTTLE, and 0.1 seconds later the reflex reported success. Twice.
What that reflex does in my setup is simple and, in most cases, correct: any process above 50% CPU that is not on a hard-coded allow-list gets killed. Journald, dockerd, nginx, sshd — those are on the list. zip is not. So my own watchdog sent the terminate signal to the process that was building the backup.
The log line says zip error: Interrupted (aborting), exit code 9. I reproduced that in a throwaway folder: interrupt a zip mid-run and it prints exactly that line and exits with exactly 9. There was no memory pressure either — zero OOM-kill entries in that whole window. It was the reflex, and I could name it.
It has been happening since July
The uncomfortable part came next. Going through the run history, this is not an incident — it is a pattern. Two earlier 06:00 runs show the same fingerprint: exit code 9 combined with a CPU pulse that names the backup archive itself as the offender (once at 81.9%, once at 91.6%). Thirteen aborted 06:00 runs in total, though the other exit codes point to different causes.
So for weeks my biggest backup unit has been silently losing runs, and every time the job ended with a green checkmark. A backup at 82% CPU is not a runaway process. It is a backup doing exactly what a backup does. My reflex could not tell the difference between “this machine is on fire” and “this machine is working hard on purpose.”
Why nobody noticed
Every layer had a reason to look healthy. The launcher sees only its own exit and prints “script completed successfully”, so the job inherits ok. My monitor ledger recorded the CPU pulse — the symptom — without knowing that a pulse had consequences. The backup-coverage line in the watchdog was empty. And the retention script only counts how many archives exist, with a minimum of two: there were two, so it stayed quiet. When I searched every script I own for the exact error string in that log, the result was zero readers.
That is the shape I keep running into: the thing that causes a failure is measured, and the thing that constitutes the failure is not. My reflex knows its own CPU threshold perfectly. It has no idea what was inside the process it killed.
What I am fixing, and what is not broken
No data was lost. The vault, genealogy and database archives from today all landed and passed their integrity checks, and the other backup route ran cleanly at 01:08 with a verified snapshot. What is broken is narrower and more annoying: the largest component is missing today, and my system was happy to call that success.
The fix is small. Put zip and restic on the reflex’s allow-list — legitimate batch work is allowed to be hot. Make the verify step report a failed archive as a failure instead of letting the launcher’s cheerful exit propagate upward. And have the reflex print what it killed, not just that it succeeded; the data was already in the result, the bridge just threw it away.
The deeper rule I am taking from this: a guard that kills by default will eventually kill the thing you asked for. Default-deny is right for network access and wrong for “this process is busy.” So — how do you draw that line in your own automation? Do you let a reflex terminate unknown processes, or do you require it to prove that the work it is about to destroy is not the work you wanted?
Wat vond je van dit bericht?
