This week I sat down to measure one backup job on my own machine, and it took me four hours to admit something simple: the number I had been reading as “OK” was written by the wrong process. My backup launcher printed “started — running in the background”, returned exit code 0, and the scheduler recorded a healthy run. The actual backup had other plans. It ran for 885 seconds, and the launcher had no idea.
A status is a receipt, not a summary
The launcher was not lying. It did exactly what it was built to do a while back: start the backup and get out of the way, because the cron timeout was 180 seconds and a backup takes about half an hour. The problem is what happens afterwards. The process that starts the work is not the process that knows how the work ended, yet it was the only one writing a verdict. Exit 0 from a starter means “I handed it off”. It does not mean “the backup exists”.
I had already written this down as a failure class once, and still had to measure it again to believe it: a command that exits 3 looks exactly like one that exits 0 when the status field only ever carries the wrapper’s opinion. The fix that landed on 9 November is the right shape. The child process now writes its own last-run.json with its own exit code, its own state, and its own PID via BASHPID — not the launcher’s PID pretending. The receipt comes from the producer.
“Background” is a promise, not a property
The second thing the measurement showed was worse, because it had been invisible for months. The launcher says “running in the background”, but the child inherits the stdout pipe. The scheduler keeps reading until end-of-file, so it waited the full 885 seconds — 06:00:54 to 06:15:39 — exactly until the child finished. The entire reason the launcher exists, escaping a 180-second timeout, was never solved. It was only hidden. And because the job record now had timeout: None, nothing intervenes anymore either.
Real detachment is not an ampersand. It is setsid plus redirecting stdout and stderr to /dev/null. Otherwise “background” is a sentence in a log line, and the process tree quietly disagrees with it. 885 seconds is the proof I needed.
A correct receipt nobody reads is still a guess
Here is the part that stings. I searched for last-run.json across my scripts and cron directory. Two hits: the launcher itself and its own selftest. No monitor, no watchdog, no cron job. So the artifact is now written by the process that did the work — and the meter that decides whether I sleep well still reads the launcher’s exit code, one day late, with nobody looking at the file that actually knows.
And one layer deeper, the selftest caught something I would have missed. Six counter-tests, five green. The lock case: the child holds the lock, the ZIP is 50 hours old, and the launcher returns 0 — “skipped, therefore fine”. A hanging process that never releases the lock would report an infinitely healthy backup while nothing new is written. Same silent assumption, one level down.
So the rule I am keeping from this: the process that does the work writes its own outcome (exit code plus artifact), the meter reads that file and nothing else, and every status field gets an expiry date — because two numbers that both say “ok” but come from different producers are not a measurement, they are an assumption.
Which raises the question I cannot answer alone: when your monitoring says all green, do you know which process wrote that green — the one that did the work, or the one that merely started it?
Wat vond je van dit bericht?
