At 03:30 last night my genealogy pipeline ran, and at 03:30 my genealogy pipeline died. The strange part: the thing it was built to protect — the backup — came out perfectly. Ninety-three megabytes, compressed, readable, containing the live database. Everything the script was supposed to deliver was on disk. And yet the four steps after it never ran at all.
All because of one line of output that says nothing is wrong.
tar tells you “it worked, but something moved”
The script archives the data directory while the app is still writing to it. That is normal, and tar has an exit code for exactly that situation: 1 means “I finished, but a file changed while I was reading it.” It is a warning, not a failure. Exit code 2 and up is the real failure.
My script starts with set -euo pipefail, which is the right default for almost everything I write. But set -e does not distinguish between “warning” and “catastrophe”. It sees a non-zero exit code and stops. So the shell treated a warning as a verdict, killed the run on line 34 and never reached the pipeline: no index rebuild, no findings import, no dossiers, no GEDCOM export.
The nightly log ends after five lines: the database dump, then tar: .: file changed as we read it. No “Klaar”, no error message, no stack trace. Just a stop.
The expensive thing succeeded, the cheap things were skipped
That is the part that bothers me most. The backup — the slow, heavy, important step — worked. The four steps behind it, which take seconds and depend on nothing that went wrong, were silently dropped. I lost a night of fresh index, one thousand and nine dossiers and a GEDCOM export, not because of a data problem, but because a guard was too blunt to read a number.
And it is intermittent. I counted the night logs: one hit in thirty-two. That is the worst kind of bug — it does not reproduce on demand, so it hides behind “it worked yesterday”. My first synthetic repro even failed to trigger it; only the second attempt, with the timing right, showed the same exit 1 alongside a perfectly readable archive.
Interpret the exit code before the shell does
The fix is small and it is not a workaround. Turn errexit off around exactly that one call, capture the code yourself, and decide: two or higher means the archive is broken and gets deleted; one means warn and continue; zero means fine. Then read the archive back and require that the live database is actually inside it before you clean up the dump.
It ran at 08:14 this morning and passed: 1161 entries, the live database present, and the full pipeline behind it — 1009 dossiers, 15 branch indexes, a 220 KB GEDCOM with 915 people. The same data, five hours later, because a warning had been read as a failure.
The general rule I am keeping: a pipeline that calls an external tool must interpret that tool’s exit codes before set -e interprets them. And cleanup belongs in a trap, not on the line after the step that can abort — mine left an eight-megabyte database dump behind precisely because the line that removes it was never reached.
So here is what I want to know: how many of your automations treat “non-zero” as one single state? Because mine did, in a script I had read dozens of times, and it cost me a night without a single error message to show for it.
Wat vond je van dit bericht?
