At 03:00 last night a cron job on my home server failed. Exit code 1, one error row, two critical pulses, and a line in the morning digest. Nothing was broken. The TLS certificate was fine, valid until 8 December, and there was nothing to renew. The job failed because two programs disagreed about the meaning of a single word: due.
The same word, measured with two rulers
My script, tailscale-cert-renew.py, does one simple thing: it reads the expiry date of a certificate, subtracts now, and asks whether it is time to renew. It keeps a constant, RENEW_DAYS = 30, and computes the remaining time as (expiry - now).days. That is the number of whole days — Python floors it. The check is if days > RENEW_DAYS: exit 0, so the first day the script considers itself due is the day days equals exactly 30.
Tailscale, the program that actually issues the certificate, has its own rule: it renews once fewer than 30 days remain — measured in real time, not in whole days.
This certificate runs from 9 September to 8 December. At 03:00 this morning the remaining time was 30 days and 6 hours 51 minutes. Floored, that is 30, so my script was due. In real time it is 30.28 days, which is not yet fewer than 30, so Tailscale said no and returned a perfectly healthy exit code 0 with the words “Public cert unchanged”.
A failure that is not an incident
That window is narrow but certain: in a 90-day cycle it opens exactly once, roughly 24 hours wide. For the rest of the cycle the two programs agree and nobody notices. Inside the window a system in perfect health reports itself broken — and every layer above it believes the report. The runner writes an error, the incident store raises an alert, the digest promotes it to “important”.
The uncomfortable part is what my code could not say. My renew() function knew two outcomes: renewed and failed. There was no word for “asked, and was correctly told not yet”. A healthy no-op had nowhere to go except the failure branch. The bug was not arithmetic. The bug was a missing noun.
Usually this kind of thing does not explode. It drifts: a job that is red every so often for no reason, an alert people learn to skip, a log line that once meant something. Quiet, repeatable, and invisible as long as you only look at whether the run said “ok”.
What the fix actually changed
The repair was to give that state a name. renew() now returns three outcomes: renewed, unchanged, failed. Unchanged is silent — no output, no alert — with exactly one exception: if seven days or fewer remain, the same answer means something is genuinely wrong and the job stays red. So “nothing to do” became a first-class result with its own rule, instead of being squeezed into “failed”.
There is a colder lesson in the paperwork. The morning note said this job had failed every night. The execution store held exactly one record: today’s. The mechanism predicts one night per cycle, and the record agreed with the mechanism, not with the prose. Two sources, two counts, one of them written by hand.
So the rule I keep returning to: at every boundary, name which system decides, in what unit, and what “nothing to do” means. A boundary written as a bare number, with no unit and no owner, is not a boundary. It is a hope — and it fails quietly, on the one day per cycle when the two sides read it differently.
Where in your own stack have you written a number that another system reads in a different unit? I would like to hear about the ones you only found afterwards. Comments are open.
Wat vond je van dit bericht?
