Home / Vault & Second Brain / Every Signal I Own Says Now

Every Signal I Own Says Now

A single small beacon on a dark plain at night

For two days I have been writing about measurements that lie. First a number that said OK and was written by the wrong process, then a queue that empties itself into a folder called done. This morning I pointed the same question at the thing that is supposed to catch all of it: my monitoring.

Here is what my monitoring looks like from the inside. I checked three heartbeats — the cron ticker, the gateway, the dashboard clients. All three exist. All three were written within the last ten minutes. Two years ago I would have called that healthy. This morning it only told me that the writers were alive, which is a much smaller claim than I thought I was making.

What I measured this morning

So I went looking for the one thing that would still speak if this machine stopped speaking: an external watcher.

The uptime monitor I installed last year is gone. The script that used to talk to it says so in its own comment: removed on 13 October, deliberately phased out. I checked anyway. Connection refused, exit code 7. Its replacement is alive — the check server answers on port 8000, a 404 on the API path I tried, but a server that answers is a server. Alive is not the same as connected, and connected is not the same as “would notice”.

Then I searched every script I own for a dead man’s switch: healthchecks, hc-ping, deadman. Zero hits. Not “none configured” — zero occurrences of the idea anywhere in 639 Python files.

Meanwhile 103 of those scripts have monitor, watchdog or check in their name. I sorted them by what they actually look at. Twenty-three look at age or silence. Fourteen look at an error or an exit code. Fifty-eight look at both. That sounds thorough, and it is, for one specific question: is something broken right now? Not one of them asks the only question that survives the machine dying: did the thing I expected at 08:30 actually arrive?


Silence is not a status

I made silence a design goal. My own standing rule is that a monitor stays quiet when everything is fine and only speaks when something deviates. That is a good rule for a human inbox. For a monitor it is catastrophic, because it puts healthy and dead on the same side of the measurement: both produce no message. The evidence that distinguishes them is not in the message. It is in the absence of one, and absence cannot be read from inside the system that produces it.

A post on r/selfhosted this week made the point better than I can. One Proxmox host died for a week and nobody noticed, because a second Pi-hole took over DNS and the nightly replication kept succeeding. Every signal stayed green by continuing to be absent. The sharpest comment under it was not about backups: your DNS redundancy worked, I would not say your backup plan was good.

What I actually want

Not more monitors. One line: a watcher outside this machine that expects a ping from me every thirty minutes and, when it does not arrive, tells me — from a box that shares no power supply, no disk and no kernel with this one. The alarm is the absence. On a healthy day I receive nothing. On a dead day my phone rings. Cost: one cron entry and one HTTP request.

The hard part is not building it. The hard part is admitting that “no errors” was never a measurement — it was a description of my own blind spot. Everything I built until now answers the question “is it broken?”. Nothing answers “is it still there?”

So here is the question I am sitting with, and I would genuinely like your answer: if your machine died tonight and stayed dead until Friday, what would tell you? Not what would fail — what would tell you.

Wat vond je van dit bericht?

Leave a Reply

Your email address will not be published. Required fields are marked *