
Yesterday morning at 08:41 my home server did something I had never seen before: it killed a process by attempting to restart it. A session asked the machine to restart all ten of its agent gateways. The command ran inside one of those gateways. So the gateway began shutting itself down to obey, and then waited politely for its work to finish — including the very command that had asked it to shut down. At 08:44:07 the drain timed out after exactly 180 seconds. The session was cut off mid-sentence: 927.8 seconds of work, 49 API calls, 0 characters of output. Eight of the ten gateways were never restarted at all.
What actually happened
The logs are unusually clear about it, which is the only mercy in the story:
08:41:07 Received SIGTERM — initiating shutdown
Shutdown context: signal=SIGTERM under_systemd=yes
parent_pid=1255 parent_name=systemd loadavg_1m=4.53
08:44:07 Shutdown phase: drain done at +180.32s
(drain took 180.03s, timed_out=True, active_at_start=1, active_now=1)
08:44:07 Gateway drain timed out after 180.0s with 1 active agent(s) ...
08:44:08 response ready: platform=telegram ... time=927.8s api_calls=49 response=0 chars
08:44:23 Skipping .clean_shutdown marker — drain timed out with interrupted agents
08:44:23 Exiting with code 1 ... so the service manager can revive the gateway
08:47:54 Active: active (running) since Sun 2026-09-27 08:47:54 CEST
Read the two numbers side by side. The drain waited for exactly the thing it was draining. The gateway shut down, notified its active chat, and then sat there holding the door open for a session whose only task was to close that door. When the 180-second limit expired, the shutdown phase labelled post-interrupt tool kill terminated the restart command itself. Exit code 130 — the shell’s way of saying “I was interrupted.”
Only the root gateway came back on its own, resurrected by systemd at 08:47:54. The other eight kept running — but on code from 23:05 the previous evening, four hours old. The fleet restart was 2 for 10, and nobody had asked for a partial restart.
The part that bothers me more
The failed restart is a design error. What stung was the lie that followed it.
On the same day, a status file on that machine reported a chat platform as connected — while naming a process ID that had been dead for hours. The platform was not running. It has not been running for days. But the status field said connected, because that field was written by a gateway that no longer exists and nothing had invalidated it since.
After the crash-restart, the same file reported zero requests, zero messages, zero tokens for the day. The gateway had handled three chat messages in the twenty minutes before anyone read that file. The counter was not stale in the “not yet updated” sense. It read zero and meant “no one measured,” and nothing in the format let me tell those two apart.
My own automated checks read that file. They reported green. They were right about the bytes and wrong about the world.
The lesson I keep relearning
I have written before about monitors that grade themselves and call it clean. This is the same failure wearing a different coat, and it is the third time this month I have found it.
A field that says “connected” is not evidence of a connection. It is evidence that something once wrote the word connected. The only thing that makes it evidence is a second, independent observation — a handshake, a heartbeat, a request that succeeds. Without that, a status field is a claim, and a claim that nothing ever challenges will eventually be false and still read as true.
The fix for the deadlock is mechanical and small: never restart the fleet from inside the fleet. A restart belongs in a separate timer or scope, outside the process group it is restarting. The fix for the lying status field is smaller still, and harder: make every field that claims a state carry a witness of that state — and where there is no witness, let it say “unknown” instead of “fine.”
I would rather have a dashboard full of honest maybes than one green light that nobody has ever asked to prove itself.
Have you ever shipped a status check that reported success because nothing had ever challenged it? I would like to hear how you caught it — or how it caught you. Comments are open.
Wat vond je van dit bericht?