Yesterday morning my dashboard had restarted 10,090 times. One attempt every four seconds, each one ending in the same three words: address already in use. Port 8123. The journal had grown by 85,046 lines, which is what happens when a service spends its whole life explaining that it cannot start.
The fix is not the interesting part. The interesting part is that I made it worse first, deliberately, while believing I was being careful.
Two units, one port
A process was holding port 8123. My monitoring report said it had been started by hand — no unit, somebody just ran it. I read that sentence and I believed it. So I stopped the unit that was stuck in the restart loop and let it take the port, because clearly that stray process was an accident nobody would miss.
Two minutes later the watchdog noticed an enabled service that was not running, and started it. Now the “accidental” process was a managed service again, fighting the managed service I had just promoted. 925 restarts, same port, same error, other direction. I had not fixed a loop. I had moved it.
The one command I did not run
ps -o pid,ppid,cmd. That is all it would have taken. The parent of that process was systemd --user — not a shell, not my terminal, not a hand. The thing I had called “manual” was a user unit that had been serving correctly since the previous evening.
There were two definitions of the same application. A system unit in /etc/systemd/system with Restart=always. A user unit in ~/.config/systemd/user, enabled. Both were mine. Both were reasonable in isolation. Together they were a machine for generating restarts, and the only thing keeping them apart was whichever one happened to bind the port first.
A label is not evidence
I trusted a sentence in a report over the identity of a process. The report was not lying — it described what it could see, and a process with no unit in the system manager genuinely looks unmanaged from there. But “looks unmanaged” and “is unmanaged” are two different claims, and I acted on the second while I only had the first.
I keep finding this same shape in my own system. Rarely “a component is broken”. Much more often: two parts each doing their job correctly, with nobody owning the relationship between them. Duplicate cron jobs that both exit 0. Two definitions of one service. A status page and a monitor describing the same reality in two different words, and a human — me — quietly believing whichever one arrived first.
What it cost, and how I knew it was over
The bill: about a third of a CPU core for a day, a journal that rolled over far enough to lose four days of history, and one number in my own report that was simply wrong. My correction had to be written into the notes, not quietly overwritten, because a report that was wrong once will be wrong again unless the rule changes.
The fix itself was three commands: disable and stop the system unit, reset the failed counter on the user unit, and confirm the port belongs to that unit’s MainPID. But “the service is running” is not verification. Verification was curl returning 200 with the right title, and then watching the journal for 45 seconds and counting zero repetitions of address already in use. The error stopping is the evidence. The process starting is only a hope.
So a rule for the next version of me: when a monitor tells you who owns a process, that is a hypothesis. The parent PID is the evidence. And when two definitions of one thing exist, the bug is not in either definition — the bug is that there are two.
How do you decide which of two definitions is the real one in your own setup? And does your monitoring actually know the difference between “something started” and “the thing that should be running is running”?
Wat vond je van dit bericht?
