The Service That Died 16,921 Times
Last week, one of my services stopped working. I didn’t notice for two days. When I finally checked, I found something I’d never seen before: a systemd user service that had tried to restart 16,921 times before giving up. No alert. No notification. Just silence.
This is the story of Buzz-relay — a small WebSocket relay that lets my AI agents talk to each other — and what I learned about the “silent death” pattern in systemd user services.
The Numbers Tell the Story
When I ran systemctl --user status buzz-relay, the restart counter read 16,921. The service had been failing every few seconds since July 29 at 18:11. Each time, systemd dutifully tried again. Each time, it failed with status=216/GROUP — a systemd error code I had to look up.
Status 216 means systemd couldn’t determine the supplementary groups for the user running the service. It’s a permissions issue at the systemd level, not a bug in the service itself. The relay binary was fine — systemd just couldn’t launch it anymore.
After 16,921 attempts over roughly 48 hours, systemd did what it’s designed to do: it stopped trying. The service went from “auto-restart” to “inactive/dead” without a single notification.
The Silent Death Pattern
What bothered me most wasn’t the failed service — it was the silence. My monitoring stack (Uptime Kuma, Hermes health checks, cron monitors) all check for running services. But none of them check for services that stopped trying to run.
This is the “silent death” pattern: a service fails so many times that the process manager gives up, and nobody notices because the monitoring tools only look at the current state, not the restart history.
I realized I have at least a dozen systemd user services running on my homelab. How many of them are one bad config change away from the same fate?
What I Fixed
The fix was simple: edit the systemd unit and clear the SupplementaryGroups= directive that was causing the GROUP error. A one-line change. But finding the root cause took digging through systemd documentation and understanding a status code I’d never seen before.
I also added a new check to my monitoring: systemctl --user list-units --state=failed — a simple command that catches any service that has given up. It’s now part of my daily health check.
The Lesson
Your monitoring is only as good as the failure modes you’ve anticipated. I had alerts for “service is down” but not for “service tried and failed 16,000 times.” The difference matters — a lot.
If you’re running a homelab, check your systemd user services today. Run systemctl --user list-units --state=failed. You might be surprised what you find.
Have you ever had a service die silently on you? What did you learn from it?
Wat vond je van dit bericht?
