I Built a Fix-Before-Alert System — My AI Fixes Its Own Problems Now

I built a system that fixes its own problems. Before I even know about them. And honestly? It’s the most liberating thing I’ve done with my homelab this year.

It started with a frustration I think every homelab owner knows: the constant stream of alerts. Disk usage warnings. A container that crashed. A backup that failed. My Telegram channel was filling up with noise, and I was training myself to ignore it. That’s dangerous — when something actually critical happens, you’ve already learned to look the other way.

So I built the Pulse Protocol. It’s a process-to-Agent push communication system that runs on my Hermes Agent stack. Every process in my homelab — cron jobs, backup scripts, system monitors — sends a “pulse” to a central bridge. The bridge translates that pulse into an action code. And then something interesting happens: reflex handlers try to fix the problem automatically.

The Fix-Before-Alert Philosophy

Here’s the core idea: only bother me when the automated fix fails. I defined 10 action codes, but the most important one is ACTION_RECOVER — the bridge sees a problem, checks if a reflex exists, and runs it. If the reflex succeeds, the event is logged silently. No Telegram message. No interruption. John stays focused on what matters.

My reflex handlers include things like:

• Disk cleanup — when a disk passes 90%, it purges old logs and temp files
• Container restart — if a Docker container crashes, it restarts it (up to 3 times per day)
• Model fallback — if Ollama fails 5 times in 10 minutes, switch to a fallback provider
• Script fixer — if a Python cron job fails with a ModuleNotFoundError, automatically patch the wrapper
• Zombie reaper — clean up stuck curl processes that accumulate over time

Only when a reflex fails — or the same problem keeps happening — does the bridge escalate to ACTION_ALERT, and I get a clean, well-structured Telegram message. No noise. Just signal.

What I Learned Building This

The biggest insight was this: most alerting systems are designed for the wrong thing. They’re designed to tell you something is wrong. But what you actually need is a system that tells you something is wrong and I’ve already tried to fix it, here’s what I did and whether it worked.

The difference is subtle but enormous. My Telegram channel went from 25-40 messages per day to about 2-3. And the messages I do get are ones I actually need to act on. The system respects my attention instead of competing for it.

I’ve been running this for about two weeks now. The reflex handlers catch about 90% of issues before I ever see them. The remaining 10% are problems I genuinely need to look at — and because I’m not drowning in noise, I actually do.

How It Could Work for You

You don’t need a full Hermes Agent setup to try this. The core concept is simple: every process in your system sends a structured signal to a central point. That point checks if the problem is known and fixable. If yes, fix it silently. If no, escalate. The key is the reflex — a deterministic, no-LLM-required handler that runs instantly.

I’m curious: how do you handle alert fatigue in your own setup? Have you ever built automated recovery scripts, or do you prefer to be notified about everything? Let me know in the comments. I’d love to hear what other people are doing to tame the noise.