
When my AI gateway crashed at 2 AM, the system fixed itself
At 02:10 this morning, my Hermes gateway died. No clean shutdown, no graceful exit — just a heartbeat that stopped mid-beat and never resumed. I found out about it later, not from an alarm, but from the log. The interesting part isn’t that it crashed. It’s that 30 seconds later, a new gateway was up, every connection was back, and one session had quietly resumed where it left off. I didn’t have to lift a finger.
This isn’t luck. It’s the difference between building a system that runs and building a system that recovers. I learned that the hard way back in July, when a VM shutdown took everything down with it — in-memory state gone, cron jobs seemingly empty, a blog post that never got published because its temporary files vanished into RAM. That incident cost me a morning of head-scratching before I realized the gateway just needed two minutes to finish waking up.
What survives a crash?
So I mapped out exactly what lives on disk and what lives only in memory. The results shaped every decision since:
- Everything that matters lives on disk. All 60+ cron schedules are restored from a JSON file on startup. Conversations, memory, skills, config — all files on disk that survive any reboot.
- In-memory state is treated as disposable. Running sessions die, active connections drop. I stopped pretending otherwise and built around it.
- One simple rule: never write intermediate files to
/tmp/. That’s RAM, and RAM forgets. Every cron job now writes to a persistent state directory on disk.
That last rule is exactly what saved me today. When the gateway crashed and restarted, my cron jobs picked up their intermediate files from disk — nothing was lost, nothing had to be regenerated from scratch.
Even better: the backups fixed themselves
An hour after the restart, my backup scanner noticed seven wiki pages had gone missing from a phone sync gone wrong — and quietly restored all seven from the Synology backup. No prompt, no ticket, no “hey, you should look at this.” Just a log line saying it was handled.
That’s the moment I realized resilience isn’t one big feature. It’s a pile of small, boring habits: persistence on disk, restart-with-always, health checks that run on boot, and backups that don’t need me to notice they ran.
The lesson
You can’t prevent every crash. But you can make sure the system treats a crash as a small bump instead of a catastrophe. The metric that matters isn’t “does it go down?” — everything goes down eventually. It’s “how fast does it come back, and how much do I have to be involved?”
This morning: 30 seconds to recover, zero involvement from me. That’s what good system design feels like.
Do you know what actually survives a restart on your setup — or are you only finding out the hard way? I’d love to hear how you handle crash resilience in your own systems.
Wat vond je van dit bericht?