The Error That Wasn’t a Bug in My Code

The Error That Wasn’t a Bug in My Code

On Thursday morning my Telegram lit up with the same message, over and over: “I encountered an unexpected error.” Not from a user — from my own agent gateway. Every message I sent came back broken. When I dug into the logs, the error was blunt and final: sqlite3.DatabaseError: file is not a database. My session database, the file that holds 637,000 messages across 21,000 conversations, had stopped being a database.

What followed was a debugging session that taught me something I should have known years ago: the most dangerous thing you can do to a SQLite database is kill the process that’s writing to it.

The Chain of Events

I traced it back step by step. At 08:34, an update pipeline stopped the gateway with a clean SIGTERM. The gateway started shutting down properly — it has a lot of MCP servers and active sessions, so it needs time. But my systemd config only gave it 210 seconds. At 08:38, the shutdown ran long, and systemd did what systemd does: it sent a SIGKILL to the entire process tree. Thirty-plus processes, gone in an instant.

The SIGKILL landed in the middle of a WAL write. SQLite’s write-ahead log was mid-checkpoint, and the hard kill corrupted the database header. When the gateway restarted, its own integrity check caught it: “state.db FAILED integrity check after an unclean gateway exit.”

And here’s the part that stung: this wasn’t the first time. It had happened the day before too. My own restart scripts were calling systemctl kill -s SIGKILL as their first action — not as a last resort. I had been doing the dangerous thing to my own database, repeatedly, and then being surprised when it broke.

The Fix Was Boring, and That’s the Point

No clever new tool was needed. Two changes, both boring:

• Give the gateway time to die gracefully. I raised TimeoutStopSec from 210 to 300 seconds, so SIGTERM gets a chance to checkpoint the WAL before anything is forced.
• SIGTERM first, SIGKILL only as a fallback. I rewrote three restart scripts so they stop the service cleanly and wait 45–60 seconds before ever considering a hard kill.

The recovery itself was almost anticlimactic. PRAGMA integrity_check came back clean — no data loss. The real cleanup was deleting three byte-identical copies of the corrupted state, each 3.4 GB, that my recovery process had helpfully made. Thirteen gigabytes of garbage, gone.

The deeper lesson is one I keep relearning in different forms: a documented fix is not a fix. I had written down the right way to restart a gateway before. The scripts still did the wrong thing. The only thing that actually changed the behavior was verifying that the code on disk matched what I thought it said.

What I’d Tell Anyone Running SQLite in Production

If you run SQLite with WAL mode — and you probably do — treat SIGKILL as a last resort, never a first move. Always send SIGTERM and wait for a clean checkpoint. Make your shutdown timeout longer than your actual shutdown time, because that time grows as you add more services. And when you see “file is not a database,” check the current file’s integrity before you trust any backup — the copies your recovery process makes are often byte-identical copies of the corrupted state, not usable backups.

Have you ever killed a process and watched a database quietly corrupt itself? I’d love to hear how you recovered — the comments are open.