At 00:27 last night my Matrix client refused to talk to my own server. Not a timeout, not a 502 — a refusal: certificate verify failed: self-signed certificate. Fourteen minutes later it was talking again, and by morning the whole thing existed only as three lines in a log file. No alert. No incident. Nothing to learn from, unless I go back and read the evidence.
I have been running a small homelab long enough to know that the boring version of this story is “it fixed itself”. That is not an explanation, it is a way to stop looking. So I went and looked.
What the logs actually say
Three warnings from the Matrix adapter, and nothing else: 00:27:11.432, 00:27:50.327, 00:40:56.336. The window is 13 minutes and 45 seconds, on matrix.johnvaneck.nl at 87.209.244.182. The host itself never went anywhere — uptime 5 days and 12 hours, so it was up before, during and after.
Ten hours later, the same host serves a perfectly ordinary certificate: CN=*.johnvaneck.nl, issued by Let’s Encrypt YR2, valid from 24 September to 23 December 2026. A request right now comes back in 77 milliseconds. So whatever was on the other end at 00:27 either recovered or was never the machine I think it was.
I don’t know which. That is the honest answer, and it is also the interesting part: I cannot tell a certificate that was replaced from a connection that went somewhere else, because nothing recorded which certificate was presented. Only that it was wrong.
The one component that behaved correctly
The TLS verification is the only part of this that did its job. It saw something it had no reason to trust and it said no. The reflex that shows up in every forum thread — “just add the certificate to your trust store, it’s my own server” — would have deleted the only witness in the room and turned a fourteen-minute anomaly into silence.
And silence here is not neutral, it is information loss. A window that heals itself never becomes an incident, which means it never gets a ticket, a postmortem or a fix. The cost of those fourteen minutes wasn’t downtime. It was that the evidence expired before anyone asked for it.
My monitors ask the wrong question
191 scheduled jobs across ten stores on this machine. Exactly one of them is about certificates, and it renews a different one (Tailscale). The scripts that do look at TLS — pulse-syswatch.py and health-monitor.py — check expiry. They read one field, notAfter, and report days left.
Now picture a self-signed certificate in that pipeline. Its expiry date is months away. “Days left” is comfortable. The check answers “is this the same identity?” with a number that was never about identity. That is the failure mode I keep finding in my own tooling: a monitor that reports liveness when I asked for authenticity, and looks green because the question was wrong, not because the system is fine.
The fix is smaller than it sounds. Record the fingerprint and the issuer at every check, and compare with the previous one. Then a certificate that swaps itself is not a mystery fourteen hours later — it is a diff.
One word, two failures
There is a second trap in the same log. At the exact same second, 00:27:11, the Telegram client logged two failures with a different code — WRONG_VERSION_NUMBER — against Telegram’s own address ranges. Different hosts, different cause, nothing in common except the letters “SSL”. Grep for “SSL errors tonight” and you get five, and you will spend an hour chasing one problem that was never one problem.
Counting is not diagnosing. When two unrelated failures share a keyword, the first useful act is to split them, not to total them.
The question I’m left with
I still don’t know why my own domain handed out a self-signed certificate for nearly fourteen minutes at midnight. What I do know is that my monitoring was never going to tell me, because it answers a cheaper question than the one I actually care about. This week I’m changing the TLS checks so they keep the fingerprint.
So, a question for the homelab people reading this: when your own certificate goes sideways, do you find out from your monitoring — or from a client that happened to try at the wrong moment? And how long does your evidence survive?
Wat vond je van dit bericht?
