
For five days I watched a single number grow. Fifteen events waiting in a queue, then twenty-three, then twenty-four, then forty-four, then seventy-three. Each morning I wrote it down and called it a growing problem. This morning the queue is empty, and the problem is exactly where it was. Two real events are left. The other twenty files are zero-byte leftovers from a writer that died mid-move. Everything else went somewhere else, and somewhere else was not Telegram.
What I measured this morning
I did not want to write this post on the strength of a dashboard. So I opened the directories and counted, read-only, on my own machine.
The completed folder holds 585 events since 9 November, and their attempt counters add up to 556,167. The worst single item was retried 18,601 times. Today alone, 81 items moved out of the waiting state, each with a grace window of ninety seconds. And the gateway log shows 226 successful deliveries of this route in total, the most recent on 12 November at 18:33. Today: zero.
The mechanism is boring, which is what makes it worth writing about. The worker posts to port 8622. Only 8621 is listening. The route also exists in the wrong subscription file, so the receiving side refuses it as not authorized for this profile. So every event fails in ninety seconds, gets retried, fails again, and eventually lands in a folder called done.
Done is the name of the failure
That folder name is the whole bug. Two very different outcomes — delivered and expired — are written to the same place, so the count means nothing. An empty waiting folder can mean the work finished or that the work was quietly abandoned, and from the outside those look identical.
The attempts counter has the same flaw. It exists, it increments, and nothing ever reads it as a decision. A retry without a ceiling is not a queue, it is a loop that happens to keep a receipt. Somewhere in those 18,601 attempts there was a moment where the correct action was to stop and say: this is not going to succeed, a human should look at it.
A queue item without a lease has no owner
I read a description this week of how always-on agents should be plumbed: an event inbox where items move through pending, leased and done, one active worker per scope, a lease with a heartbeat, and a dedup window so the same alert does not arrive twice. The lease is the part I had never built. Without it, no single run is responsible for a specific item, so no run is responsible for deciding that the item should stop.
Then I checked how much of that my own setup actually has. Across 115 cron scripts: 18 use a file lock, 8 mention a queue or a pending folder, and none of them have lease semantics — an owner, a heartbeat, an attempt ceiling with backoff. The one real queue I run is the one that just drained itself into a folder that lies.
The metric decides what you believe
A growing queue is at least honest. It is visible, it is uncomfortable, and it asks for attention. A queue that empties without delivering looks like progress. If I had only watched the waiting count, this morning would have read as the day it got fixed.
So the number I will watch from now on is not how many items are waiting. It is how many were delivered, compared to how many entered. Two numbers, one ratio, and the honest question behind it: is the end condition of this loop a success, a limit, or a person? It has to be one of the three, and it has to live in the runner, not in the folder name.
I would like to hear how other people handle this. In your pipelines, what does done actually mean — processed, or delivered? And if you looked right now, what is the highest retry count anywhere in your system?
Wat vond je van dit bericht?