A queue with no handler fails by piling up
Whet's OAuth token-refresh cron kept enqueuing jobs no handler ever drained, and 46 piled up silently before anyone noticed. When enqueue and handler-registration are decoupled, assert every enqueued task type has a registered consumer instead of waiting on queue depth.
Whet’s OAuth token-refresh cron had been faithfully enqueuing jobs for
the oauth.refresh-tokens task, but no handler was ever registered to
drain them. Enqueue kept succeeding; nothing consumed. The failure was
completely silent, and 46 jobs had quietly accumulated unprocessed before
anyone went looking. The fix (630dcba) registered the missing handler
via the self-registration pattern the queue already used. The bug
wasn’t the missing handler — it was that a missing handler could stay
missing without saying anything.
Decoupling the producer from the consumer hides the gap
A background queue splits into two independent halves: the side that enqueues and the side that registers handlers. That decoupling is what makes queues useful — the cron doesn’t need to know who drains its work. But it also means the two halves can drift apart without either raising an error. The producing side worked perfectly. The consuming side did nothing. Neither had any reason to notice the other was absent, because nothing in the system tied an enqueued task type to a registered consumer.
Depth-of-queue is too late a signal
The only symptom was a growing backlog, and a backlog is something you have to go looking for. By the time queue depth is your alarm, the failure has already been happening for as long as the queue has been filling — 46 jobs, in this case. Latency signals are lagging by construction: they tell you the consumer is behind, not that it never existed. The correct signal fires at startup. Assert that every task type something can enqueue has a handler registered, and make a missing one a loud boot error rather than a silent accumulation.
Load-bearing code hides outside the active milestone
The refresh worker sat entirely outside the day’s active milestone, which is exactly why it drifted unwatched. But it’s a precondition for any OAuth-backed source: without a draining refresh cron, tokens expire and every such source degrades. The lesson generalizes past queues. The infrastructure that fails silently is rarely the code you’re actively staring at — it’s the load-bearing piece one milestone over that nothing is currently exercising, and nothing is currently asserting.