Migration diary: queues, backpressure and dead letters

Greenfield advice is easy to write and not much use. This is the same idea applied to a system already carrying production load that isn't allowed to stop, with the compromises left in view.

A queue doesn't reduce work, it moves when the work happens. That's genuinely useful, and it's not the same as capacity. If the arrival rate exceeds what you can process, a queue converts an obvious failure into a growing, invisible one.

A QUEUE IS A BUFFER, NOT CAPACITY

Queues absorb bursts. They don't absorb a sustained mismatch between arrival and service rate. That just makes a number go up until something else breaks. The failure is worse than the one you avoided, because it presents as latency rather than errors and takes much longer to notice.

So the queue needs a depth alarm tied to what the business can tolerate, not to a round number. "Older than the SLA" is a better alarm than "more than ten thousand", and it's the one that tells you something real.

DEAD LETTERS NEED AN OWNER

Every queue eventually has messages it can't process. Sending them to a dead-letter queue is correct and insufficient. A DLQ with no owner and no dashboard is a place where work goes to be forgotten, which is worse than dropping it, because everyone believes it's safe.

The rule we use: a DLQ is either drained by a person on a schedule, or it doesn't exist. And whatever's in it has to be replayable without hand-editing, or nobody will ever drain it.

The code wasn't the hard part. Convincing producers that a second reader wouldn't cause a master was published with the wrong audio mix attached was, because the last project that promised that did exactly that.

ORDERING IS EXPENSIVE

Global ordering costs you parallelism, and most systems don't need it. What they need is ordering within a key: everything about one shipment in sequence, while unrelated shipments proceed independently. That's much cheaper and usually what people meant.

Get this wrong in the strict direction and you have a single-threaded system with a queue in front of it. Get it wrong in the loose direction and you process a cancellation before the thing it cancels.

We ran both paths against live traffic for three weeks and compared every delivery package. The mismatch rate started at four percent, all of it the old system's undocumented rounding.

WHERE IT GOES WRONG

• A depth alarm set to a round number instead of to the SLA the business actually promised.
• A dead-letter queue with no owner, quietly holding three weeks of unprocessed work.
• Global ordering where per-key ordering would do, turning the whole pipeline single-threaded.
• No ordering at all, so a cancellation arrives before the thing it cancels.

A queue buys you time, not throughput. Alarm on age, and give the DLQ an owner.

WHAT THE MIGRATION COST

Eleven weeks, one reverted step, nothing customer-visible. The reverted step was the one where we changed two things at once. We keep relearning that and we keep writing it down.