How student records teams end up putting a queue in front of the problem
Why putting a queue in front of the problem keeps looking like the sensible option, and what it costs by year two.
It's a well-meant mistake. That's exactly why it sticks around. Every team that makes it can explain why, and the explanation holds up right until the system has to survive its second year.
A queue doesn't reduce work, it moves when the work happens. That's genuinely useful, and it's not the same as capacity. If the arrival rate exceeds what you can process, a queue converts an obvious failure into a growing, invisible one.
A QUEUE IS A BUFFER, NOT CAPACITY
Queues absorb bursts. They don't absorb a sustained mismatch between arrival and service rate. That just makes a number go up until something else breaks. The failure is worse than the one you avoided, because it presents as latency rather than errors and takes much longer to notice.
So the queue needs a depth alarm tied to what the business can tolerate, not to a round number. "Older than the SLA" is a better alarm than "more than ten thousand", and it's the one that tells you something real.
DEAD LETTERS NEED AN OWNER
Every queue eventually has messages it can't process. Sending them to a dead-letter queue is correct and insufficient. A DLQ with no owner and no dashboard is a place where work goes to be forgotten, which is worse than dropping it, because everyone believes it's safe.
The rule we use: a DLQ is either drained by a person on a schedule, or it doesn't exist. And whatever's in it has to be replayable without hand-editing, or nobody will ever drain it.
This is where the reconciliation spreadsheet comes from. Somebody built it once to settle an argument. Now some 45,000 enrolments a year depends on a file with one author, no tests, and a filename ending in _v4_final.
ORDERING IS EXPENSIVE
Global ordering costs you parallelism, and most systems don't need it. What they need is ordering within a key: everything about one shipment in sequence, while unrelated shipments proceed independently. That's much cheaper and usually what people meant.
Get this wrong in the strict direction and you have a single-threaded system with a queue in front of it. Get it wrong in the loose direction and you process a cancellation before the thing it cancels.
The bill shows up in the handover notes: nine paragraphs on which of two systems to trust for a given enrolment record. When a withdrawal was backdated after the census had been filed, it took two days to work out what had actually happened, in what order.
WHERE IT GOES WRONG
- Global ordering where per-key ordering would do, turning the whole pipeline single-threaded.
- No ordering at all, so a cancellation arrives before the thing it cancels.
- A depth alarm set to a round number instead of to the SLA the business actually promised.
- A dead-letter queue with no owner, quietly holding three weeks of unprocessed work.
A queue buys you time, not throughput. Alarm on age, and give the DLQ an owner.
THE REPLACEMENT
The fix isn't more discipline. Discipline wears off. The fix is a structure where the wrong thing can't be said, so nobody has to remember the rule at 3am.