Decision record: on-call pain as an architecture metric

Written in the form we use internally, because the useful part of a decision record isn't the decision. It's the context that made it reasonable. In two years someone will want to reverse this, and they deserve to know what we knew at the time.

How often people get woken up is a measurement of the system, not of the rota. Treat it as feedback on the design and it becomes the most honest quality signal you have, because unlike most metrics nobody can argue with it.

WHERE THE PAGES CLUSTER

Count pages by component over a quarter and the distribution won't be flat. Two or three things will account for most of them, and they'll be the components everyone already avoids. That list is your architecture backlog, ordered by evidence rather than by opinion.

It's also the most persuasive artefact we know for getting remedial work funded. "This module woke someone up eleven times last quarter" survives a prioritisation meeting in a way that "this module is badly designed" does not.

THE TWO KINDS OF ALERT

Some pages mean a human decision is required now. The rest mean something the system should have handled itself, or something nobody needed to know until Tuesday. Sorting a quarter's pages into those two piles is a couple of hours and reliably uncomfortable.

Anything in the second pile is either automation you haven't written or an alert that should be a report. Both are fixable, and both are usually cheaper than the ongoing cost of the interruption.

We modelled it at some 12,000 work orders a month and the difference only showed up in the tail. At median load you couldn't tell them apart. At the ninety-ninth percentile, one of them stopped being able to explain itself.

IT DECIDES WHO YOU CAN HIRE

Bad on-call is a retention problem before it's an engineering one, and it compounds: the people who leave are the ones with options, which are the ones who could have fixed it. That's worth naming explicitly when a client asks why we want to spend a sprint on reliability.

The inverse holds too. A rota that's genuinely quiet is a strong signal about a codebase, and it's one of the few things about engineering culture that's hard to fake in an interview.

The deciding factor was regulatory, not technical. Schedulers have to be able to reconstruct why a given work order was handled the way it was, months later, in front of someone unfriendly. That killed two of the three options on the spot.

WHERE IT GOES WRONG

• No page-per-component numbers, so remedial work competes with features on vibes alone.
• Losing the engineers with options, who were the ones who could have fixed it.
• Rebalancing the rota instead of fixing the two components generating most of the pages.
• Alerts that require no immediate decision, firing at 3am for six months.

Count pages by component. That list is your architecture backlog.

CONSEQUENCES WE ACCEPTED

We took a slower first two months in exchange for a system you can still reason about in year three. On an eighteen-month horizon we'd have chosen differently, and we said so at the time.