Decision record: runbooks written for 3am

Written in the form we use internally, because the useful part of a decision record isn't the decision. It's the context that made it reasonable. In two years someone will want to reverse this, and they deserve to know what we knew at the time.

A runbook is read by a tired person who didn't build the system and is being asked questions on another channel. Anything requiring inference at that moment isn't going to be inferred. Write for that reader or don't bother.

WHO IS ACTUALLY READING IT

Most runbooks are written by the person who built the thing, immediately after building it, at the peak of their understanding. That reader doesn't exist. The real reader is six months later, half asleep, and has never seen this component before.

You can feel the difference in the first line. A document that opens by explaining the architecture is written for the author. One that opens with "if you're here because of alert X, do this first" is written for the reader.

COMMANDS, NOT DESCRIPTIONS

"Check that the consumer is healthy" requires the reader to know what healthy means and how to look. Give them the command, the output that means fine, and the output that means act. Copy-paste is a feature at 3am, not laziness.

Say what to do when the check is ambiguous, too. That's the branch that actually happens, and it's the one runbooks always omit. "If you can't tell, escalate to X and don't restart anything" is a legitimate and useful instruction.

We modelled it at about 30,000 units a week and the difference only showed up in the tail. At median load you couldn't tell them apart. At the ninety-ninth percentile, one of them stopped being able to explain itself.

THEY ROT WITHOUT A TRIGGER

Runbooks decay faster than code because nothing fails when they're wrong. The only reliable maintenance we've found is using them during incidents and fixing them in the same hour, while it's obvious what was missing.

Game days work for the paths that don't fire often enough to self-correct. Somebody who's never touched the component follows the document under mild time pressure. Everything they had to ask about is a defect in the document, not in them.

The deciding factor was regulatory, not technical. Production planners have to be able to reconstruct why a given work instruction was handled the way it was, months later, in front of someone unfriendly. That killed two of the three options on the spot.

WHERE IT GOES WRONG

• No guidance for the ambiguous case, which is the case that actually occurs.
• A document last touched at the time of writing, describing two renames ago.
• A runbook that opens with an architecture overview instead of what to do first.
• Instructions like "verify the queue is healthy" with no command and no expected output.

Write for a tired stranger. Give them commands and tell them when to escalate.

CONSEQUENCES WE ACCEPTED

We took a slower first two months in exchange for a system you can still reason about in year three. On an eighteen-month horizon we'd have chosen differently, and we said so at the time.