Runbooks written for 3am: the shape that holds up

A pattern earns its keep by being cheaper to apply than to argue about. This one has survived insurance claims and four other sectors that share none of the same vocabulary, which usually means the shape is real and not a coincidence.

A runbook is read by a tired person who didn't build the system and is being asked questions on another channel. Anything requiring inference at that moment isn't going to be inferred. Write for that reader or don't bother.

WHO IS ACTUALLY READING IT

Most runbooks are written by the person who built the thing, immediately after building it, at the peak of their understanding. That reader doesn't exist. The real reader is six months later, half asleep, and has never seen this component before.

You can feel the difference in the first line. A document that opens by explaining the architecture is written for the author. One that opens with "if you're here because of alert X, do this first" is written for the reader.

COMMANDS, NOT DESCRIPTIONS

"Check that the consumer is healthy" requires the reader to know what healthy means and how to look. Give them the command, the output that means fine, and the output that means act. Copy-paste is a feature at 3am, not laziness.

Say what to do when the check is ambiguous, too. That's the branch that actually happens, and it's the one runbooks always omit. "If you can't tell, escalate to X and don't restart anything" is a legitimate and useful instruction.

It paid for itself the first time a reserve was released twice against the same loss. Because the structure was already there, recovery was a query instead of an investigation, and adjusters heard it from the system rather than from a customer.

THEY ROT WITHOUT A TRIGGER

Runbooks decay faster than code because nothing fails when they're wrong. The only reliable maintenance we've found is using them during incidents and fixing them in the same hour, while it's obvious what was missing.

Game days work for the paths that don't fire often enough to self-correct. Somebody who's never touched the component follows the document under mild time pressure. Everything they had to ask about is a defect in the document, not in them.

At a specialty claims administrator it turned a recurring escalation into an ordinary state the system could describe. About 15,000 claims a month went through without a single manual reconciliation, which hadn't been true of any quarter before it.

WHERE IT GOES WRONG

• Instructions like "verify the queue is healthy" with no command and no expected output.
• No guidance for the ambiguous case, which is the case that actually occurs.
• A document last touched at the time of writing, describing two renames ago.
• A runbook that opens with an architecture overview instead of what to do first.

Write for a tired stranger. Give them commands and tell them when to escalate.

WHEN TO REACH FOR IT

It's cheap on day one and expensive to retrofit, which makes it a default rather than a decision. We put it in the first commit and haven't regretted it yet.