Reading a freight logistics system through runbooks written for 3am

We inherit systems more often than we start them. This is how we read one: what we look at first, and what each thing tells us about decisions made by people we'll never meet.

A runbook is read by a tired person who didn't build the system and is being asked questions on another channel. Anything requiring inference at that moment isn't going to be inferred. Write for that reader or don't bother.

WHO IS ACTUALLY READING IT

Most runbooks are written by the person who built the thing, immediately after building it, at the peak of their understanding. That reader doesn't exist. The real reader is six months later, half asleep, and has never seen this component before.

You can feel the difference in the first line. A document that opens by explaining the architecture is written for the author. One that opens with "if you're here because of alert X, do this first" is written for the reader.

COMMANDS, NOT DESCRIPTIONS

"Check that the consumer is healthy" requires the reader to know what healthy means and how to look. Give them the command, the output that means fine, and the output that means act. Copy-paste is a feature at 3am, not laziness.

Say what to do when the check is ambiguous, too. That's the branch that actually happens, and it's the one runbooks always omit. "If you can't tell, escalate to X and don't restart anything" is a legitimate and useful instruction.

The exception handlers were the most honest documentation in the repo. Every branch that swallowed an error marked a spot where dispatchers had once been paged and somebody had made it stop.

THEY ROT WITHOUT A TRIGGER

Runbooks decay faster than code because nothing fails when they're wrong. The only reliable maintenance we've found is using them during incidents and fixing them in the same hour, while it's obvious what was missing.

Game days work for the paths that don't fire often enough to self-correct. Somebody who's never touched the component follows the document under mild time pressure. Everything they had to ask about is a defect in the document, not in them.

The schema told the real story. Three nullable columns added in one migration, all named after a load tender field, all populated for precisely the weeks when two carriers were committed against the same trailer was an open ticket.

WHERE IT GOES WRONG

• A runbook that opens with an architecture overview instead of what to do first.
• Instructions like "verify the queue is healthy" with no command and no expected output.
• No guidance for the ambiguous case, which is the case that actually occurs.
• A document last touched at the time of writing, describing two renames ago.

Write for a tired stranger. Give them commands and tell them when to escalate.

THE VERDICT

Better than its reputation. Most of what looked like bad engineering turned out to be a sensible answer to a constraint that had since been lifted, and nobody had gone back to check.