Post-mortem: incident reviews that change the system

The alert fired at a bad hour, as they do. The system wasn't down. That was the problem. It was up and confidently serving wrong answers, which is a lot worse than an outage because nobody comes looking.

Most incident reviews produce a document and no change. The difference between the ones that work and the ones that don't isn't blamelessness. It's whether any action item removes a decision rather than asking someone to be more careful.

THE TIMELINE IS THE WHOLE EXERCISE

Building an honest minute-by-minute timeline is where the learning is, and it's the part people rush. Almost always it shows that the technical fault was brief and the expensive part was the twenty minutes establishing what was happening, or the ninety minutes waiting for someone with access.

That's a much better finding than the root cause, because it's about how you respond to a whole class of problems rather than one bug you've now fixed and won't see again.

ACTIONS THAT SURVIVE

Two kinds of action item survive contact with the next quarter: ones that delete a possibility, and ones that shorten the feedback loop. "Add a constraint so this state can't exist" survives. "Be careful when editing this config" does not, and everyone in the room knows it while they're writing it down.

We now sort the list into those two categories before anyone gets assigned anything, and drop the rest. A shorter list that happens beats a thorough list that doesn't.

The trigger was boring. A timezone change silently duplicated an hour of consumption. The system had no way to represent that, so it picked one, and settlement analysts spent the next day proving it wrong with exported CSVs.

BLAMELESS IS A MEANS

The reason to run these without blame isn't kindness, it's data. People who expect consequences describe what they should have done rather than what they did, and then the timeline is fiction and the exercise is worthless.

It's worth stating that reason out loud, because "blameless" said without explanation sounds like a policy about feelings, and people rightly don't trust policies about feelings during a stressful review.

By the time anyone looked, several million reads a day had gone through the affected path. Only a slice of it was wrong, and we couldn't tell which slice without a full replay. The replay was the one thing we'd never tested.

WHERE IT GOES WRONG

• Action items that ask people to be careful, quietly abandoned within a quarter.
• A timeline assembled from memory a week later, describing what people wish had happened.
• "Blameless" declared without explaining why, so nobody believes it.
• A root cause identified and fixed, while the twenty-minute diagnosis delay goes unexamined.

Keep the actions that delete a possibility. Drop the ones that ask for more care.

WHAT CHANGED AFTERWARDS

Two action items survived: the two that removed a decision. Everything on the list that asked someone to be more careful was quietly dead within a quarter, which is roughly what we expected when we wrote it.