Post-mortem: restores, not backups

The alert fired at a bad hour, as they do. The system wasn't down. That was the problem. It was up and confidently serving wrong answers, which is a lot worse than an outage because nobody comes looking.

A backup is a claim. A restore is evidence. Everyone monitors the first and very few people practise the second, which is why the discovery that backups were incomplete almost always happens on the worst day.

THE GREEN TICK MEANS LESS THAN IT LOOKS

Backup jobs report that they finished, which is not the same as having produced something usable. We've found backups missing a schema added a year earlier, backups of a replica that had silently stopped replicating, and backups nobody had the key for. All three reported success daily.

The only check that means anything is a restore into a scratch environment with an assertion at the end: does this contain the row we wrote five minutes before the snapshot? Automate that and the green tick starts being worth something.

TIME THE RESTORE

Everyone has a recovery time objective. Very few have measured one. A restore that takes eleven hours is fine if that's the agreed number and a catastrophe if the business believed it was one. The gap is usually large and always discovered under pressure.

Measure it end to end, including the parts nobody counts: finding the right snapshot, provisioning somewhere to put it, replaying anything after the snapshot, and someone confirming it's correct. The database restore is often the quick bit.

The trigger was boring. A reserve was released twice against the same loss. The system had no way to represent that, so it picked one, and adjusters spent the next day proving it wrong with exported CSVs.

THE PARTIAL RESTORE IS THE COMMON ONE

Total loss is rare. What actually happens is one tenant's data corrupted by a bad migration, or one table truncated by a script. Restoring everything to fix that means throwing away everyone else's last six hours, which nobody will authorise.

So the capability worth having is selective: extract one tenant, or one table, from a snapshot into a place where you can inspect it before merging anything back. That path is worth building before you need it, because designing it during an incident is how second incidents happen.

By the time anyone looked, about 15,000 claims a month had gone through the affected path. Only a slice of it was wrong, and we couldn't tell which slice without a full replay. The replay was the one thing we'd never tested.

WHERE IT GOES WRONG

• An encrypted backup nobody could decrypt because the key rotated.
• A recovery time objective that was agreed but never measured, off by a factor of ten.
• Only whole-system restore available, when the actual incident affected one tenant.
• A backup of a replica that had silently stopped replicating three weeks earlier.

You don't have backups. You have restores you've tested, and claims.

WHAT CHANGED AFTERWARDS

Two action items survived: the two that removed a decision. Everything on the list that asked someone to be more careful was quietly dead within a quarter, which is roughly what we expected when we wrote it.