What restores, not backups actually costs

The question is never whether something is worth doing in the abstract. It's whether it's worth doing at your volume, with your failure rate, valuing your team's time properly. So let's do the sum.

A backup is a claim. A restore is evidence. Everyone monitors the first and very few people practise the second, which is why the discovery that backups were incomplete almost always happens on the worst day.

THE GREEN TICK MEANS LESS THAN IT LOOKS

Backup jobs report that they finished, which is not the same as having produced something usable. We've found backups missing a schema added a year earlier, backups of a replica that had silently stopped replicating, and backups nobody had the key for. All three reported success daily.

The only check that means anything is a restore into a scratch environment with an assertion at the end: does this contain the row we wrote five minutes before the snapshot? Automate that and the green tick starts being worth something.

TIME THE RESTORE

Everyone has a recovery time objective. Very few have measured one. A restore that takes eleven hours is fine if that's the agreed number and a catastrophe if the business believed it was one. The gap is usually large and always discovered under pressure.

Measure it end to end, including the parts nobody counts: finding the right snapshot, provisioning somewhere to put it, replaying anything after the snapshot, and someone confirming it's correct. The database restore is often the quick bit.

At around 2,500 applications a month with a two percent exception rate, underwriters were absorbing about nine hours of manual reconciliation a week. That's the number the build had to beat, and it's a lower bar than anyone in the room expected.

THE PARTIAL RESTORE IS THE COMMON ONE

Total loss is rare. What actually happens is one tenant's data corrupted by a bad migration, or one table truncated by a script. Restoring everything to fix that means throwing away everyone else's last six hours, which nobody will authorise.

So the capability worth having is selective: extract one tenant, or one table, from a snapshot into a place where you can inspect it before merging anything back. That path is worth building before you need it, because designing it during an incident is how second incidents happen.

The interesting term wasn't engineering time. It was the cost of an approval was issued against a stale valuation landing once in the wrong quarter, which the client could size to the pound and we couldn't size at all.

WHERE IT GOES WRONG

• Only whole-system restore available, when the actual incident affected one tenant.
• A backup of a replica that had silently stopped replicating three weeks earlier.
• An encrypted backup nobody could decrypt because the key rotated.
• A recovery time objective that was agreed but never measured, off by a factor of ten.

You don't have backups. You have restores you've tested, and claims.

WHERE THE MODEL BREAKS

Do the numbers before the meeting, not during it. A decision that survives arithmetic tends to survive the next reorg too, because the reasoning outlives the people who made it.