Notes from a regional freight brokerage: restores, not backups

The first thing we asked for wasn't a spec. It was thirty load tender records picked at random, exceptions left in. That sample told us more about the place than the requirements document did.

A backup is a claim. A restore is evidence. Everyone monitors the first and very few people practise the second, which is why the discovery that backups were incomplete almost always happens on the worst day.

THE GREEN TICK MEANS LESS THAN IT LOOKS

Backup jobs report that they finished, which is not the same as having produced something usable. We've found backups missing a schema added a year earlier, backups of a replica that had silently stopped replicating, and backups nobody had the key for. All three reported success daily.

The only check that means anything is a restore into a scratch environment with an assertion at the end: does this contain the row we wrote five minutes before the snapshot? Automate that and the green tick starts being worth something.

TIME THE RESTORE

Everyone has a recovery time objective. Very few have measured one. A restore that takes eleven hours is fine if that's the agreed number and a catastrophe if the business believed it was one. The gap is usually large and always discovered under pressure.

Measure it end to end, including the parts nobody counts: finding the right snapshot, provisioning somewhere to put it, replaying anything after the snapshot, and someone confirming it's correct. The database restore is often the quick bit.

The clearest thing we saw was how dispatchers handled a contested load tender. On paper it's one step. In practice it's five, three of them over the phone, none of them written down. Which is why nobody could ever explain tender acceptance rate to their director.

THE PARTIAL RESTORE IS THE COMMON ONE

Total loss is rare. What actually happens is one tenant's data corrupted by a bad migration, or one table truncated by a script. Restoring everything to fix that means throwing away everyone else's last six hours, which nobody will authorise.

So the capability worth having is selective: extract one tenant, or one table, from a snapshot into a place where you can inspect it before merging anything back. That path is worth building before you need it, because designing it during an incident is how second incidents happen.

Here it showed up as a queue nobody owned. Roughly 40,000 loads a month went through it, and dispatchers had learned to check it twice a day because the alternative was two carriers were committed against the same trailer. A better queue wasn't the answer. Making ownership a property of the load tender was.

WHERE IT GOES WRONG

• A backup of a replica that had silently stopped replicating three weeks earlier.
• An encrypted backup nobody could decrypt because the key rotated.
• A recovery time objective that was agreed but never measured, off by a factor of ten.
• Only whole-system restore available, when the actual incident affected one tenant.

You don't have backups. You have restores you've tested, and claims.

WHAT WE TOOK AWAY

The work shipped and tender acceptance rate moved, but the thing we're proudest of is smaller than the system: dispatchers stopped keeping a private spreadsheet. That's usually the honest signal that the model finally matches the job.