How the arithmetic of automation broke a laboratory sample tracking system

The alert fired at a bad hour, as they do. The system wasn't down. That was the problem. It was up and confidently serving wrong answers, which is a lot worse than an outage because nobody comes looking.

Whether something's worth automating is a sum, and the sum is usually not close once you do it honestly. The mistake is doing it on the happy path and ignoring the ongoing cost of the exceptions.

THE TERMS PEOPLE LEAVE OUT

The build cost gets estimated. What gets missed is the exception handling, which is most of the code, and the ongoing maintenance as the surrounding systems change underneath you. An integration is not a one-off cost, it's a subscription.

The other omission is the cost of automation being wrong at volume. A person processing a hundred items makes a handful of individual errors. A rule processing a hundred thousand makes the same error a hundred thousand times, and the cleanup is a project.

PARTIAL AUTOMATION IS USUALLY THE ANSWER

Handling the eighty percent that's unambiguous and routing the rest to a person is often several times better value than chasing full coverage. The last twenty percent contains all the ambiguity, so it's most of the work and most of the risk.

It's a harder thing to sell, because "we automated eighty percent" sounds like a partial success. It's usually the whole win, and the remaining twenty percent is where people should be spending their attention anyway.

The trigger was boring. A batch was reported before its confirmatory run had finished. The system had no way to represent that, so it picked one, and lab managers spent the next day proving it wrong with exported CSVs.

THE NUMBER THAT ACTUALLY MATTERS

Hours saved is the metric everyone reaches for and it's rarely the one that justifies the work. More often it's a reduction in variance: fewer things going wrong, fewer statutory deadlines missed, fewer disputes. That's the number to put in the business case.

Ask what happens today when it goes wrong, and how often. Frequency times consequence is usually a bigger number than the labour saving, and unlike the labour saving it's a number the client already believes.

By the time anyone looked, around 20,000 samples a month had gone through the affected path. Only a slice of it was wrong, and we couldn't tell which slice without a full replay. The replay was the one thing we'd never tested.

WHERE IT GOES WRONG

• Justifying the work on hours saved, when the real return is fewer things going wrong.
• Costing the happy path, when exception handling is most of the code.
• Treating an integration as a one-off build rather than an ongoing subscription.
• Chasing the last twenty percent, which holds all the ambiguity and most of the risk.

Do the sum with exceptions included. Then automate the eighty percent and route the rest.

WHAT CHANGED AFTERWARDS

Two action items survived: the two that removed a decision. Everything on the list that asked someone to be more careful was quietly dead within a quarter, which is roughly what we expected when we wrote it.