Designing the approval step, from first principles
Designing the approval step from the start, for a room that contains both engineers and the people who'll live with it.
Most write-ups of this start in the middle. We'll start earlier than strictly necessary, because nine times in ten the confusion comes from a word two people are using differently, not from the mechanism.
"A human reviews it" is where automation designs go to die. An approval step with no thought behind it becomes a rubber stamp within a month, and then you have the cost of a review with none of the protection.
RUBBER STAMPS FORM FAST
Give someone a hundred items a day where ninety-eight are fine and they will approve all hundred by Thursday. That's not carelessness, it's a rational response to a signal with no information in it. The design is at fault, not the reviewer.
So the honest question isn't whether to have review, it's what fraction the reviewer can meaningfully attend to. If the system can't identify the interesting ten percent, it's asking a person to do a job it hasn't equipped them for.
REVIEW THE UNCERTAIN, NOT THE SAMPLE
Random sampling feels rigorous and finds nothing. What works is routing by uncertainty: send the cases where the system's confidence is low, where the amount is unusual, where inputs disagree. Then a queue of thirty items genuinely contains thirty decisions.
This needs the system to have a calibrated notion of its own uncertainty, which is real work and the actual engineering in most of these projects. Skip it and you're back to sampling.
Made concrete: a meter read passes through several pairs of hands, and each of them believes it owns the record. What's above is how you let all of them be right without letting any of them silently overwrite the others.
MAKE THE REVIEW TEACH THE SYSTEM
A reviewer who overturns a decision knows something the system doesn't. If that only produces a corrected record, you've thrown the valuable part away. Capture the reason as structured data and the exception rate should fall over time.
It also gives you a measurement worth watching: overturn rate by category. Rising means something's drifted. Falling to zero means you can widen what gets handled automatically. Unlike most automation metrics, both directions are actionable.
Picture a distributed generation operator at several million reads a day. The naive version works perfectly in testing and falls over the first day two settlement analysts touch the same meter read in the same second.
WHERE IT GOES WRONG
- Random sampling instead of routing by uncertainty, so review finds nothing.
- Overturns recorded as corrections with no reason, throwing away the only useful signal.
- An approval step added to satisfy an audit requirement, with nobody measuring whether it does anything.
- A hundred items a day where ninety-eight are fine, approved in a batch by Thursday.
Route by uncertainty, and capture why the reviewer disagreed.
THE SHORT VERSION
None of this is advanced. It's ordinary practice that gets skipped when a date is close, and then paid for with interest by whoever's on call.