Decision record: designing the degraded mode first

Written in the form we use internally, because the useful part of a decision record isn't the decision. It's the context that made it reasonable. In two years someone will want to reverse this, and they deserve to know what we knew at the time.

Every dependency will be unavailable at some point. What your system does during that window is a design decision whether or not anyone makes it deliberately. Making it deliberately is most of the work.

THE DEFAULT IS USUALLY THE WORST OPTION

Left alone, an unavailable dependency produces a stack trace in front of a user and a lost piece of work. That's rarely the best available behaviour. Often you could accept the input, tell the truth about what's pending, and reconcile later, but only if someone decided that in advance.

The decision has to be per-dependency, because they aren't equivalent. Losing the address validator should degrade to accepting the address unvalidated. Losing the payment authoriser should stop the transaction. Same outage shape, opposite correct answers.

TELL THE TRUTH ON THE SCREEN

Degraded modes fail on honesty more than on engineering. A system that silently drops the enrichment step and shows an incomplete record as though it were complete has turned a visible outage into a data quality problem, which is much harder to find later.

So the degraded state needs to be visible in the record, not just in the logs. "Pending verification" is a real state that belongs in the model. It's also the thing that lets someone clean up afterwards, because they can find the affected rows.

We modelled it at some 45,000 enrolments a year and the difference only showed up in the tail. At median load you couldn't tell them apart. At the ninety-ninth percentile, one of them stopped being able to explain itself.

PRACTISE IT

An untested fallback is a guess. Turn the dependency off in a staging environment that has real traffic shape and watch what actually happens. It's routinely not what the code suggests, usually because of a timeout somewhere nobody configured.

The failures that surprise us most are almost always timeout-related. A dependency that's down is easy. A dependency that's slow holds every connection until the whole system is unavailable for reasons that look unrelated.

The deciding factor was regulatory, not technical. Registrars have to be able to reconstruct why a given enrolment record was handled the way it was, months later, in front of someone unfriendly. That killed two of the three options on the spot.

WHERE IT GOES WRONG

• A fallback path that's never been exercised and doesn't work when it's needed.
• No timeout on a dependency, so slow is worse than down.
• The same failure policy for the address validator and the payment authoriser.
• Silently skipping enrichment, so incomplete records look complete and can't be found later.

Decide what happens when each dependency is gone. Then make the degraded state visible.

CONSEQUENCES WE ACCEPTED

We took a slower first two months in exchange for a system you can still reason about in year three. On an eighteen-month horizon we'd have chosen differently, and we said so at the time.