Notes from a post-production house: designing the degraded mode first

The first thing we asked for wasn't a spec. It was thirty delivery package records picked at random, exceptions left in. That sample told us more about the place than the requirements document did.

Every dependency will be unavailable at some point. What your system does during that window is a design decision whether or not anyone makes it deliberately. Making it deliberately is most of the work.

THE DEFAULT IS USUALLY THE WORST OPTION

Left alone, an unavailable dependency produces a stack trace in front of a user and a lost piece of work. That's rarely the best available behaviour. Often you could accept the input, tell the truth about what's pending, and reconcile later, but only if someone decided that in advance.

The decision has to be per-dependency, because they aren't equivalent. Losing the address validator should degrade to accepting the address unvalidated. Losing the payment authoriser should stop the transaction. Same outage shape, opposite correct answers.

TELL THE TRUTH ON THE SCREEN

Degraded modes fail on honesty more than on engineering. A system that silently drops the enrichment step and shows an incomplete record as though it were complete has turned a visible outage into a data quality problem, which is much harder to find later.

So the degraded state needs to be visible in the record, not just in the logs. "Pending verification" is a real state that belongs in the model. It's also the thing that lets someone clean up afterwards, because they can find the affected rows.

The clearest thing we saw was how producers handled a contested delivery package. On paper it's one step. In practice it's five, three of them over the phone, none of them written down. Which is why nobody could ever explain on-time delivery to their director.

PRACTISE IT

An untested fallback is a guess. Turn the dependency off in a staging environment that has real traffic shape and watch what actually happens. It's routinely not what the code suggests, usually because of a timeout somewhere nobody configured.

The failures that surprise us most are almost always timeout-related. A dependency that's down is easy. A dependency that's slow holds every connection until the whole system is unavailable for reasons that look unrelated.

Here it showed up as a queue nobody owned. North of 600 deliverables a month went through it, and producers had learned to check it twice a day because the alternative was a master was published with the wrong audio mix attached. A better queue wasn't the answer. Making ownership a property of the delivery package was.

WHERE IT GOES WRONG

• The same failure policy for the address validator and the payment authoriser.
• Silently skipping enrichment, so incomplete records look complete and can't be found later.
• A fallback path that's never been exercised and doesn't work when it's needed.
• No timeout on a dependency, so slow is worse than down.

Decide what happens when each dependency is gone. Then make the degraded state visible.

WHAT WE TOOK AWAY

The work shipped and on-time delivery moved, but the thing we're proudest of is smaller than the system: producers stopped keeping a private spreadsheet. That's usually the honest signal that the model finally matches the job.