Retries, jitter and the herd, as actually built

We inherit systems more often than we start them. This is how we read one: what we look at first, and what each thing tells us about decisions made by people we'll never meet.

Retries are the most common way a small problem becomes an outage. A dependency wobbles, everything retries at once, and the additional load guarantees the wobble becomes a failure. The retry policy is a capacity decision.

SYNCHRONISED CLIENTS

Fixed backoff synchronises your callers. Everything that failed at the same moment retries at the same moment, so the dependency gets a load spike precisely while it's least able to serve one. It recovers, gets hit again, and you've built an oscillator.

Randomised backoff fixes it, and it's a one-line change that nobody makes until they've seen the graph. The graph is unmistakable: evenly spaced spikes at exactly your retry interval.

KNOW WHAT'S WORTH RETRYING

A timeout might succeed next time. A validation error will not, and retrying it three times just multiplies your logs. Classify the failure before deciding, and default to not retrying when you can't tell. An unbounded retry of a permanent error is a way to fill a disk.

Retries also need a budget across the whole system, not just per call. Three layers each retrying three times is twenty-seven attempts for one logical request, which is how a modest problem turns into a self-inflicted denial of service.

The exception handlers were the most honest documentation in the repo. Every branch that swallowed an error marked a spot where production planners had once been paged and somebody had made it stop.

STOP TRYING

A circuit breaker is just the recognition that continuing to call something that's clearly down helps nobody, including you. Fail fast, degrade honestly, and check occasionally whether it's back.

The half-open state is where these go wrong. Let everything through at once when the timer expires and you've re-created the herd you were avoiding. Let one request through, and decide based on that.

The schema told the real story. Three nullable columns added in one migration, all named after a work instruction field, all populated for precisely the weeks when a revised instruction reached the floor after the batch had run was an open ticket.

WHERE IT GOES WRONG

• Three layers retrying three times each: twenty-seven attempts per logical request.
• A circuit breaker that lets all traffic through at once when it half-opens.
• Fixed backoff synchronising every client into evenly spaced load spikes.
• Retrying validation errors, filling the logs and never succeeding.

Jitter always, classify before retrying, and budget attempts across the whole path.

THE VERDICT

Better than its reputation. Most of what looked like bad engineering turned out to be a sensible answer to a constraint that had since been lifted, and nobody had gone back to check.