Retries, jitter and the herd, from first principles

Most write-ups of this start in the middle. We'll start earlier than strictly necessary, because nine times in ten the confusion comes from a word two people are using differently, not from the mechanism.

Retries are the most common way a small problem becomes an outage. A dependency wobbles, everything retries at once, and the additional load guarantees the wobble becomes a failure. The retry policy is a capacity decision.

SYNCHRONISED CLIENTS

Fixed backoff synchronises your callers. Everything that failed at the same moment retries at the same moment, so the dependency gets a load spike precisely while it's least able to serve one. It recovers, gets hit again, and you've built an oscillator.

Randomised backoff fixes it, and it's a one-line change that nobody makes until they've seen the graph. The graph is unmistakable: evenly spaced spikes at exactly your retry interval.

KNOW WHAT'S WORTH RETRYING

A timeout might succeed next time. A validation error will not, and retrying it three times just multiplies your logs. Classify the failure before deciding, and default to not retrying when you can't tell. An unbounded retry of a permanent error is a way to fill a disk.

Retries also need a budget across the whole system, not just per call. Three layers each retrying three times is twenty-seven attempts for one logical request, which is how a modest problem turns into a self-inflicted denial of service.

Made concrete: a claim file passes through several pairs of hands, and each of them believes it owns the record. What's above is how you let all of them be right without letting any of them silently overwrite the others.

STOP TRYING

A circuit breaker is just the recognition that continuing to call something that's clearly down helps nobody, including you. Fail fast, degrade honestly, and check occasionally whether it's back.

The half-open state is where these go wrong. Let everything through at once when the timer expires and you've re-created the herd you were avoiding. Let one request through, and decide based on that.

Picture a specialty claims administrator at about 15,000 claims a month. The naive version works perfectly in testing and falls over the first day two adjusters touch the same claim file in the same second.

WHERE IT GOES WRONG

• Retrying validation errors, filling the logs and never succeeding.
• Three layers retrying three times each: twenty-seven attempts per logical request.
• A circuit breaker that lets all traffic through at once when it half-opens.
• Fixed backoff synchronising every client into evenly spaced load spikes.

Jitter always, classify before retrying, and budget attempts across the whole path.

THE SHORT VERSION

None of this is advanced. It's ordinary practice that gets skipped when a date is close, and then paid for with interest by whoever's on call.