Retries, jitter and the herd

A pattern earns its keep by being cheaper to apply than to argue about. This one has survived credit operations and four other sectors that share none of the same vocabulary, which usually means the shape is real and not a coincidence.

Retries are the most common way a small problem becomes an outage. A dependency wobbles, everything retries at once, and the additional load guarantees the wobble becomes a failure. The retry policy is a capacity decision.

SYNCHRONISED CLIENTS

Fixed backoff synchronises your callers. Everything that failed at the same moment retries at the same moment, so the dependency gets a load spike precisely while it's least able to serve one. It recovers, gets hit again, and you've built an oscillator.

Randomised backoff fixes it, and it's a one-line change that nobody makes until they've seen the graph. The graph is unmistakable: evenly spaced spikes at exactly your retry interval.

KNOW WHAT'S WORTH RETRYING

A timeout might succeed next time. A validation error will not, and retrying it three times just multiplies your logs. Classify the failure before deciding, and default to not retrying when you can't tell. An unbounded retry of a permanent error is a way to fill a disk.

Retries also need a budget across the whole system, not just per call. Three layers each retrying three times is twenty-seven attempts for one logical request, which is how a modest problem turns into a self-inflicted denial of service.

It paid for itself the first time an approval was issued against a stale valuation. Because the structure was already there, recovery was a query instead of an investigation, and underwriters heard it from the system rather than from a customer.

STOP TRYING

A circuit breaker is just the recognition that continuing to call something that's clearly down helps nobody, including you. Fail fast, degrade honestly, and check occasionally whether it's back.

The half-open state is where these go wrong. Let everything through at once when the timer expires and you've re-created the herd you were avoiding. Let one request through, and decide based on that.

At a specialist asset lender it turned a recurring escalation into an ordinary state the system could describe. Around 2,500 applications a month went through without a single manual reconciliation, which hadn't been true of any quarter before it.

WHERE IT GOES WRONG

• A circuit breaker that lets all traffic through at once when it half-opens.
• Fixed backoff synchronising every client into evenly spaced load spikes.
• Retrying validation errors, filling the logs and never succeeding.
• Three layers retrying three times each: twenty-seven attempts per logical request.

Jitter always, classify before retrying, and budget attempts across the whole path.

WHEN TO REACH FOR IT

It's cheap on day one and expensive to retrofit, which makes it a default rather than a decision. We put it in the first commit and haven't regretted it yet.