The arithmetic of idempotency as a contract

The question is never whether something is worth doing in the abstract. It's whether it's worth doing at your volume, with your failure rate, valuing your team's time properly. So let's do the sum.

Anything reachable over a network will be called twice. Not because of a bug, but because a timeout gives the caller no way to know whether the first attempt landed. Make the second call harmless and a whole category of incident disappears.

THE TIMEOUT PROBLEM

When a request times out, the caller knows exactly one thing: no response arrived. It doesn't know if the work happened. Its two options are to retry, and risk doing the thing twice, or not retry, and risk not doing it at all. Neither is safe unless you've made a promise about repeats.

That promise is the contract. The caller sends a key it chose; you guarantee that the same key means the same operation, no matter how many times it arrives. Now retrying is boring, which is what you want at 3am.

THE KEY HAS TO COME FROM THE CALLER

Generating the key server-side defeats the point. The caller can't tell you "this is the same request as before" if it doesn't own the identifier. The key belongs to the intent, so it's created once, when the decision is made, and reused on every attempt.

It also needs a lifetime and a stored result. Recording that a key was used isn't enough; you have to be able to return what happened last time, or the caller still can't tell success from a duplicate. That's the part most implementations skip.

At about 15,000 claims a month with a two percent exception rate, adjusters were absorbing about nine hours of manual reconciliation a week. That's the number the build had to beat, and it's a lower bar than anyone in the room expected.

export async function reserve(key: IdempotencyKey, cmd: Reserve) {
  const seen = await store.get(key)
  if (seen) {
    // Not an error. The caller timed out and is asking again.
    if (seen.fingerprint !== fingerprint(cmd)) throw new KeyReuseConflict(key)
    return seen.result
  }

  const result = await db.tx(async (t) => {
    const r = await allocate(t, cmd)
    // Result and key committed together, or neither.
    await store.put(t, key, { fingerprint: fingerprint(cmd), result: r })
    return r
  })

  return result
}

WHERE IT MATTERS MOST

Money and side effects. Anything that moves a balance, sends a message, allocates a resource or notifies a human deserves this treatment. Read paths generally don't need it, which is a useful way to keep the machinery contained.

The awkward cases are the ones with an external side effect you don't control. There the honest answer is to record your intent before you act, so a replay can at least tell you what you were trying to do.

The interesting term wasn't engineering time. It was the cost of a reserve was released twice against the same loss landing once in the wrong quarter, which the client could size to the pound and we couldn't size at all.

WHERE IT GOES WRONG

• Generating the key server-side, which makes it impossible for a caller to identify a repeat.
• Writing the key outside the transaction that does the work, leaving a window where one exists without the other.
• Keys with no expiry, so the table grows forever and nobody dares add an index.
• Recording that a key was used but not what it returned, so retries still can't tell success from duplicate.

The caller owns the key. You owe them the same answer every time.

WHERE THE MODEL BREAKS

Do the numbers before the meeting, not during it. A decision that survives arithmetic tends to survive the next reorg too, because the reasoning outlives the people who made it.