Idempotency as a contract: the decision and what it cost

Written in the form we use internally, because the useful part of a decision record isn't the decision. It's the context that made it reasonable. In two years someone will want to reverse this, and they deserve to know what we knew at the time.

Anything reachable over a network will be called twice. Not because of a bug, but because a timeout gives the caller no way to know whether the first attempt landed. Make the second call harmless and a whole category of incident disappears.

THE TIMEOUT PROBLEM

When a request times out, the caller knows exactly one thing: no response arrived. It doesn't know if the work happened. Its two options are to retry, and risk doing the thing twice, or not retry, and risk not doing it at all. Neither is safe unless you've made a promise about repeats.

That promise is the contract. The caller sends a key it chose; you guarantee that the same key means the same operation, no matter how many times it arrives. Now retrying is boring, which is what you want at 3am.

THE KEY HAS TO COME FROM THE CALLER

Generating the key server-side defeats the point. The caller can't tell you "this is the same request as before" if it doesn't own the identifier. The key belongs to the intent, so it's created once, when the decision is made, and reused on every attempt.

It also needs a lifetime and a stored result. Recording that a key was used isn't enough; you have to be able to return what happened last time, or the caller still can't tell success from a duplicate. That's the part most implementations skip.

We modelled it at roughly 40,000 loads a month and the difference only showed up in the tail. At median load you couldn't tell them apart. At the ninety-ninth percentile, one of them stopped being able to explain itself.

export async function reserve(key: IdempotencyKey, cmd: Reserve) {
  const seen = await store.get(key)
  if (seen) {
    // Not an error. The caller timed out and is asking again.
    if (seen.fingerprint !== fingerprint(cmd)) throw new KeyReuseConflict(key)
    return seen.result
  }

  const result = await db.tx(async (t) => {
    const r = await allocate(t, cmd)
    // Result and key committed together, or neither.
    await store.put(t, key, { fingerprint: fingerprint(cmd), result: r })
    return r
  })

  return result
}

WHERE IT MATTERS MOST

Money and side effects. Anything that moves a balance, sends a message, allocates a resource or notifies a human deserves this treatment. Read paths generally don't need it, which is a useful way to keep the machinery contained.

The awkward cases are the ones with an external side effect you don't control. There the honest answer is to record your intent before you act, so a replay can at least tell you what you were trying to do.

The deciding factor was regulatory, not technical. Dispatchers have to be able to reconstruct why a given load tender was handled the way it was, months later, in front of someone unfriendly. That killed two of the three options on the spot.

WHERE IT GOES WRONG

• Recording that a key was used but not what it returned, so retries still can't tell success from duplicate.
• Generating the key server-side, which makes it impossible for a caller to identify a repeat.
• Writing the key outside the transaction that does the work, leaving a window where one exists without the other.
• Keys with no expiry, so the table grows forever and nobody dares add an index.

The caller owns the key. You owe them the same answer every time.

CONSEQUENCES WE ACCEPTED

We took a slower first two months in exchange for a system you can still reason about in year three. On an eighteen-month horizon we'd have chosen differently, and we said so at the time.