Idempotency Is Not a Header. It Is a Contract.

The first time I was told to make an endpoint idempotent, I did what most people do: I added a unique index on a request id, wrapped the insert in a try/catch, and swallowed the duplicate key error. Done. Idempotent.

It was not. It handled one of the three ways a payment webhook actually duplicates, and it was the least dangerous one.

I want to write down what I understand now, because "just make it idempotent" is advice that sounds complete and is not.

Why the duplicates happen at all

Idempotency is not a nice-to-have that guards against a rare glitch. Duplicate delivery is the normal operating mode of the systems you are integrating with, and it happens for a structural reason.

The provider sends you a callback. Your service processes it correctly, writes the row, and starts constructing a 200. Then the connection times out. From the provider's side, that call has no outcome — it cannot distinguish "the request never arrived" from "the request was fully processed and the response was lost." Both look identical from outside.

Facing that ambiguity, every serious provider makes the same choice: retry. They pick at-least-once delivery, because losing a payment notification is worse than sending it twice. That choice is correct. It also means the duplicate is not a bug on their side that will eventually get fixed — it is a permanent, designed property of the interface, and absorbing it is your job.

Three different duplicates

Here is the part that took me longest. "Duplicate" is not one event.

The identical retry. Same provider transaction, same amount, same payload, delivered twice. This is the common case and the one a unique index handles.

The re-delivery after state change. The provider sends paid, then re-sends paid an hour later — but in between, the user requested a refund and the payment is now refunded. Naively re-applying the second paid walks the payment backwards into a state it already left. A unique-index check on request id may not catch this at all, because retries an hour apart sometimes carry a fresh id.

The semantic duplicate. Two genuinely different provider transactions that represent one user intention — the classic double-click, or a user who paid, saw a spinner, and paid again. These are not duplicates at the protocol level. Both are real, both took real money. No amount of request-id deduplication touches this. It requires a business rule, and the honest answer is usually "detect, do not silently merge" — flag it, refund one, tell someone.

If you only defend against the first kind, you will feel protected and still get hurt by the second and third.

What actually makes it work

Three pieces, and the order matters.

1. Decide what identity means. Not "the request has an id" but "these two messages describe the same event in the world." Usually that key is (provider, provider_transaction_id), not your own request id — because it is the provider's identifier that stays stable across their retries. Whatever you choose, it goes in a unique constraint in the database, not in application logic. A check in code is a race; a constraint is a guarantee.

CREATE UNIQUE INDEX payment_events_provider_txn_uniq
  ON payment_events (provider, provider_transaction_id);

2. Make the state machine refuse illegal moves. Idempotency and state validity are the same problem seen twice. If a payment can only go pending → paid → refunded, then an incoming paid for something already refunded must be rejected — not because it is a duplicate, but because it is an invalid transition. Encode that explicitly and the second class of duplicate stops being a special case.

const ALLOWED: Record<Status, Status[]> = {
  pending:  ['paid', 'cancelled'],
  paid:     ['refunded'],
  refunded: [],
  cancelled: [],
};
 
function canTransition(from: Status, to: Status): boolean {
  return ALLOWED[from].includes(to);
}

3. Return the same answer, not an error. This is the part people get wrong most often. When a completed, identical duplicate arrives, the correct response is the original successful response, not a 409. The provider asking twice should learn the same thing both times. An error tells them something went wrong, and a well-behaved provider responds to errors by — retrying. You have built a loop.

The word completed is carrying weight there. Two cases are genuinely different and a 409 is the honest answer to both: the same key arriving with a different payload (you cannot replay a result for a request you never processed), and a replay landing while the original is still in flight (there is no result yet — say so, with a Retry-After). Stripe behaves this way for exactly these two cases. The rule is not "never 409"; it is "never 409 a duplicate you have already successfully answered."

The other half of this problem — the retry your client sends, and the atomic claim that stops it charging twice — is in The Payment That Charged Twice

Log the evidence before you trust the logic

The change that did the most for me was not in the idempotency logic at all. It was adding an append-only table that records every inbound callback before anything is decided about it: provider, raw payload, received timestamp, the identity key extracted from it, and what the service concluded.

This costs storage and buys the ability to answer questions. "Did this callback arrive twice, or did we process it twice?" is unanswerable without it, and it is exactly the question you will be asked, at speed, by someone who is not interested in your architecture.

It also enables the technique I would now recommend to anyone rewriting a payment path: run the new logic in shadow mode first. Both old and new implementations see every event. Only the old one is authoritative. The new one writes what it would have done. After a week, you compare. Divergences are either bugs in the new code or bugs in the old code you had never noticed, and both are worth finding before the new path controls money.

Shipping the new logic as the decision-maker on day one means finding those divergences in production, with real transactions, on a schedule set by chance.

What I would tell myself two years ago

Idempotency is not a middleware you install. It is three decisions — what counts as the same event, what transitions are legal, what a repeat caller should hear — plus a durable record of what actually arrived.

The middleware version protects you from the boring duplicate. The three decisions protect you from the interesting ones, and the interesting ones are the reason anyone cares.