The idempotency race
Your payment API checks the idempotency key and returns the stored result for a retry. The app retries while the first request is still at the bank. How many times is the customer charged?
In 30 seconds
- The problem
- Customers are charged twice even though the API has idempotency keys, and the database shows one payment.
- Why it happens
- The key is checked with a SELECT and saved after the charge. A retry that arrives while the first request is at the bank finds nothing too, so both charge.
- The fix
- Claim the key with INSERT … ON CONFLICT DO NOTHING before calling the bank. Only the request that inserts the row charges.
- In an interview
- Don't check the idempotency key, claim it. One INSERT … ON CONFLICT DO NOTHING before the charge: whoever inserts the row charges, everyone else reads it. A retry that finds it pending gets PROCESSING, not a second charge.
Why it matters
This version passes review and every sequential test: retry after the first request finished, and the SELECT finds the row. Our proof does it 50 times and the bank sees 50 charges.
In our run, 50 customers paid during a slow sale and every 2nd app retried while its first request was still at the bank. All 25 retries passed the key check: 25 customers charged twice, ₹50,000 extra, and every request answered PAID. With the claim moved first, each key reached the bank once and the retries got PROCESSING.
Words you need
- Idempotency key
- A unique id the client creates once per payment and sends with every retry of it. The server uses it to recognise a retry and return the first result instead of charging again.
- Race condition
- A bug that only appears when two requests overlap in an unlucky order. Each request is correct on its own.
- Unique index
- A database rule that rejects a second row with the same value. A safety net: it turns a silent duplicate into an error.
- Atomic
- Done as one step that nothing can interrupt. Another request sees it either not started or fully finished.
Picture it
Priya taps Pay. The bank is slow, so her app retries with the same key while the first request is still waiting.
SELECT the key, charge, then save
- Request 1: SELECT key → nothing
- Request 1 waits at the bank
- Retry: SELECT key → still nothing
- Both charge; second INSERT does nothing
Charged twice, both told PAID, one row in the table.
Claim the key first
- Request 1: INSERT key, pending → wins
- Request 1 waits at the bank
- Retry: INSERT key → 0 rows, reads pending
- Retry answers PROCESSING
Charged once. The app polls instead of paying again.
The fixes people try
SELECT, then charge, then INSERT
Failsif (!exists(key)) charge(); insert()A retry mid-charge passes the same check. The bank call sits right inside the gap.
Add ON CONFLICT DO NOTHING to the save
FailsINSERT … ON CONFLICT DO NOTHING (after charge)Stops the error, hides the evidence. Two charges, one row.
Wrap it in a transaction
FailsBEGIN; SELECT; charge; INSERT; COMMITAt the default isolation level both transactions still see no row. And the bank call doesn't belong in a transaction.
SELECT … FOR UPDATE
Half a fixSELECT … WHERE key = $1 FOR UPDATELocks a row that exists. A brand-new key has no row to lock yet.
Claim the key first
WorksINSERT … 'pending' ON CONFLICT DO NOTHING RETURNING idThe unique index lets exactly one request insert the row, and only that request calls the bank.
The fix and its cost
Same ON CONFLICT DO NOTHING, opposite position. Moved before the charge, it stops being a way to swallow errors and becomes the lock: whoever inserts the row owns the payment.
A request that gets zero rows back reads the row instead. Succeeded means return the stored result. Pending means the first request is still at the bank, so answer PROCESSING and let the app poll. Never charge from that branch.
This is Airbnb's pre-RPC phase in Orpheus: record the request and its key before the network call.
What it costs
Interviewers also listen for: why the check looked fine
Check yourself
1Why does the SELECT check work in tests but not in production?Show answerHide
Tests retry after the first request finishes. In production a retry can arrive while the first is still at the bank, before anything was saved.
2Why does the table show only one payment?Show answerHide
The second INSERT hits the unique key and ON CONFLICT DO NOTHING quietly drops it. The second charge exists only at the bank.
3What should a retry get while the first request is still in progress?Show answerHide
PROCESSING. The app shows payment in progress and checks again later. It must not start a second charge.
The code
pay.bug.ts
Example code. The tested version, with a proof test against real Postgres, lands in the repo.
Star the repo for the runnable version
Each trap in this world ships with the bug, the fix and a proof test that fires concurrent requests at a real database.
Remember
Don't check the key, claim it. INSERT before the charge, and a pending row means PROCESSING, never a second charge.