Let’s talk

The idempotency race

Your payment API checks the idempotency key and returns the stored result for a retry. The app retries while the first request is still at the bank. How many times is the customer charged?

In 30 seconds

The problem
Customers are charged twice even though the API has idempotency keys, and the database shows one payment.
Why it happens
The key is checked with a SELECT and saved after the charge. A retry that arrives while the first request is at the bank finds nothing too, so both charge.
The fix
Claim the key with INSERT … ON CONFLICT DO NOTHING before calling the bank. Only the request that inserts the row charges.
In an interview
Don't check the idempotency key, claim it. One INSERT … ON CONFLICT DO NOTHING before the charge: whoever inserts the row charges, everyone else reads it. A retry that finds it pending gets PROCESSING, not a second charge.

Why it matters

This version passes review and every sequential test: retry after the first request finished, and the SELECT finds the row. Our proof does it 50 times and the bank sees 50 charges.

In our run, 50 customers paid during a slow sale and every 2nd app retried while its first request was still at the bank. All 25 retries passed the key check: 25 customers charged twice, ₹50,000 extra, and every request answered PAID. With the claim moved first, each key reached the bank once and the retries got PROCESSING.

Words you need

Idempotency key
A unique id the client creates once per payment and sends with every retry of it. The server uses it to recognise a retry and return the first result instead of charging again.
Race condition
A bug that only appears when two requests overlap in an unlucky order. Each request is correct on its own.
Unique index
A database rule that rejects a second row with the same value. A safety net: it turns a silent duplicate into an error.
Atomic
Done as one step that nothing can interrupt. Another request sees it either not started or fully finished.

Picture it

Priya taps Pay. The bank is slow, so her app retries with the same key while the first request is still waiting.

SELECT the key, charge, then save

  1. Request 1: SELECT key → nothing
  2. Request 1 waits at the bank
  3. Retry: SELECT key → still nothing
  4. Both charge; second INSERT does nothing

Charged twice, both told PAID, one row in the table.

Claim the key first

  1. Request 1: INSERT key, pending → wins
  2. Request 1 waits at the bank
  3. Retry: INSERT key → 0 rows, reads pending
  4. Retry answers PROCESSING

Charged once. The app polls instead of paying again.

The fixes people try

  • SELECT, then charge, then INSERT

    Fails
    if (!exists(key)) charge(); insert()

    A retry mid-charge passes the same check. The bank call sits right inside the gap.

  • Add ON CONFLICT DO NOTHING to the save

    Fails
    INSERT … ON CONFLICT DO NOTHING (after charge)

    Stops the error, hides the evidence. Two charges, one row.

  • Wrap it in a transaction

    Fails
    BEGIN; SELECT; charge; INSERT; COMMIT

    At the default isolation level both transactions still see no row. And the bank call doesn't belong in a transaction.

  • SELECT … FOR UPDATE

    Half a fix
    SELECT … WHERE key = $1 FOR UPDATE

    Locks a row that exists. A brand-new key has no row to lock yet.

  • Claim the key first

    Works
    INSERT … 'pending' ON CONFLICT DO NOTHING RETURNING id

    The unique index lets exactly one request insert the row, and only that request calls the bank.

The fix and its cost

Same ON CONFLICT DO NOTHING, opposite position. Moved before the charge, it stops being a way to swallow errors and becomes the lock: whoever inserts the row owns the payment.

A request that gets zero rows back reads the row instead. Succeeded means return the stored result. Pending means the first request is still at the bank, so answer PROCESSING and let the app poll. Never charge from that branch.

This is Airbnb's pre-RPC phase in Orpheus: record the request and its key before the network call.

What it costs

A pending state the app has to handle, and a plan for payments stuck in pending if the first request dies mid-charge. That is a later trap this week.

Interviewers also listen for: why the check looked fine

Check-then-act is safe only when nothing else can act in between. Here the slowest step, the bank call, sits between the check and the write, so the window is as long as the bank is slow.

Check yourself

  1. 1Why does the SELECT check work in tests but not in production?Show answer

    Tests retry after the first request finishes. In production a retry can arrive while the first is still at the bank, before anything was saved.

  2. 2Why does the table show only one payment?Show answer

    The second INSERT hits the unique key and ON CONFLICT DO NOTHING quietly drops it. The second charge exists only at the bank.

  3. 3What should a retry get while the first request is still in progress?Show answer

    PROCESSING. The app shows payment in progress and checks again later. It must not start a second charge.

The code

GitHub

pay.bug.ts

import type { Pool } from "pg";

// The idempotency key the way most people add it.
export async function pay(db: Pool, bank: Bank, req: PayRequest) {
  const seen = await db.query(
    "SELECT charge_id FROM payments WHERE idempotency_key = $1",
    [req.idempotencyKey],
  );
  if (seen.rows.length > 0) return { status: "PAID", chargeId: seen.rows[0].charge_id };

  // A retry that arrives while we wait here also found nothing above.
  const charge = await bank.charge(req.orderId, req.amount);

  await db.query(
    `INSERT INTO payments (idempotency_key, order_id, amount, status, charge_id)
     VALUES ($1, $2, $3, 'succeeded', $4)
     ON CONFLICT (idempotency_key) DO NOTHING`, // hides the second charge
    [req.idempotencyKey, req.orderId, req.amount, charge.id],
  );
  return { status: "PAID", chargeId: charge.id };
}

Example code. The tested version, with a proof test against real Postgres, lands in the repo.

Star the repo for the runnable version

Each trap in this world ships with the bug, the fix and a proof test that fires concurrent requests at a real database.

Remember

Don't check the key, claim it. INSERT before the charge, and a pending row means PROCESSING, never a second charge.