Payment Idempotency
An Experienced Engineer’s Walkthrough for Backend Engineers
Payment idempotency means: submitting the same payment request multiple times (e.g. user double-clicks “Pay,” client retries after a timeout, or a bug sends the request twice) should only result in one charge. Without it, one order can be charged twice — or more — which is unacceptable for money and leads to refunds, support load, and loss of trust. As a newcomer, it’s easy to assume “we only call the payment provider once,” but in production you get duplicate requests; the only way to be safe is to design for idempotency from the start. The mechanisms are: idempotency key (so we recognize “same request”), state machine (pending → in progress → paid/failed/closed so we never charge twice), and reconciliation with the payment provider (to catch inconsistencies). This article is written as if I’m sitting next to you: we’ll go through why double charge happens, the end-to-end flow, the state machine, and the main design points.
Lesson 1: Why Idempotency Is Non-Negotiable
The Problem — How Double Charge Happens
A typical flow: user clicks “Pay” → request hits your server → you call the payment provider (WeChat Pay, Alipay, Stripe, etc.) → the provider debits the user and (eventually) returns success. But in production, the response from the provider can be slow (e.g. 5–10 seconds) or the network can drop it. The client (or an API gateway) might time out and retry the same request. Your server then receives a second “Pay for order X” request and, if you don’t recognize it as the same logical payment, you call the provider again. The provider may charge the user again for the same order. So one order, two charges. The same can happen with double-clicks (two requests in quick succession), two browser tabs, or a bug that sends duplicate requests. As an experienced engineer, I assume every payment request can arrive more than once; the only safe design is to make the operation idempotent: same request (identified by a key) always yields the same outcome (one charge, one result).
So the system must recognize “this is the same logical request” (via an idempotency key, e.g. order_id or client-provided key) and return the same result without calling the provider again for that key — or call the provider in an idempotent way (e.g. same out_trade_no so the provider also deduplicates).
Lesson 1 Takeaway
Payment requests must be idempotent. The same idempotency key (e.g. order_id or client-provided key) must always map to the same outcome: one charge, one result.
Lesson 2: The End-to-End Flow
Sequence: Request → Check → Provider → State
In words:
- Request carries order_id or idempotency_key.
- Check DB: If a record exists and is in a final state (paid, closed, failed), return the original result — do not call the provider again. This is the idempotent path.
- If no record or non-final (e.g. pending, in_progress): Reserve the slot (e.g. unique index on key + insert, or SELECT FOR UPDATE), then call the payment provider with a stable out_trade_no (or equivalent). Providers that support idempotency will only process once per out_trade_no.
- Provider success → update state to paid, persist; return success to client.
- Provider in progress (e.g. user must complete on provider’s page) → update state to in_progress; client polls or waits for callback from provider.
- Provider failure → update state to failed; return failure to client.
- Duplicate requests (same key, concurrent): Only one request wins the slot (insert or lock); others see "already exists" or "in progress" and return the existing result or "in progress."
Lesson 2 Takeaway
Idempotency = same key → same result. Before calling the provider, check DB; if final state, return. Concurrency = unique key + insert or lock so only one request proceeds to charge.
Lesson 3: State Machine
Payments should move through well-defined states so that duplicate or retry requests can be handled consistently.
States (Typical)
- Pending: Request received; not yet sent to provider (or sent but not confirmed).
- InProgress: Sent to provider; waiting for user action or callback (e.g. 3DS, redirect).
- Paid: Successfully charged. Final.
- Failed: Provider returned failure or timeout. Final.
- Closed: Cancelled or refunded; no further charge. Final.
Transitions
- Only allowed transitions (e.g. Pending → InProgress → Paid/Failed) should be permitted. Final states are immutable — no further charge attempts for that key.
- Duplicate request when state is Paid or Closed → return success or "already closed." When Failed → return failure. When InProgress → return "in progress" or poll/callback result.
Lesson 3 Takeaway
A state machine with final states (paid, failed, closed) ensures that duplicate requests never trigger a second charge. Enforce transitions in code and DB.
Lesson 4: Main Points Table
| Point | Description |
|---|---|
| Idempotency key | Order id or client-provided idempotency_key; unique index in DB |
| State machine | Final states (paid, closed, failed) immutable; only valid transitions allowed |
| Provider | Use provider idempotency (e.g. out_trade_no); same ID = same charge on their side |
| Reconciliation | Periodically reconcile with provider; fix over-charge, under-charge, or missing records |
Concurrency
- Same order, concurrent requests: Use unique index on idempotency key + insert to "grab" the slot, or SELECT FOR UPDATE; only one request proceeds to call the provider; others return "in progress" or the existing result.
Reconciliation
- Reconciliation is the last line of defense: compare your DB with the provider’s records (e.g. daily). Handle anomalies (over-charge, under-charge, missing or extra records) with a defined process (manual or automated).
Lesson 4 Takeaway
Idempotency key + state machine + provider idempotency prevent double charge. Reconciliation catches inconsistencies and provider-side edge cases.
Key Rules (Summary)
- Idempotency key required: Reject if missing, or generate server-side key and return to client for retries.
- State machine must be strict; final states irreversible; in-progress, closed, etc. well-defined.
- Reconciliation is last line of defense; have a process when provider and local records differ.
What's Next
See Idempotency Design, Distributed Transactions. See Database Transactions for atomic state updates.