Payment API Idempotency: Idempotency Keys, Retries, Exactly-Once Effects and Double-Charge Prevention (2026)

A customer taps “Pay”, the mobile network stalls, and the client library times out after ten seconds. Did the charge happen? The client cannot know. If it retries blindly it may bill the customer twice; if it gives up it may lose the sale. Every networked payment system lives inside this ambiguity, and the standard answer is the idempotency key: a client-chosen token that lets the server recognise a retry and replay the original outcome instead of repeating the side effect.

This matters more in 2026 than it did a decade ago because retries are now everywhere: SDKs, service meshes, load balancers, queue consumers, webhook redeliveries and AI agents that call payment tools on a user’s behalf all re-send requests automatically. The IETF is also finishing a standard header for this pattern, so the contract is stabilising.

You will leave with a precise model of why duplicates occur, a storage design that survives races and crashes, a runnable Python and Postgres implementation, and an honest account of why “exactly-once” is really effectively-once.

What this covers: the failure model, the key protocol (Stripe and the IETF draft), storage and locking, cross-service idempotency with sagas and the outbox, the processor and card-network layer, a worked timeline, code, and failure modes. This is systems analysis, not financial or legal advice.

Context and Background

Idempotence is a mathematical property: applying an operation many times has the same effect as applying it once. HTTP already encodes this. RFC 9110 classifies GET, HEAD, OPTIONS, PUT and DELETE as idempotent methods, while POST and PATCH are not. Creating a payment is almost always a POST, which is exactly the method a client cannot safely retry. That mismatch is the entire problem.

The failure that creates duplicates is the ambiguous outcome. A request has three observable results for the client: a response arrived, the connection failed before the request was processed, or the connection failed after processing. The client sees only a timeout in the last two cases, and they are indistinguishable. In distributed systems terms, a timeout converts a delivery guarantee into a choice between at-most-once (do not retry, risk loss) and at-least-once (retry, risk duplication). No protocol on an unreliable network gives both for free; this is the well-known two generals problem applied to a card charge.

Payment providers converged on the same remedy. Stripe’s API accepts an Idempotency-Key header on POST requests; for API v1 it saves the status code and body of the first request made with a given key once endpoint execution begins, and returns that saved response, including 500 errors, to retries with the same key. Keys can be up to 255 characters, Stripe suggests V4 UUIDs or other high-entropy random strings, and for v1 keys may be pruned automatically once they are at least 24 hours old (see Stripe’s idempotent requests documentation). Other processors and banks expose equivalent mechanisms under different names: request IDs, client reference IDs, or merchant order numbers with uniqueness enforcement.

The IETF HTTPAPI working group is standardising the header as draft-ietf-httpapi-idempotency-key-header. As I write this, the working copy is still an Internet-Draft (intended status: Standards Track), not an RFC, so treat its details as stable-in-spirit but subject to change. We examine it below. For the broader API-design view, see our guide to idempotent API design, and for the retry policies that generate this traffic, see payment retries and dunning.

Why Retries Create Duplicates: The Failure Model

Before designing the fix, enumerate where the duplicate can be created. A single “pay” action crosses at least five boundaries, and each boundary can lose either the request or the reply.

The five ambiguity points

Client to edge. The mobile app or browser sends the request; the radio drops it, or the reply. Users also double-tap buttons and refresh pages after a payment.
Edge to service. Load balancers and service meshes such as Envoy or NGINX can retry on connection errors, and some policies retry on 5xx. A retry on an upstream that actually processed the request creates a second delivery the client never sees.
Service to database. The service writes its own record, then crashes before replying. A restart or failover retry re-executes the handler.
Service to processor. The gateway call to the acquirer or processor times out. The authorisation may have been approved at the issuer even though the answer never came back.
Message delivery. Brokers deliver at-least-once by default. A consumer that crashes after the side effect but before acknowledging will see the message again. Webhook senders likewise redeliver on non-2xx responses or timeouts.

The pattern is symmetrical: wherever an acknowledgement can be lost, the sender must either retry or accept uncertainty. Since payments cannot accept silent loss, every layer retries, and every layer therefore needs a defence.

Why “just check first” does not work

A tempting fix is read-then-write: query whether an order has been paid, and charge only if not. This is a classic time-of-check to time-of-use race. Two concurrent retries both read “not paid”, both proceed, and both charge. Even with a single client, the load balancer’s silent retry can race the original request. The check and the claim must be one atomic operation, which in practice means a uniqueness constraint or a lock held in a transactional store.

Naturally idempotent versus unsafe operations

Not every payment operation needs a key. Setting a value (“set shipping address to X”) is naturally idempotent; incrementing a value or appending to a ledger is not. A charge, a refund, a payout, a transfer and a ledger entry are all non-idempotent effects, so they all need a deduplication mechanism. A useful rule: if replaying the request would change money, balances or externally visible state a second time, it needs a key, a natural business key with a uniqueness constraint, or a conditional write.

The Idempotency Key Protocol

The idempotency key protocol has a small, well-defined contract. The client generates a unique token for each logical operation, sends it with every attempt of that operation, and the server guarantees that all attempts with the same key produce one effect and one canonical response.

What the IETF draft specifies

The Internet-Draft defines Idempotency-Key as an Item Structured Header Field (per RFC 9651) whose value must be a String, for example "8e03978e-40d5-43e8-bc93-6894a57f9324". Key points from the current working draft:

The key must be unique and must not be reused with a different request payload. A UUID (RFC 9562) or similar random identifier is recommended.
A server may combine the key with a fingerprint of the request payload (a checksum of the whole body or selected fields, for example) to decide whether a request is a true retry.
Servers may expire keys and should publish their expiry policy in their documentation.
Error semantics: 400 Bad Request when the header is missing on an operation that requires it; 422 Unprocessable Content when the key is reused with a different payload; 409 Conflict when a retry arrives while the original request is still in flight.
After a 409, the client may retry without changing anything; after 400 or 422, it must correct the request first.

The working copy has been revised repeatedly, so check the latest revision at the IETF datatracker before hard-coding behaviour. I cite the draft’s semantics, not a final RFC number, because none exists as of this writing.

What Stripe does differently from the draft

Stripe’s v1 behaviour is subtly different in a way that matters. It saves a result only after endpoint execution begins; if parameters fail validation, or the request conflicts with another request executing concurrently, no result is saved and the request can simply be retried. It also compares incoming parameters against the original and errors when they differ. And because it replays even 500 responses, a client that wants a fresh attempt after a server error must use a new key. The v2 API instead aims to complete failed or partially failed work on retry. This difference between “replay the stored answer” and “drive the operation to completion” is a genuine design axis, which we return to under failure modes.

Figure 1: A lost response followed by a retry with the same idempotency key. The server replays the stored result and the processor is called exactly once.

Figure 1 shows the essential choreography. The first attempt inserts the key before any side effect, calls the processor, and stores the response; the response is lost; the retry finds the stored row and returns the saved answer without touching the processor. The critical ordering rule is that the key is recorded before the side effect, never after. Recording it afterwards reopens the same race the protocol exists to close.

Who generates the key, and when

The client generates the key, and it must identify the intent, not the attempt. A fresh UUID per button press, persisted locally before the first send, is the usual approach. Generating a new key inside the retry loop defeats the purpose entirely. For server-to-server flows, a deterministic key derived from a business identity works well, for example a hash of the tenant, order ID and operation type. Two properties must hold: the key is stable across retries of the same intent, and different intents never collide. Avoid placing personal data such as an email address in a key, since keys appear in logs and Stripe explicitly advises against sensitive data in them.

Storage Design: Making the Key Durable and Atomic

The key store is the heart of the system, and most implementation bugs live here. Treat it as a small state machine backed by a transactional store, not as a cache.

The record schema

A minimal idempotency record holds six things:

Scope: the tenant or API credential, so one customer’s key cannot collide with or read another’s. The IETF draft’s security section recommends composite cache keys for exactly this reason.
Key: the client-supplied token, stored exactly as received after format validation.
Request fingerprint: a hash (SHA-256 is typical) over a canonical form of the method, path and body. This is what detects a key reused for a different request.
Status: in_progress, completed, or a retryable failure marker.
Stored response: status code, headers that matter, and body, so a replay is byte-identical.
Timestamps and lock metadata: created time for TTL pruning, plus a lock owner and lease expiry for crash recovery.

The primary key is (scope, key). The uniqueness constraint is what provides atomic claim: whichever transaction inserts first wins, and every other concurrent attempt gets a constraint violation or a “do nothing” result it can inspect. This is far more robust than application-level locking, because the database arbitrates.

The decision flow

Figure 2: The server-side decision flow. A new key claims a row, a matching completed key replays, a mismatched fingerprint returns 422, and an in-flight key returns 409.

Figure 2 maps directly onto the IETF error semantics. A missing row means this is a first attempt: claim it and proceed. An existing row with a different fingerprint is a client bug or an attack, so reject with 422. An existing row that matches and is completed is a normal retry, so replay. An existing row that matches but is in_progress is a concurrent duplicate; return 409 and let the client back off, or, in a more forgiving design, wait briefly for completion and then replay.

Fingerprinting: what to hash

Hash the parts of the request that define intent: the amount, currency, payment method reference, customer and destination. Exclude volatile fields such as timestamps, trace IDs, and client-generated request IDs that legitimately differ across retries. Canonicalise before hashing: sort JSON object keys, normalise number formats, and strip insignificant whitespace, otherwise a retry that serialises differently would be falsely rejected. The opposite mistake, hashing too little, lets a buggy client reuse a key for a different amount and silently receive the first charge’s response. A short rule helps: if two requests would legitimately be different payments, their fingerprints must differ.

Response replay and what to store

Store the response the client would have seen, not merely a “done” flag. A flag forces the retry to reconstruct the answer, which may now be different (a status that changed, a resource that was since modified). Stripe’s documented behaviour of returning the saved status code and body, even for a 500, is this principle in action: the replay is a faithful record of the first outcome. Be deliberate about secrets; if the response contains a token or client secret, either store it encrypted or ensure it is safe to return again.

TTL and pruning

Keys cannot live forever, since the table would grow without bound and old keys would collide with legitimately repeated operations. Stripe’s v1 documentation says keys may be removed once at least 24 hours old, after which a reused key generates a new request. Choose a window longer than your longest realistic retry horizon, including client-side persistent retry queues and offline mobile apps, but short enough to bound storage. Seven days is a common internal choice for payouts and batch flows; that is a judgment, not a standard. Publish the window, as the IETF draft advises, so integrators know when a key stops protecting them.

Capacity: a worked illustrative estimate

Suppose, illustratively, a platform handles 200 payment writes per second at peak, with a 24-hour retention. That is roughly 17 million keys per day at a sustained rate (200 x 86,400 = 17.28 million; real traffic is peaky, so average will be lower). At roughly 1 KB per row including the stored response, that is about 17 GB of live data plus indexes. This fits comfortably in a single Postgres instance with partitioning by day, where pruning means dropping a partition rather than running a slow DELETE. These numbers are illustrative; measure your own response sizes.

Races, Locks, and Crash Recovery

Idempotency is easy in the happy path and hard in the three adversarial cases: concurrent duplicates, crashes mid-flight, and retries that arrive while the first attempt is still running.

Concurrent duplicates

Two identical requests arrive within milliseconds, perhaps from a load-balancer retry. Both attempt INSERT ... ON CONFLICT DO NOTHING. Exactly one inserts (affected row count 1); the other gets zero rows and must read the existing record. If that record is in_progress, the loser should not run the side effect. It returns 409 (as the IETF draft prescribes) or polls with a short timeout. Resist the urge to let both run and “reconcile later”; that is how double charges happen.

The crash window

The dangerous window is between claiming the key and storing the result. If the process dies there, the row is stuck in in_progress forever, and every retry gets 409. There are two defensible policies:

Lease with expiry. The claim records a lock owner and a lease deadline (say 60 seconds, longer than the processor timeout). A retry that finds an expired lease may reclaim the row. This restores liveness but reintroduces ambiguity: the dead process may have already called the processor.
Recovery phases. Break the operation into recorded steps (for example started, processor_called, ledger_written). A reclaiming worker inspects the last completed phase and resumes from there rather than restarting. This is the approach Brandur Leach popularised in his write-up on idempotency keys with atomic phases in Postgres.

The second policy is more work but is the only one that is safe when the side effect is external and non-transactional, because the step “call the processor” can itself be made idempotent by passing the processor a deterministic reference derived from your key.

The external call is the hard part

Database writes can share a transaction with the key row. A call to a card processor cannot. If you call the processor inside a database transaction and the transaction later rolls back, the money has still moved. If you commit first and then call, a crash between the two loses the intent. The robust answer is to pass your own idempotency identifier downstream: when you call the processor, send it a key derived from yours (for example pay_{your_key}), so a re-executed call is deduplicated by the processor. This chains idempotency across the hop. Where the downstream does not support keys, you fall back to query-before-retry using a stable merchant reference, plus reconciliation as a safety net.

Isolation levels and advisory locks

Under Postgres READ COMMITTED, INSERT ... ON CONFLICT is sufficient to serialise claims on the same key, since the unique index is checked at insert time. If you prefer a lock-then-check pattern, pg_advisory_xact_lock(hashtext(scope || key)) serialises concurrent attempts for the transaction’s duration. Beware holding any database lock across a slow external call: it pins a connection and, at scale, exhausts the pool. The in-progress marker with a lease exists precisely so that the claim is durable but the lock is not held during network I/O.

A Runnable Python and Postgres Implementation

The following example implements the decision flow of Figure 2 using psycopg (v3) and Postgres. It is deliberately small; production code adds structured logging, metrics, canonical JSON handling and a real processor client. The processor call is simulated so the file runs standalone.

CREATE TABLE idempotency_keys (
    scope          text        NOT NULL,
    key            text        NOT NULL,
    fingerprint    text        NOT NULL,
    status         text        NOT NULL CHECK (status IN ('in_progress','completed')),
    response_code  integer,
    response_body  jsonb,
    lock_expires   timestamptz,
    created_at     timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (scope, key)
);
CREATE INDEX idempotency_keys_created_idx ON idempotency_keys (created_at);

import hashlib
import json
import uuid
import psycopg
from psycopg.types.json import Jsonb

LEASE_SECONDS = 60


def fingerprint(body: dict) -> str:
    canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()


def call_processor(reference: str, body: dict) -> dict:
    # Simulated downstream call. A real gateway would receive `reference`
    # as ITS idempotency key so a re-executed call is deduplicated there too.
    return {"id": "ch_" + uuid.uuid5(uuid.NAMESPACE_URL, reference).hex[:12],
            "amount": body["amount"], "currency": body["currency"],
            "status": "succeeded"}


def handle_charge(conn: psycopg.Connection, scope: str, key: str, body: dict):
    fp = fingerprint(body)

    # 1. Atomic claim. Exactly one concurrent caller inserts the row.
    with conn.transaction():
        claimed = conn.execute(
            """
            INSERT INTO idempotency_keys (scope, key, fingerprint, status, lock_expires)
            VALUES (%s, %s, %s, 'in_progress', now() + make_interval(secs => %s))
            ON CONFLICT (scope, key) DO NOTHING
            RETURNING 1
            """,
            (scope, key, fp, LEASE_SECONDS),
        ).fetchone()

    if not claimed:
        row = conn.execute(
            """SELECT fingerprint, status, response_code, response_body,
                      lock_expires < now() AS lease_expired
               FROM idempotency_keys WHERE scope = %s AND key = %s""",
            (scope, key),
        ).fetchone()
        stored_fp, status, code, resp, lease_expired = row
        if stored_fp != fp:
            return 422, {"error": "idempotency key reused with different payload"}
        if status == "completed":
            return code, resp                       # replay the original answer
        if not lease_expired:
            return 409, {"error": "original request still in progress"}
        # Lease expired: take over with a compare-and-set on the lease.
        with conn.transaction():
            took = conn.execute(
                """UPDATE idempotency_keys
                   SET lock_expires = now() + make_interval(secs => %s)
                   WHERE scope = %s AND key = %s AND status = 'in_progress'
                     AND lock_expires < now() RETURNING 1""",
                (LEASE_SECONDS, scope, key),
            ).fetchone()
        if not took:
            return 409, {"error": "another worker reclaimed the request"}

    # 2. Side effect OUTSIDE any held lock, with a derived downstream reference.
    result = call_processor(f"pay_{scope}_{key}", body)

    # 3. Persist the response so every retry replays it.
    with conn.transaction():
        conn.execute(
            """UPDATE idempotency_keys
               SET status = 'completed', response_code = 201, response_body = %s
               WHERE scope = %s AND key = %s""",
            (Jsonb(result), scope, key),
        )
    return 201, result

Three details deserve attention. First, the claim is one statement, so the database decides the winner. Second, the reclaim uses a compare-and-set on lock_expires, so two workers cannot both take over an expired lease. Third, the processor call receives a reference derived from the key, which makes step 2 safe to re-execute if the process dies before step 3. To exercise it, run two handle_charge calls in separate threads with the same key and watch one return 201 and the other 409 (or, after completion, 201 replayed).

Across Services: Sagas, the Outbox, and Effectively-Once

A key store protects one API. Real payments span several services: an order service, a payment service, a ledger, a fraud engine, a notification service. Idempotency has to hold on every internal hop, not just at the public edge.

Exactly-once is a property of effects, not delivery

The phrase exactly once is the most abused term in this field. A network cannot guarantee that a message is delivered once; it can guarantee at-most-once or at-least-once. What you can build is exactly-once effects: at-least-once delivery combined with an idempotent consumer, so the state change happens once even if the message arrives many times. Some people call this effectively-once. It is a property of the whole pipeline, and it breaks at the first non-idempotent hop.

Kafka illustrates the boundary honestly. Its idempotent producer and transactions give exactly-once semantics within Kafka: a consume-transform-produce loop that writes output and commits offsets atomically. The guarantee does not extend to an external system such as a card processor or an email provider; there you still need an idempotent sink. We cover the mechanics in Kafka exactly-once semantics.

The dual-write problem and the outbox

A payment service typically must update its database and publish an event (payment.succeeded). Doing both without a shared transaction is a dual write: if the database commit succeeds and the publish fails, downstream never learns; if the order is reversed, downstream learns of a payment that rolled back. The transactional outbox removes the problem by writing the event to an outbox table in the same database transaction as the state change. A separate relay, or change data capture (CDC) tailing the write-ahead log, publishes outbox rows to the broker afterwards.

The outbox gives at-least-once publication, which is exactly why consumers need an inbox: a table of processed message IDs with a uniqueness constraint, checked in the same transaction as the consumer’s own write. The message ID is the idempotency key for the internal hop. Our deep dive on the transactional outbox pattern and CDC covers the relay mechanics.

Figure 3: Outbox on the producer side, inbox on each consumer. Together they convert at-least-once delivery into exactly-once effects per consumer.

Figure 3 shows both halves. The payment API commits its state and an outbox row atomically. The relay publishes, possibly more than once after a crash. Each consumer records the message ID in its own inbox table in the same transaction as its effect, so a redelivered message hits the unique constraint and is skipped. The ledger consumer and the notification consumer each own an independent inbox, because dedupe state must live with the effect it protects.

Sagas and compensations

When a business operation spans services without a distributed transaction, the saga pattern sequences local transactions and defines compensating actions for rollback. Sagas intensify the idempotency requirement because both the forward steps and the compensations may be retried by the orchestrator. A “refund the payment” compensation that runs twice must not refund twice; a “reserve inventory” step that is redelivered must not reserve twice. Every saga step and every compensation therefore needs a stable step identifier, typically (saga_id, step_name), used as its idempotency key. See saga orchestration versus choreography for the coordination trade-offs.

A subtle point: compensations must be idempotent and commutative with a late-arriving original. If a “cancel” is processed before the delayed “create” it cancels, the system needs a tombstone so the late create is rejected. Without it, a reordered pair silently produces the opposite of what was intended.

Ledgers: idempotency at the system of record

At the ledger layer the same idea applies in a stricter form. Each posting should carry a unique external reference so that replaying a command cannot create a second journal entry. Double-entry databases enforce this with unique transfer IDs; an event-sourced ledger enforces it through deterministic event IDs or an expected-version check on the aggregate. Both approaches are discussed in TigerBeetle versus Postgres for ledgers and in event sourcing and bitemporal audit trails. The ledger is the last line of defence: even if every upstream layer fails, a unique constraint on the posting reference refuses the duplicate.

The Processor and Card-Network Layer

Beneath your API sits infrastructure that predates HTTP idempotency headers and speaks different dialects. Understanding it explains why reconciliation never goes away.

Retries at the message level

Card authorisations traditionally travel as ISO 8583 messages. Rather than a generic idempotency key, the protocol relies on identifiers carried in the message such as the System Trace Audit Number (STAN, data element 11) and the Retrieval Reference Number (RRN, data element 37), plus the transmission time and acquirer identifiers, which together let a receiver recognise a repeat. Field usage varies by network and by host implementation, so treat this as a general description and consult the specific network’s specification for the exact rules.

The conventional protocol response to uncertainty is the reversal. If an acquirer sends an authorisation request and times out, it does not simply re-send it as a new request; it sends a reversal message (in the common ISO 8583 message-type scheme, an 04xx class message) to cancel the authorisation that may or may not have happened, then decides whether to try again. Repeat variants of messages exist so a receiver can tell a retransmission from a new transaction. This “reverse on doubt” discipline is the card world’s analogue of the replay-on-retry discipline in HTTP.

Why your key stops at the gateway

Your idempotency key protects the hop between your client and your API. Whether it protects the hop between your gateway and the acquirer depends on whether you map it onto something the downstream honours: a processor request ID, a merchant reference, or an order number with a uniqueness rule. Pass it through whenever the processor supports it. When it does not, you need a read-before-retry step that queries by merchant reference, and a conservative policy that treats unknown outcomes as “pending” rather than “failed”.

Figure 4: Every layer retries. The key store protects the API; downstream references, reversals and reconciliation protect the layers beneath it.

Figure 4 makes the layering explicit. The client SDK, the load balancer and the API each retry on their own schedule; the card network adds reversals when an issuer times out; and reconciliation closes the loop by matching what you believe happened against processor and settlement files. When all else fails, reconciliation engine architecture is what detects a duplicate that slipped through, so it is part of the idempotency design, not a separate concern.

Authorise-and-capture and the two-step trap

Many payment flows split authorisation from capture. Each step needs its own key or deterministic reference, because capturing twice or capturing a stale authorisation are distinct failure modes. A single key reused across authorise and capture would make the second call replay the first response, which is wrong. Key by operation: (order_id, "authorize") and (order_id, "capture").

A Worked Failure Timeline

Concrete timelines expose what prose hides. Consider the scenario in which a response is lost and the client retries, comparing a system without a key to one with a key. Times are illustrative.

Time	Event (no idempotency key)	Event (with idempotency key)
T+0.0s	Client POSTs charge of 49.00	Client POSTs with key K1
T+0.1s	API calls processor, which approves	API claims K1, calls processor, which approves
T+0.4s	API writes order as paid	API stores response under K1
T+0.5s	Reply lost at the load balancer	Reply lost at the load balancer
T+10s	Client times out, retries	Client times out, retries with K1
T+10.1s	API calls processor again, approves again	API finds K1 completed
T+10.3s	Customer billed 98.00	API replays the original 201 response
Outcome	Double charge, refund and support ticket	One charge, client sees success

Now a harder variant: the API process crashes at T+0.2s, after the processor approved but before the response was stored. With only a key row marked in_progress and no downstream reference, the retry at T+10s hits 409 until the lease expires, then re-executes the processor call and double charges. With a derived downstream reference (pay_K1) the processor recognises the repeat and returns the original authorisation. With a phased record, the reclaiming worker also sees processor_called was reached and skips straight to finalisation. The three designs differ only in what is recorded and passed downstream, which is the whole craft.

Another failure: concurrent double-click

A user double-clicks within 80 ms. The browser generated K1 once and sent it twice. Both requests reach different API pods. The unique index lets one claim; the other receives 409. A well-behaved client treats 409 as “wait and retry the same key”, then receives the replayed success. A poorly written client treats any non-2xx as a failure and shows an error even though the payment succeeded, so test your client’s 409 handling explicitly.

Trade-offs, Gotchas, and What Goes Wrong

Idempotency is not free, and several well-intentioned designs fail in practice.

Replaying errors can be wrong. Stripe’s v1 behaviour of replaying a stored 500 is consistent but means a transient server fault becomes permanent for that key. If your system stores failures, distinguish deterministic failures (card declined, validation error: safe to replay) from transient ones (timeout, dependency down: should release the key so a retry re-executes). Getting this classification wrong either strands customers behind a cached outage or re-runs operations that should not repeat.

Key reuse across different operations. Teams sometimes reuse an order ID as the key for authorise, capture and refund. The second call then replays the first response. Always include the operation in the key or the scope.

Weak fingerprints. If the fingerprint ignores the amount, a client bug that reuses a key with a new amount silently returns the old charge. If it includes volatile fields, honest retries are rejected with 422. Test both directions.

TTL expiry during long retry horizons. A mobile app that queues payments offline for two days will outlive a 24-hour window. The retry then looks brand new and charges again. Align the retention window with the longest retry path you permit, or make the client expire its own queued intents sooner.

Clock and lease hazards. Lease-based recovery assumes the original worker is actually dead. A paused process (garbage collection, a stalled VM) can wake after its lease expired and complete a call that a second worker also made. Downstream references and fencing tokens, not wall-clock leases alone, are the defence.

Hot keys and storage amplification. Storing full responses multiplies write volume. Store only what a replay needs, compress large bodies, and partition by time so pruning is a metadata operation.

Security. Keys are client-controlled input. Validate format and length, scope them to the authenticated principal, and avoid logging them alongside secrets. The IETF draft’s security considerations warn about cross-client leaks from low-entropy keys, so enforce composite storage keys.

The anti-pattern list.

Checking “was this paid?” and then charging, without an atomic claim.
Generating a new key inside the retry loop.
Recording the key after the side effect.
Holding a database lock across the processor call.
Treating a timeout as a definitive failure and releasing inventory or reversing the order.
Assuming a message broker’s exactly-once setting covers an external API.

Idempotency does not fix business duplicates. Two distinct, deliberate purchases of the same item are not retries and must not be merged. Keys express intent identity; they cannot decide whether a user meant to buy twice. That belongs to product rules and fraud controls.

Practical Recommendations

For a new payment API, start from the premise that every write will be retried by someone you do not control, and design the key contract first, not as a patch.

Require an Idempotency-Key on every non-idempotent POST and PATCH, and reject requests that omit it with a 400 that links to documentation, matching the IETF draft. Scope keys per credential, store them in your system of record with a unique constraint, and claim before any side effect. Fingerprint the request and return 422 on mismatch. Replay stored responses byte-for-byte, and release the key for transient failures only. Publish your retention window.

Inside the platform, give each hop an identity: derive downstream references from the inbound key, use an outbox for events and an inbox for consumers, and key saga steps by (saga_id, step). Finish with a unique reference on every ledger posting and a reconciliation job that treats any duplicate as a defect to investigate rather than noise.

A short checklist to review a design against:

[ ] Key is client-generated, per intent, stable across retries.
[ ] Atomic claim via unique constraint before any side effect.
[ ] Fingerprint detects reuse with a different payload.
[ ] 409 for in-flight, 422 for mismatch, 400 for missing key.
[ ] Stored response replayed, with deterministic versus transient failures distinguished.
[ ] Lease or phased recovery for crashes between claim and completion.
[ ] Derived downstream reference passed to the processor.
[ ] Outbox plus inbox for internal events; idempotent saga steps and compensations.
[ ] Unique posting reference in the ledger.
[ ] Retention window longer than the longest retry path, and documented.
[ ] Reconciliation alerts on duplicates.
[ ] Concurrency tests that fire parallel identical requests and kill workers mid-flight.

Finally, test the failure paths on purpose. Chaos-style tests that drop responses after the processor call, kill workers between claim and completion, and replay each message twice will surface almost every bug described above before a customer does.

Frequently Asked Questions

What is an idempotency key in a payment API?

An idempotency key is a unique token the client sends with a request so the server can recognise retries of the same logical operation. The server records the key before performing the side effect, stores the response, and returns that stored response to any later request with the same key. The result is that a charge happens once even if the client sends the request several times because of timeouts or network errors.

How long should an idempotency key be valid?

Long enough to cover your longest realistic retry path, short enough to bound storage. Stripe’s v1 documentation says keys may be pruned once they are at least 24 hours old, after which a reused key creates a new request. Many internal systems use longer windows for payouts and batch work. Whatever you choose, publish it, because the IETF draft says servers should document their expiration policy.

What happens if the same key is sent with a different request body?

The server should reject it. The IETF draft specifies 422 Unprocessable Content for a key reused with a different payload, and Stripe likewise errors when parameters differ from the original request. This protects against client bugs where a key is accidentally reused for a different amount or customer. Detection depends on a request fingerprint, typically a hash of the canonicalised body, stored alongside the key.

Is exactly-once delivery possible in payments?

Not at the network level. Delivery guarantees are at-most-once or at-least-once. What systems achieve is exactly-once effects: at-least-once delivery combined with idempotent consumers, so the state change occurs once. Kafka transactions provide this inside Kafka, but external sinks such as card processors still need their own deduplication through keys, unique references or inbox tables.

Should the server return 409 or wait when a duplicate arrives mid-flight?

The IETF draft says a retry arriving while the original is still outstanding should receive a conflict error, and the client may retry without changing the request. Some servers instead wait briefly for completion and replay the result, which is friendlier but ties up resources. Returning 409 with a retry hint is simpler and safer under load, provided clients handle it correctly.

Do GET and DELETE requests need idempotency keys?

No. RFC 9110 defines GET, HEAD, OPTIONS, PUT and DELETE as idempotent methods, and Stripe advises against sending keys on GET and DELETE because they have no effect. Keys matter for POST and PATCH, where repeating the request can repeat the effect. Note that a DELETE can still return different status codes on repeat (for example 404 the second time) even though the server state is the same.