Lesson 5 of 8 · 48 min

Idempotency, exactly-once illusions, and webhooks

Idempotency keys, concurrent claim, webhook verification and dedupe, at-least-once consumers, and effect-once for billing and LLM runs.

Lesson 5 · Idempotency

Keys, webhooks, at-least-once reality

Exactly-once is a product of design, not a broker checkbox

Networks retry. Users double-click. Webhooks redeliver. Queues are at-least-once. Idempotency means performing the effect once even when the request arrives many times. Payments, LLM billing, and seat provisioning all fail interviews and prod when you shrug this off.
Say “effect-once” out loud: the side effect (charge, enqueue, provision) happens once; duplicates return the original result. That requires a stable key, durable storage of outcomes, and careful concurrency (unique constraints).

Idempotency keys (Stripe pattern)

Client sends Idempotency-Key: … on POST. Server stores key → response (or in-progress). Retries with same key + same payload replay the result. Different payload with same key → conflict. Keys expire after a window (e.g. 24h) — document it.
python
1# Idempotency store (sketch)2# UNIQUE(idempotency_key, tenant_id)3def create_run(tenant_id, key, body):4	row = db.get_idem(tenant_id, key)5	if row:6		if row.body_hash != hash(body):7			raise Conflict('idempotency key reuse with different body')8		return row.response  # replay9	try:10		with db.transaction():11			db.insert_idem_in_progress(tenant_id, key, hash(body))12			run = db.insert_run(...)13			resp = serialize(run)14			db.complete_idem(tenant_id, key, resp)15			return resp16	except UniqueViolation:17		# concurrent first writers — wait/reload replay18		return wait_for_idem(tenant_id, key)

What to key on

Prefer client-generated UUIDs for user-driven POSTs. For webhooks, use provider event id. For queue consumers, use business id (run_id, invoice_id) not a random message id if redelivery mints new message ids. Key scope includes tenant so tenants cannot collide.

Webhooks — verify, dedupe, order loosely

Verify signatures (HMAC/timestamp window). Treat deliveries as at-least-once. Dedupe on event id. Do not assume total order — design state machines that tolerate out-of-order (ignore stale transitions). Return 2xx quickly after durable accept; process heavy work async.
python
1# Webhook receiver2def handle_provider_webhook(headers, raw_body):3	verify_signature(headers, raw_body)  # else 4014	event = parse(raw_body)5	if db.seen(event.id):6		return 200  # duplicate delivery7	with db.transaction():8		db.mark_seen(event.id)9		db.enqueue_outbox(event)10	return 20011# Slow work happens in workers; keep webhook path fast.

At-least-once consumers

Pattern: begin work → side effect → mark processed in the same durable store when possible. If you mark processed before side effect, crashes skip work. If you side effect before mark, duplicates re-enter — side effect must be safe. Outbox + idempotent handlers is the boring architecture that works.

LLM and billing specifics

Token spend is money. A duplicated run create can double cost. Use keys on create-run and on provider requests when supported. Store provider request ids. Reconcile usage ledgers with unique (run_id, meter_type) lines so double consumers do not double-bill customers.

Testing idempotency

Automated tests: send same key twice → one row. Concurrent double POST → one row. Webhook redelivery → one transition. Chaos: kill worker mid-flight and redeliver. If you cannot test it, you do not have it.
Load tests should include a retry storm scenario: 10% of clients re-POST the same key under latency. Watch unique constraint contention and response times. Idempotency that works in a unit test but collapses under parallel first-writers is unfinished.

Partial failures and saga-shaped flows

Multi-step: create run → charge hold → call provider → finalize charge. If step 3 fails, compensate or leave a durable “needs reconcile” state — do not pretend a distributed transaction exists across Stripe and OpenAI. Idempotency keys on each external step plus a state machine beat hand-wavy two-phase commit talk.

Human replays and admin tools

On-call will re-drive a webhook or requeue a run. Admin “replay” must use the same idempotent paths as production consumers — a special code path that skips dedupe will double-send. Button in admin UI should call the worker entrypoint with the original business id, not invent a new one.

Idempotency windows and GDPR deletes

Keys expire; ledger rows may need retention policies separate from raw prompts. When a user deletion request arrives, define whether historical billing lines remain (often yes, for finance) while prompt bodies go. Idempotency stores should not become an accidental PII lake — store hashes of bodies when you only need equality checks.
text
1IDEMPOTENCY CHECKLIST (ship with the feature)2[ ] Client can supply a stable key (header or body)3[ ] Unique constraint on (tenant, key)4[ ] Concurrent first-writers handled (wait/replay)5[ ] Body hash compared on reuse6[ ] TTL/retention documented7[ ] Webhook event ids deduped8[ ] Worker side effects keyed by business id9[ ] Tests: sequential retry + concurrent double POST10[ ] Admin replay uses same keys
If any checkbox is missing on a money or token path, treat it as a launch blocker. Product can ship without a fancy UI; it cannot ship with double-charge under retries and call that an edge case.

Interview answers — idempotency

  1. 01Q: Exactly-once? Usually at-least-once + idempotent handlers + durable dedupe keys producing effect-once.
  2. 02Q: Idempotency-Key header? Client-stable key for POST; server stores response; replays on retry; conflict if body differs.
  3. 03Q: Webhook duplicates? Signature verify + event id unique insert + fast 200 + async process.
  4. 04Q: Where store keys? DB with unique constraint (and TTL job) or Redis with care for durability needs on money paths.
  5. 05Q: Message id vs business id? Prefer business id when infrastructure redelivers with new envelope ids.
  6. 06Q: Unsafe retries? Non-idempotent POST without keys; non-conditional status transitions; side effects outside transactions without dedupe.
  7. 07Q: Out-of-order webhooks? Versioned state machine; ignore stale; never assume arrival order equals causal order.
  8. 08Q: How long keep keys? Document retention (e.g. 24h–72h) balancing storage vs client retry windows.
  9. 09Q: Provider calls? Pass vendor idempotency keys when available; map your run_id to provider request metadata.
  10. 10Q: Double-click submit? UI disable helps UX; server idempotency is the real fix.
articleStripe Engineering — IdempotencyStripedocsStripe Docs — Idempotent requestsStripedocsAWS — API Gateway & Lambda idempotency powertoolsAWS PowertoolsdocsWebhook best practices (Stripe)Stripe

Checkpoint

Client retries POST /charges with the same Idempotency-Key after a timeout. Server behavior?

AAlways create a second charge so accounting has more rows to reconcile later.BReturn the original charge result if the key exists with the same body; do not charge twice.CDelete the key and require the client to use a new key every retry.
Sign up free to answer and see why

Checkpoint

Two concurrent requests arrive with the same new idempotency key. What makes this safe?

AA unique constraint (or equivalent atomic claim) so only one writer creates the resource; the other waits/replays.BSleep 50ms in one thread so the other finishes first in practice.CDisable HTTP/2 so concurrency cannot happen.
Sign up free to answer and see why

Checkpoint

Partner webhook delivers the same event id three times. Correct receiver pattern?

AProcess side effects every time — partners only redeliver when something new happened.BVerify signature, insert event id uniquely, ignore duplicates with 2xx, process once via durable pipeline.CReturn 500 on duplicates so the partner stops sending forever.
Sign up free to answer and see why

Checkpoint

Why is “Kafka exactly-once” insufficient for “email users once”?

ABecause email servers reject all messages from Kafka by policy worldwide.BExternal email APIs are outside the broker transaction; reprocessing can resend unless you dedupe with a business key.CExactly-once means consumers never crash, so one attempt always finishes.
Sign up free to answer and see why

Checkpoint

Best idempotency key for an LLM run creation from a product UI?

AServer timestamp string rounded to the current minute so keys are easy to guess.BClient-generated UUID per submit intent, sent as Idempotency-Key, scoped by tenant on the server.CThe model name (gpt-x) because each model should only run once globally.
Sign up free to answer and see why

Can you implement idempotent POSTs and webhook dedupe with a unique-constraint story?

New to itGetting thereConfident

Takeaways

  • Assume at-least-once everywhere networks and queues exist.
  • Idempotency keys + unique constraints implement effect-once.
  • Webhooks: verify, dedupe event ids, ack fast, process async.
  • External side effects need their own dedupe — broker EOS is not enough.

Next: caching patterns, invalidation, N+1, and cursor pagination performance.

Sources

Free to read · better with Enzo

Learn it with Enzo

Save your progress, answer the checkpoints, and let Enzo quiz you on what you just read.

Continue with EnzoSign up free