Payment webhooks the hard parts: idempotency, retries and reliability

The mental model: at-least-once, not exactly-once

Every serious payment provider — Stripe, Trust Payments, Norbr, HiPay — delivers webhooks at-least-once. The same event can arrive twice; events can arrive out of order; and if your endpoint returns anything but a quick 2xx, the provider retries. If you treat a webhook as a one-time, in-order, exactly-once message, you will eventually double-charge, double-fulfil, or double-email someone.

Step 1 — Verify the signature

Always verify the provider's signature against the raw request body before doing anything. This is also why webhook routes need the unparsed body — verifying against a re-serialized JSON object will fail.

Step 2 — Deduplicate by event id

Record every processed event id. If you've seen it, acknowledge with 2xx and stop. This single table turns at-least-once delivery into effectively once-handled.

CREATE TABLE processed_events (
  id text PRIMARY KEY,            -- provider event id
  processed_at timestamptz DEFAULT now()
);

Step 3 — Make the handler idempotent

Even with dedup, design the effect so that running it twice is harmless: upsert instead of insert, check current state before transitioning, key external actions by the event id. Idempotency is a property of the handler, not just the dedup table.

async function onPaymentSucceeded(evt) {
  if (await seen(evt.id)) return ack();           // dedup
  await db.tx(async (t) => {
    const inv = await t.invoice(evt.data.invoiceId);
    if (inv.status === 'paid') return;            // idempotent guard
    await t.markPaid(inv.id, evt.id);
    await t.recordEvent(evt.id);
  });
  await queue.add('send-receipt', { invoiceId: evt.data.invoiceId });
}

Step 4 — Acknowledge fast, work async

Return 2xx as soon as you've durably recorded the event; push slow work (emails, PDFs, downstream API calls) to a queue (I use BullMQ on Redis). A slow webhook handler causes provider timeouts and a storm of retries — exactly what you don't want during a payment spike.

Key takeaways

• Treat webhooks as at-least-once and possibly out of order.
• Verify signature → dedupe by event id → idempotent handler → ack fast → queue the slow work.
• Add periodic reconciliation; never assume every webhook was delivered.