How to Deduplicate Webhook Retries

Use the stable delivery id as a durable idempotency key, then separate receipt from processing so a crash does not lose work or repeat an external effect.

4 min read

Your receiver returned too slowly, the connection closed, and the same event arrived again. That is expected under at-least-once delivery. The bug begins only if the second POST sends a second alert, starts a second model run, or repeats another external effect.

Deduplication needs durable state. An in-memory set disappears in exactly the restart that is most likely to trigger a retry.

Use the delivery id, not the package id

RelayLink places the same UUID in:

  • X-RelayLink-Delivery
  • The JSON body's delivery_id

That UUID identifies one queued delivery to one receiver. It stays stable across attempts. Use it as the unique key in your receiver's idempotency store.

Do not deduplicate on package_id. One package may legitimately produce separate deliveries to multiple receivers, and your service may need to record each one. Do not use the signature either: each attempt gets an attempt timestamp, and the signature covers that timestamp, so a retry can carry a different signature.

After verification, require the header delivery id and body delivery id to match.

Separate receipt from processing

A reliable receiver first records that it owns the event, then processes it outside the HTTP request. A durable inbox row might contain:

  • Delivery id as a unique key
  • Event name
  • A digest of the exact body bytes
  • Receipt time
  • State such as pending, processing, or completed
  • Last processing error

The callback path is:

  1. Verify the signature and timestamp.
  2. Parse the verified body and require its event and delivery id to match the headers.
  3. Compute a digest over the exact body bytes.
  4. In one transaction, insert the inbox row and a work-queue row together.
  5. Commit and return 2xx.

If the unique insert reports that the delivery already exists, compare the stored digest. The same digest means the event was already accepted: return 2xx without adding another work item. A different digest under the same id violates the wire contract; quarantine it rather than processing or silently merging it.

Putting the inbox row and queue item in one commit is the useful part. A row that says “seen” without any work behind it is only a durable way to lose the event.

The mark-before failure window

The simplest mark-before algorithm is:

insert delivery id
perform work

It blocks duplicates, but a crash between those lines leaves a permanent idempotency record and no completed work. Every retry sees the record and exits successfully. The event is lost from the application's point of view.

A pending inbox plus durable queue closes that window. The queue worker can resume after a process restart. The receipt means “accepted for processing,” not “effect completed.”

The mark-after failure window

The opposite algorithm is:

perform work
insert delivery id

It avoids marking unfinished work as complete, but a crash after the effect and before the insert lets the retry perform the effect again. That is dangerous for sends, charges, deployments, or any action that is not naturally repeatable.

Moving the final insert closer to the effect reduces the window but does not remove it when the effect belongs to another system. Use the delivery id as an idempotency key in that downstream call when possible. Otherwise record an intent first, make a reconciler inspect uncertain outcomes, or design the effect so repeating it reaches the same state.

Byte-identical retries help diagnosis

RelayLink serializes the body once when it creates the delivery. A retry uses the same delivery id and byte-identical body. The timestamp and signature headers may differ because they are generated for each attempt.

Keeping the digest avoids putting sender metadata or a raw body in routine logs while still detecting a contract violation. Do not log the raw body, signing secret, or full receiver URL; a webhook path may itself contain a token.

Always verify each retry's own signature before looking up the idempotency record. Otherwise anyone who learns a delivery id could make your endpoint return success to an unauthenticated request.

Make the MCP fetch part of the worker

A package.received callback contains no briefing. The queued worker authenticates to the supplied mcp_url and calls get_package with package_id.

Keep agent execution after that authenticated fetch, not in the callback. Mark the delivery completed only when your intended workflow has reached its defined completion point. If processing fails, retry it from your own queue without asking RelayLink to repeat the HTTP delivery.

This gives each layer one job: RelayLink retries transport, the receiver deduplicates receipt, and the worker owns processing recovery.

Frequently asked questions

Which value should I use to deduplicate RelayLink webhooks?
Use X-RelayLink-Delivery, mirrored as delivery_id in the body. It stays stable across retries of the same queued delivery.
Can I use the signature as the idempotency key?
No. RelayLink creates a timestamp and signature for each POST attempt, so they may change even though the delivery id and JSON body remain the same.
Should I mark a delivery processed before or after doing the work?
Neither simple choice closes both crash windows. Durably record the delivery and enqueue work atomically, then make the worker's downstream effects idempotent and record completion.