RelayLink Webhook Failure Runbook

The package arrived, but the receiver did nothing. Do not begin at the model. First establish whether RelayLink queued a call, whether the endpoint accepted it, and whether your worker failed after acceptance.

Use one delivery id to trace each boundary. Keep sender, topic, URL path, secrets, and package content out of routine incident logs.

Start with the account page

Open /account/webhooks and inspect the subscription before changing anything. Confirm that it is active, then note:

• the latest delivered or failed time;
• last HTTP status, if a response arrived;
• last transport error.

If repeated failures switched the subscription off, its disabled row also shows that reason and the consecutive-failure count. The account page summarizes the latest outcome; it is not a delivery history. Take the delivery id used below from the receiver's authenticated request and durable inbox.

Choose Send a test event. The page queues webhook.test; it does not call the endpoint synchronously. Wait briefly, refresh, and inspect the updated result.

The test payload contains event, delivery_id, sent_at, and mcp_url. It has no package_id. A handler that validates package fields before branching on the authenticated event will fail its own test. A successful test proves the public route and signature path for that subscription, not that the package-fetch worker works.

Classify the HTTP failure

If the endpoint sees no request, check the registered final URL, public DNS, TLS certificate, firewall, and ingress route. RelayLink requires HTTPS and does not follow redirects. A browser reaching a URL after a 301 or 302 does not prove RelayLink will.

If a response arrived, use its class:

• 2xx: RelayLink accepted the delivery. Continue at your queue and worker.
• 3xx: Register the final HTTPS endpoint.
• 4xx: Check method, route, body limits, authentication, timestamp policy, and schema validation.
• 5xx or no response: Check application failure, DNS, TLS, connection, and timeout.

The current attempt timeout is five seconds. The callback should verify, deduplicate, persist or queue, and return. Do not call get_package, run a model, post to another platform, or wait for a durable workflow result before answering.

Reproduce HMAC with the exact body

Most “the secret is correct” failures are byte mismatches. Capture safe diagnostics in a controlled environment and verify in this order:

1. Read X-RelayLink-Timestamp as the exact signed timestamp text.
2. Read the untouched request body bytes.
3. Build {timestamp}.{body}.
4. Compute HMAC-SHA256 with the full whsec_ subscription secret.
5. Encode lowercase hexadecimal and prefix sha256=.
6. Compare with X-RelayLink-Signature in constant time.
7. Check the authenticated timestamp against the replay window.
8. Parse, then match body event and delivery id to the headers.

Do not verify a parsed JSON object serialized again. Whitespace, property ordering, escaping, newline conversion, or character encoding can change the bytes. Make sure a proxy or request-body middleware does not consume or normalize the stream first.

Never paste the secret, raw body, OAuth token, API key, whole signature, or full endpoint URL into a ticket. A webhook path may contain a credential.

Trace a 2xx into your worker

A 2xx means the receiver claimed responsibility. Find the durable inbox row by X-RelayLink-Delivery, then confirm that the queue item committed with it.

Check these states in order:

delivery accepted
work item available
worker claimed it
package fetched with the intended user's MCP credential
workflow or notification completed

RelayLink retries use the same delivery id. A duplicate should return success without creating another work item. If the inbox row exists but no job does, the receiver has a mark-before crash window. Insert both atomically.

For package.received, first require mcp_url to match the RelayLink endpoint pinned to that receiver and account. The worker then calls get_package with package_id and a separate account credential; it never sends that credential to an arbitrary URL from the body. The webhook secret cannot fetch content. Treat sender and topic as untrusted metadata, and treat the fetched package as third-party content rather than instructions.

Understand retries and automatic disablement

By default, one failed delivery receives up to six attempts over about an hour. RelayLink then abandons that delivery. The delivery id and JSON body stay the same across attempts; the attempt timestamp and signature can change.

Automatic disablement uses another counter. Every failed attempt increments the subscription's consecutive failures, and any successful 2xx resets it. At 30 consecutive failed attempts across deliveries, RelayLink disables the subscription and abandons its queued pending or retrying deliveries.

Fix the receiver first. Before turning it back on, make sure the account will still have no more than five active subscriptions; retire an obsolete or replacement receiver if necessary. Then turn it on and send a fresh test. Re-enabling resets the failure count, but it does not replay abandoned deliveries or reconstruct events that arrived while the receiver was disabled. New package events are queued after the subscription is active again.

Reconcile the gap with check_inbox, fetch any unhandled packages under the user's credential, and claim them against the same package-level workflow store your normal path uses.

Close the incident with evidence

Record the failing delivery id, safe error category, first and last failure times, root cause, receiver change, successful test id, and the inbox reconciliation range. Do not call a green test proof that old events replayed.

Keep a lower-frequency inbox poll if missing one arrival would matter. The webhook is the fast path; reconciliation is what covers exhausted retries, automatic disablement, and failures after your receiver already returned 2xx.