Secure Webhooks: Verify, Replay-Protect, Monitor

Webhooks let payment platforms, identity providers, commerce systems, and internal services notify an application when something changes. They are useful because the receiving system does not need to poll. They are risky for the same reason: an internet-facing endpoint can cause a business action.

The safe model is simple: verify the sender's signed request before processing it, reject stale or replayed deliveries, make the business action idempotent, and observe every decision without storing secrets. A WAF, CDN, or rate limit can reduce unwanted traffic, but none of them proves that a webhook payload is authentic.

Start with a webhook threat model

Document each endpoint before configuring controls. Record its provider, event types, authentication method, expected content type and body size, retry behavior, peak volume, and the business action it can trigger. A payment-confirmed event, password-reset event, and catalog update should not all receive the same policy.

At a minimum, design for these failures:

• A forged request that claims to come from a trusted provider.
• A valid event replayed after it has already been processed.
• Duplicate or out-of-order deliveries caused by normal retries.
• An old event delivered after the business state has changed.
• A malformed, oversized, or unexpected payload.
• A burst that exhausts the receiver, queue, or downstream dependency.

Verify the signed request before parsing it

Webhook providers normally sign a representation of the raw request body with a shared secret or private key. Follow the provider's exact scheme. Do not invent a compatible-looking version: the signed components, encoding, timestamp format, and key rotation model are part of the security contract.

A safe verification sequence

1. Require HTTPS and validate the TLS certificate on every network hop.
2. Read the raw body exactly once, before JSON parsing or whitespace normalization changes it.
3. Extract the provider's signature and timestamp headers.
4. Recreate the signed message according to the provider's documentation.
5. Compute or verify the signature with the current secret or public key.
6. Compare secrets in constant time where the platform does not already provide a safe verifier.
7. Reject timestamps outside a short, documented acceptance window.
8. Only then parse the payload and decide whether the event type is authorized for this endpoint.

Keep signing keys in a secrets manager, never in source code or browser-delivered configuration. Support overlap during key rotation so a provider can move from an old key to a new one without losing valid deliveries. A signature confirms message integrity and the key holder; it does not authorize every event type or business operation automatically.

Stop replays and duplicate effects

A valid request can be sent more than once. Providers retry when they do not receive a timely successful response, and an attacker who captures a valid delivery may attempt a replay. Use both freshness and durable idempotency.

• Freshness: reject an old signed timestamp. The window should cover realistic clock drift and transit delay, not hours or days by default.
• Replay tracking: store an event ID, signature nonce, or provider delivery ID long enough to reject a repeated request within the threat window.
• Idempotent business actions: record the processed event ID and make downstream writes safe to repeat. A duplicate payment event must not create a duplicate shipment or credit.
• Ordering: do not assume events arrive in sequence. Fetch the current state from the authoritative API when an event can be superseded.

Use a database constraint, durable queue, or transactional idempotency store rather than an in-memory process map. Multiple application instances and restarts make local memory unreliable.

Validate the payload and limit the work

After sender verification, treat the payload as untrusted input. Validate its schema, required fields, enum values, and size before it reaches expensive processing. Allow only expected methods and content types. Reject a request body that is too large before parsing it.

Do not make a webhook route broadly permissive because a provider needs to call it. A browser challenge may be appropriate for suspicious human traffic, but it will break machine-to-machine callbacks. Instead, use narrow route rules, a provider-aware rate limit, body-size limits, and an explicit signature verifier in the application.

For a high-value endpoint, separate receipt from processing. Verify the event, persist a minimal durable record, return a prompt success response, then process it asynchronously. This reduces provider retries and gives the team a controlled place to handle temporary dependency failures or dead-letter events.

Protect the edge and origin

Place the endpoint behind the same controlled delivery path as the rest of the application, but do not rely on the edge as the only control:

• Expose only the required hostname and path.
• Allow the exact methods and content types the provider uses.
• Apply an endpoint-specific request-rate and concurrency budget.
• Keep the origin private or default-deny direct public access.
• Authenticate the edge-to-origin connection and trust forwarded client headers only from the known proxy.
• Log the edge decision, application request ID, signature result, event ID, and processing result together.

See API Protection at the Edge, Rate Limiting, and How to Protect Your Origin Server for the surrounding controls.

Test the failure modes before production

Use a non-production endpoint and provider test deliveries to exercise valid and invalid cases. Test an invalid signature, an expired timestamp, a duplicate event, a missing required field, an oversized body, an unknown event type, a rotated key, a slow queue, and a downstream failure. Confirm that a valid retry does not repeat a business effect.

Do not use production secrets in logs or test fixtures. Redact authorization headers, signatures, bodies containing personal data, and customer identifiers. Keep only the fields needed to investigate a delivery: timestamp, request ID, provider, endpoint, event type, verification outcome, response status, and a safe correlation ID.

Operational checklist

Before an endpoint goes live, confirm that the provider's official signature-verification library or algorithm is in use; replay protection and durable idempotency are tested; event types are allowlisted; body and concurrency limits exist; retries are understood; direct origin access is blocked; secrets can rotate; and alerting distinguishes rejected traffic from failed processing.

Authoritative references

• OWASP Web Service Security Cheat Sheet
• OWASP API10: Unsafe Consumption of APIs
• GitHub: Validating webhook deliveries
• Cloudflare: Protect APIs