Webhook Signature Verification: A Safe Debugging Checklist

Webhook signature verification is supposed to answer a narrow security question: did the sender that knows the shared secret authenticate these exact bytes? In practice, a failed check can pull an API support team into hours of guesswork. The secret looks right, the JSON parses, and the sender reports a successful delivery, yet the receiver keeps returning an unauthorized response.

The fastest safe fix is not to print every header, payload, and secret into a ticket. It is to isolate the signature inputs one by one while keeping credential material out of chat, logs, screenshots, and model context. This checklist gives support engineers and integration owners a repeatable path from a vague signature mismatch to a verified cause and a durable prevention step.

Most webhook schemes use a message authentication code such as HMAC. The sender combines a secret with a defined byte sequence, computes a digest, and places the result in a request header. The receiver independently builds the same byte sequence with the same secret and algorithm. Matching digests show that the message was authenticated and was not modified after signing. RFC 2104 defines HMAC; each provider's webhook documentation defines the provider-specific signing recipe around it.

That last distinction matters. HMAC-SHA256 is not a complete webhook protocol. One sender may sign only the raw body. Another may sign a timestamp, delimiter, and raw body. Header formats, hexadecimal or Base64 encoding, version prefixes, tolerance windows, and secret scopes can all differ. Copying a verifier from a different provider can produce a valid HMAC implementation that never validates the delivery in front of you.

Begin with one failed delivery, not a stream of loosely related failures. Record the provider's delivery identifier, event type, receiving endpoint, HTTP status, receipt time, content type, signature-header presence, and application release. Capture the raw request body in a restricted diagnostic path only if your data policy permits it. If the payload can contain customer or account data, keep that capture short-lived and access controlled.

Do not request the signing secret in a support ticket. Do not paste it into a public replay tool. Do not log the full signature header by default either; while a digest is not the secret, collecting authentication artifacts without purpose expands the incident surface. Ask the customer or operator to confirm a masked prefix, endpoint identity, environment, or secret version instead. Those facts usually prove whether both sides selected the same credential without exposing it.

The most common implementation mistake is verifying a reconstructed payload instead of the bytes the sender signed. Framework middleware may parse JSON before the route runs. Re-serializing that object can change whitespace, property order, escaped characters, Unicode representation, or the final newline. The JSON still means the same thing to an application, but the byte sequence is different, so the digest must differ.

GitHub's webhook validation guidance explicitly warns that the payload and headers must not be modified before verification. Stripe's troubleshooting guide similarly centers the request body as received and calls out framework-specific body handling. The safe pattern is to read the body once as bytes or an exact string, verify the signature against that raw value, and only then parse it into a domain object. If global middleware consumes the stream first, configure a raw-body exception for the webhook route.

Prove this layer with byte-level evidence. Compare the receiver's captured byte length and SHA-256 body fingerprint with a locally stored delivery fixture. Test a payload containing non-ASCII characters, escaped slashes, nested objects, and a trailing newline. A verifier that works only for compact ASCII JSON is not ready for production webhook traffic.

Next, write the signing recipe beside the verifier. Name the header, version, hash algorithm, delimiter, timestamp unit, encoding, and exact signed content. For GitHub's recommended path, the receiver reads `X-Hub-Signature-256`, computes HMAC-SHA256 over the payload using the webhook secret, and compares against the prefixed result. Stripe's header can contain a timestamp and multiple signatures; its library reconstructs the provider's signed payload and handles those details for you.

Woes outbound webhooks provide another concrete recipe. The implementation signs `<timestamp>.<raw JSON body>` with HMAC-SHA256 and sends `Woes-Signature: t=<unix>,v1=<hex digest>`. A Woes receiver therefore must not sign only the JSON, use milliseconds, substitute a different delimiter, or decode the digest as Base64. This is an example of why the sender's documentation and an observed test vector should be the source of truth.

Create a deterministic fixture with a fixed secret, fixed timestamp, and fixed raw body. Store the expected header alongside it in tests, using a clearly fake secret. That fixture separates transport and framework problems from cryptographic code. If the unit test fails, the recipe is wrong. If it passes while production fails, investigate body capture, configuration selection, proxies, or deployment state.

A byte-perfect message can still fail when the receiver uses the wrong secret or interprets the digest incorrectly. Confirm that the endpoint is reading the secret assigned to that exact provider account, webhook endpoint, workspace, and environment. Test and production endpoints commonly have different credentials. A recently recreated endpoint may look identical in the dashboard while owning a new secret.

Check the output representation before comparing. A hexadecimal digest is not interchangeable with Base64, and a header prefix such as `sha256=` or `v1=` is metadata rather than digest bytes. Decode both values into equal-length byte arrays, reject malformed input, and use a constant-time comparison primitive. Node's `timingSafeEqual` documentation notes that the surrounding code also needs timing-safe design; a length guard is still necessary because the function requires equal-size inputs.

Avoid a verifier that accepts whichever algorithm the request names. The integration contract should choose the supported algorithm and version. Otherwise, an attacker may be able to steer verification toward a weaker or unintended path. Treat unknown versions, duplicate malformed fields, invalid encodings, and unexpected digest lengths as authentication failures with sanitized diagnostics.

A valid HMAC proves knowledge of the secret and integrity of the signed message. By itself, it does not prove that the message is new. If the signature includes a timestamp, enforce a bounded tolerance after parsing it safely. Woes's verification helper defaults to a five-minute window. Stripe's signature scheme also uses a timestamp and tolerance concept. Choose a window that accounts for realistic delivery delay and clock skew without leaving replay acceptance open indefinitely.

Timestamp validation is only one replay control. Persist the provider's delivery or event identifier after successful authentication, then make event handling idempotent. A legitimate provider may retry when your endpoint times out or returns a failure. The correct response is usually to authenticate every attempt but apply the business effect once. Do not mark an event as processed before the durable transaction succeeds, or a crash can turn a temporary failure into silent data loss.

Keep clocks synchronized and surface skew as an operator diagnostic rather than telling customers to weaken the window. If many otherwise correct deliveries fail on timestamp tolerance at once, compare host time with a trusted source, check whether seconds were mistaken for milliseconds, and inspect queue delay before increasing the threshold.

Once the verifier is decomposed, the failure pattern usually points to a small set of causes. A missing signature header suggests that the sender has no secret configured, the wrong webhook endpoint is being inspected, or an intermediary removed the header. A mismatch for every payload suggests a wrong secret, algorithm, encoding, or signing recipe. A mismatch only for formatted or multilingual JSON strongly suggests raw-body mutation. A failure only after a deployment suggests middleware order, environment configuration, or secret-version drift.

A timestamp failure with a matching digest points to clock skew, queue delay, wrong time units, or an intentionally replayed delivery. Intermittent failures during rotation point to a cutover problem: some senders or receivers still use the previous secret while the other side has already switched. Build sanitized reason codes such as `signature_missing`, `timestamp_invalid`, `timestamp_out_of_range`, `digest_malformed`, and `signature_mismatch`. Keep the external response generic while making those categories visible to authorized operators.

Rotate a webhook secret when exposure is suspected, when policy requires it, or when ownership changes. Do not use rotation as the first debugging move: it destroys a useful reference point and can create a second outage if consumers are not ready. First prove which endpoint and secret version are active. Then follow the provider's supported rotation model, which may allow old and new secrets to overlap for a limited period.

OWASP's secrets guidance emphasizes lifecycle controls including creation, rotation, revocation, expiration, least privilege, and auditing. Apply those controls to webhook secrets. Store them in a dedicated secret system or encrypted application store, restrict who can reveal or replace them, never return them in routine reads, and audit administrative changes without recording plaintext. After cutover, revoke the old secret, run a signed test delivery, and confirm both success and intentional rejection with the retired value.

Woes applies these boundaries to its customer-configured outbound webhooks. Workspace admins can create HTTPS endpoints, select supported events, copy a generated signing secret once, send a test delivery, review delivery history, disable an endpoint, or rotate its secret. The browser-facing endpoint record carries safe metadata and a masked prefix; the encrypted secret lives in a separate server-only table and is decrypted only for delivery.

Each generic delivery uses the timestamped HMAC-SHA256 recipe described above and includes a delivery ID, event type, event ID, and JSON envelope. Endpoint, secret, and delivery rows carry workspace IDs, and database checks require child records to belong to the same workspace as their endpoint. Woes also requires HTTPS webhook URLs and rejects embedded URL credentials. These are implementation facts, not a claim that signature verification replaces endpoint authorization, payload validation, rate limiting, monitoring, or incident response.

For broader API troubleshooting, Woes keeps stored API authentication separate from ingested source content and applies credentials server-side for guarded live calls. Results are redacted before they are shown in support-agent context. That separation supports a safer debugging posture: collect the smallest useful fact, verify behavior through a controlled path, and hand off when account-specific or security-sensitive evidence is still missing.

Turn the investigation into a regression test before closing it. Your test suite should include the provider's official example when available, your own fixed test vector, a modified body, a wrong secret, a malformed header, a stale timestamp, an unknown version, Unicode content, a trailing newline, and a duplicate delivery. Exercise the request adapter as well as the pure verifier so middleware regressions cannot hide behind passing crypto unit tests.

In production, measure accepted, rejected, retried, and duplicate deliveries by endpoint and reason category without putting secrets or raw sensitive payloads into metrics. Alert on a sudden rise in missing signatures, mismatches, stale timestamps, or endpoint failures. Keep a short operator runbook that names the signing recipe, secret owner, safe evidence fields, test-delivery path, rotation procedure, and escalation contact.

When webhook signature verification fails, resist the urge to rotate credentials, disable checks, or collect every sensitive artifact. Reduce the problem to one delivery and one documented signing contract. Preserve the raw bytes, reconstruct the signed message exactly, confirm the endpoint-specific secret and encoding, compare safely, and then add freshness and duplicate controls.

That method resolves the ordinary bugs—parsed bodies, wrong environments, milliseconds, prefixes, encodings, and rotation drift—without turning a support conversation into a secret-sharing channel. It also leaves the integration stronger than you found it: covered by fixtures, observable through safe reason codes, and supported by a rotation plan that operators can execute with confidence.