Design a Webhook Service: Reliable, Secure Event Ingestion

Webhooks let one system notify another the moment something happens: instead of you polling an API, the provider pushes an HTTP request when an event occurs. Stripe sends payment_intent.succeeded, Shopify sends orders/create, GitHub sends push. This case study designs the receiving side: a service that accepts webhook events reliably, processes them, and keeps a record. (For the general concept see webhooks.)

Requirements

1. Accept API calls carrying event notifications (payment processed, order shipped), run the corresponding operations, and persist both the original event and the result for tracking, auditing and debugging.
2. Make sure accepted events get processed even when components fail; this data is often critical.

Out of scope: sending webhooks to others, a UI for browsing events.

• High availability: resilient to component failures.
• Low latency: under 200 ms end to end.
• At-least-once processing: every event the service accepts is processed at least once. See delivery semantics.

API

High-level design

Starting simple

It works, but the handler does everything: HTTP, business logic and persistence. If it crashes after processing but before saving, the event is lost, and a traffic spike ties up handlers doing slow work.

Buffering with a message queue

Put a message queue between the handler and the processing. The handler only receives the request and enqueues it; queue consumers do the heavy lifting and persist results. The queue holds events durably so nothing is lost while consumers are slow or down. Benefits: failure recovery, load buffering, and independent scaling.

1. Send event: the provider (say Shopify) POSTs to our_domain.com/webhook.
2. Enqueue: the handler puts it on the queue for asynchronous processing.
3. Return 200: only after enqueuing succeeds, so the provider knows the event was accepted even though it isn't processed yet.
4. Fetch: a consumer takes the event (one by one or in batches).
5. Process: run the business logic, e.g. update a payment's status.
6. Persist: store the original event, the outcome and status updates.
7. Write succeeds: the result is recorded for audits and debugging.
8. Dequeue: only now is the message acknowledged and removed.

Handling failure in each component

Monitor every component (latency, error rates, queue depth, consumer lag, resource saturation) with tools like Prometheus and Grafana or New Relic, and alert before providers start retrying en masse. See observability.

# Request handler: verify, enqueue durably, and only then return 200
import json, os
from flask import Flask, request, abort
from kafka import KafkaProducer

app = Flask(__name__)
producer = KafkaProducer(bootstrap_servers="localhost:9092", acks="all", retries=5)
SECRET = os.environ["WEBHOOK_SECRET"].encode()

@app.post("/webhook")
def receive():
    body = request.get_data()                                   # raw bytes: needed for the signature
    if not verify_signature(body, request.headers.get("X-Signature", ""), SECRET):
        abort(401)
    event = json.loads(body)
    future = producer.send("webhook-events", key=event["id"].encode(), value=body)
    future.get(timeout=2)                                       # wait for the broker to persist it
    return "", 200                                              # accepted; processing is async

Deep dive: securing the endpoint

The endpoint is public, so anyone could POST fake "payment succeeded" events. Three layers of defense:

1. HMAC signatures: provider and receiver share a secret key. The provider computes an HMAC of the payload (usually with a timestamp) using the secret and sends it in a header; the receiver recomputes it over the raw body and compares. Without the secret, nobody can forge a valid request.
2. IP allowlisting: accept requests only from the provider's published IP ranges (firewall or network rules). It shrinks the attack surface but depends on those IPs not changing.
3. Rate limiting: e.g. 100 requests per minute per client or IP, so an attacker can't overwhelm the service (denial of service). See rate limiting.

import hashlib, hmac, time

def verify_signature(body: bytes, header: str, secret: bytes, tolerance_s: int = 300) -> bool:
    """Header format like Stripe's: 't=1760950000,v1=<hex hmac>'."""
    try:
        parts = dict(p.split("=", 1) for p in header.split(","))
        ts, sig = int(parts["t"]), parts["v1"]
    except (ValueError, KeyError):
        return False
    if abs(time.time() - ts) > tolerance_s:                  # reject replays of old requests
        return False
    expected = hmac.new(secret, f"{ts}.".encode() + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)                 # constant-time comparison

Deep dive: duplicate requests

Duplicates come from provider retries (a lost 200), client issues or deliberate replays, and at-least-once processing makes them normal. Handle them idempotently. See idempotency.

• Idempotency keys: providers include a unique event_id (or request ID). Record each processed ID with a timestamp in a dedicated table; before processing, check it, and skip the event if it's already there. Doing the insert in the same transaction as the business change makes it race-free.
• Queue-level deduplication: some queues deduplicate by message ID within a window (e.g. AWS SQS FIFO's deduplication ID). Use the event ID as the message ID and size the window to how late duplicates can arrive. It reduces load but doesn't replace idempotent processing.

-- Consumer: process each provider event exactly once in effect
BEGIN;
INSERT INTO processed_events (event_id, received_at) VALUES ('evt_123', now())
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;                 -- no row returned → duplicate: ROLLBACK and ack the message
UPDATE invoices SET status = 'paid', updated_at = now() WHERE stripe_invoice_id = 'in_456';
COMMIT;                             -- then ack the queue message

Deep dive: out-of-order events

Network delays mean events can arrive in a different order than they were sent. Processing Stripe payments, you might receive invoice.paid before invoice.created. The handlers must not assume order:

• On `invoice.paid`: fetch the latest invoice from the Stripe API using the invoice ID in the event, so you have complete, current data; upsert it locally with status PAID; then trigger paid-invoice business logic.
• On `invoice.created` arriving later: compare its created time with the updated time of the data you already hold; it's older, so skip it.

The takeaways: correctness must not depend on event order or uniqueness; handlers should be idempotent and stateless; treat the provider's API as the source of truth instead of trusting local state; and use event IDs and timestamps to recognize outdated events.