Data Pipeline Error Handling: Dead-Letter Queues, Retries and Monitoring

Production pipelines fail all the time: network partitions, schemas that drift between producers and consumers, malformed records from buggy upstream systems, external services that rate-limit or time out. A robust pipeline expects this. Its three goals: don't lose data, don't block progress on unrelated records, and surface problems for someone to fix.

Dead-letter queues: don't drop, don't block

When a record can't be processed you can drop it (silent data loss), block the pipeline (one bad record stops everything), or sideline it: put it in a dead-letter queue (DLQ) and keep going. Good records flow to the output; failed ones go to the DLQ with details of why.

Operators then triage the DLQ: transient failures (timeouts, rate limits) usually succeed when replayed; data bugs (malformed JSON) are fixed upstream and discarded; pipeline bugs mean fixing the code and reprocessing. Enrich every DLQ entry with the original record, error message, timestamp, pipeline version and the failing stage, and tag an error category (validation, serialization, downstream timeout) to prioritize.

Kinds of failure

Retrying the right way

• Immediate retry is usually wrong: the service timed out because it's overloaded, and instant retries add load and keep it down: a feedback loop of failure.
• Exponential backoff spaces attempts out (1 s, 2 s, 4 s, 8 s), giving the dependency time to recover.
• Jitter adds randomness. Without it, 100 workers that failed together retry together (a thundering herd) and fail again; with it, they spread out (1.3 s, 0.9 s, 1.7 s…).
• Circuit breaker: if a dependency keeps failing (say 50% failures over 10 seconds), stop calling it and fail fast; probe periodically and resume when it recovers. That protects both you and the struggling service. See circuit breakers.
• Cap retries: without a limit, records retry forever, burning resources and hiding the real problem.

retry_delay = base_delay * 2 ** attempt + random(0, base_delay)

A typical policy: try; on failure retry up to 3 times with exponential backoff and jitter; then write to the DLQ, alert, and move on to other records. Network blips recover in 2–3 retries; longer outages (deployments, database failover) may need more attempts or longer backoff. See retries and timeouts.

import json, random, time
from kafka import KafkaConsumer, KafkaProducer

MAX_RETRIES, BASE_DELAY = 3, 1.0
consumer = KafkaConsumer("orders", bootstrap_servers="localhost:9092",
                         group_id="orders-loader", enable_auto_commit=False)
producer = KafkaProducer(bootstrap_servers="localhost:9092")

class Transient(Exception): ...           # timeouts, 429s, connection resets

def process(record: dict):
    if record.get("quantity", 0) < 0:
        raise ValueError("negative quantity")       # permanent: retrying won't help
    write_to_warehouse(record)                       # may raise Transient

for msg in consumer:
    error = None
    try:
        record = json.loads(msg.value)
        for attempt in range(MAX_RETRIES + 1):
            try:
                process(record); error = None; break
            except Transient as e:
                error = e
                time.sleep(BASE_DELAY * 2 ** attempt + random.uniform(0, BASE_DELAY))
    except (ValueError, json.JSONDecodeError) as e:   # bad data: no retries
        error = e
    if error is not None:
        producer.send("orders.dlq", json.dumps({
            "original": msg.value.decode(errors="replace"), "error": repr(error),
            "category": type(error).__name__, "topic": msg.topic, "offset": msg.offset,
            "pipeline_version": "2026.10.1", "failed_at": time.time()}).encode())
    consumer.commit()                                  # progress continues either way

Monitoring: make failures visible

Pipelines have no user staring at them, so failures must surface through monitoring, not complaints:

Treat pipelines as production systems: health dashboards, alerts before users notice, on-call rotations with escalation paths, runbooks for common failures, and capacity planning. A silent pipeline failure that corrupts analytics for a week is often worse than a user-facing outage fixed in minutes. See observability.