Lambda vs Kappa Architecture

Batch processing is complete but slow; stream processing is fast but struggles with late data and corrections. Some systems need both at once. An e-commerce finance team needs daily sales accurate to the penny (they reconcile against payment processors), while operations needs orders per minute live, to spot outages now. All-batch makes the dashboard useless for incidents; all-stream risks missing late events in financial reports. Lambda and Kappa are the two classic architectures for getting both.

Lambda architecture: three layers

Popularized by Nathan Marz, Lambda runs batch and stream in parallel and merges their results:

• Batch layer: keeps the complete, immutable master dataset (HDFS or S3) and periodically recomputes views from scratch, e.g. a nightly job over all history. Slow (hours) but complete and accurate; late events and corrections are picked up on the next run.
• Speed (real-time) layer: stream processing that computes incremental updates since the last batch run. Fast (seconds to minutes) but only covers events that have already arrived.
• Serving layer: answers queries by merging the batch view with the speed layer's delta. When a new batch finishes, its results replace the speed layer's estimates and the speed layer resets.

Worked example: sales by category

1. Batch (2 AM daily): a Spark job reads a year of order events from S3, aggregates by category (Electronics $45.2M, Clothing $28.1M, …) and writes a batch view stamped 2024-01-15 02:00:00.
2. Stream (continuous): a Flink job consumes the Kafka orders topic and keeps running totals since 2 AM in Redis (Electronics $180K, Clothing $95K), counting only events that have arrived.
3. Query (any time): "Electronics sales?" → batch $45.2M + speed $180K = $45.38M.
4. Next 2 AM: the batch recomputes Electronics as $45.4M including all of yesterday, the batch view updates, the speed layer's counters reset to zero, and queries now combine $45.4M with new deltas.

The maintenance problem

• Duplicate logic: every business rule exists twice (say Spark SQL and Flink Java). Edge cases diverge, and users see different numbers depending on when they query relative to the batch run.
• Dual infrastructure: two clusters, two technologies, two sets of deployments, monitoring and failure modes. A batch bug means fix and rerun; a speed-layer bug means fix, redeploy and maybe reconcile history.
• Synchronization: the serving layer must track which batch run is current, which speed data belongs with it, and the handoff when a new batch lands. That logic is error-prone.

Lambda still makes sense when compliance requires matching existing batch systems (the real-time layer is treated as an approximation that batch periodically corrects), when mature batch infrastructure already exists and you're adding streaming incrementally, or when some computations genuinely need full history and can't be made incremental (such as ML model training).

Kappa architecture: stream only

Jay Kreps (a co-creator of Kafka) proposed the simplification: if you have a replayable event log, batch processing is just stream processing replayed from the beginning. The log (Kafka or similar) is the system of record, retaining events for days, weeks or longer; one streaming application consumes it and writes to serving stores. There is no batch layer: reprocessing means replaying the log through the same code.

Reprocessing through replay

A dashboard shows orders per hour by region, and you discover all "CA" orders were counted as California instead of Canada:

1. Fix the region mapping in the single stream processor.
2. Deploy stream-processor-v2 starting at offset 0, writing to a new table orders_by_region_v2.
3. v2 replays the whole log with the corrected logic while v1 keeps serving live data.
4. When v2's lag is near zero, point the dashboard at orders_by_region_v2.
5. Decommission v1 and drop the old table.

Reprocessing follows exactly the same code path as live processing: one implementation, one test suite, one deployment pipeline, so batch and stream results can't drift apart.

Why modern infrastructure makes Kappa feasible

• Durable long-term storage: Kafka keeps events on disk with configurable retention (and tiered storage), so a log with months of history is effectively an append-only database.
• Independent consumption: separate consumer groups read the same log independently; a replay from offset 0 doesn't disturb the live job.
• Scalable replay: consumers parallelize across partitions, so replay runs as fast as disk and compute allow.

Where Kappa works and where it struggles

• Works well when events are the source of truth, reprocessing time is acceptable, real-time and historical logic are identical, and log retention covers the history you need. See event sourcing.
• Struggles when history predates the log (you can't replay what was never captured), when replaying petabytes would take days, when computations need true batch semantics (some ML training, huge historical joins), or when sinks can't handle replayed data cleanly.

Lambda vs Kappa

For new projects on modern infrastructure, Kappa usually wins on simplicity. Enterprises with heavy batch investment, compliance tied to batch systems, or history that was never captured as events can use Lambda to add real-time without disrupting proven systems. Unified engines (Flink, Spark Structured Streaming, Beam) blur the line further by running one codebase over both bounded and unbounded data; see stream processing frameworks.