Stream Processing Frameworks: Flink, Kafka Streams, Spark and Beam

You can build windowing, event time and delivery guarantees on raw Kafka consumers, but then you also write your own state management, consumer rebalancing and exactly-once coordination. Stream processing frameworks do that hard part so you write business logic. The common choices are Apache Flink, Kafka Streams, Spark Structured Streaming and Apache Beam.

What a framework gives you

• Managed state: per-key state (running counts, window contents) tracked for you and redistributed when workers scale up or down.
• Fault tolerance: automatic checkpoints and recovery, so a failed worker resumes from the last checkpoint. No loss and exactly-once also need replayable sources (to re-read events after the checkpoint) and transactional or idempotent sinks (so reprocessed output isn't written twice).
• Windowing and event time: tumbling, sliding and session windows with watermarks for late events. See windows and watermarks.
• Exactly-once semantics: coordinated commits so events aren't lost or double-counted. See delivery semantics.

Flink vs Kafka Streams: how you run them

In interviews, the point is why these frameworks exist: they take care of stateful, fault-tolerant, exactly-once processing so you focus on the logic.

Unified batch and stream processing

Lambda runs batch and stream separately, and Kappa uses streaming for everything. Modern frameworks instead treat them as the same computation over different data shapes: batch is bounded data, streaming is unbounded, but filter, map, aggregate and join are the same. Write the pipeline once; point it at historical files for a backfill or at a live stream for real-time results. The engine optimizes bounded input for throughput and complete results, and unbounded input for latency and late data. See Lambda vs Kappa.

Apache Flink: stream-first

Flink treats batch as a bounded stream, with the same DataStream and Table APIs for both. In streaming mode it processes events as they arrive with continuous state, watermarks and late-data handling; in batch execution mode it reads complete input and adds optimizations such as sort-based joins. Strengths: native event-time processing, exactly-once via distributed snapshots, robust state management, and true sub-second streaming rather than micro-batches. It excels at complex stateful applications: session analysis, complex event processing, real-time ML features.

Apache Spark: batch-first

Spark added streaming through Structured Streaming, with the same DataFrame API. Only the source changes:

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("sales-by-category").getOrCreate()

def sales_by_category(df):                       # identical logic for batch and stream
    return df.groupBy("category").agg(F.sum("amount").alias("revenue"))

# Batch: a bounded dataset (e.g. a backfill)
batch_df = spark.read.parquet("s3a://data/orders/")
sales_by_category(batch_df).write.mode("overwrite").parquet("s3a://reports/sales/")

# Stream: unbounded events from Kafka
schema = "category STRING, amount DOUBLE, ts TIMESTAMP"
stream_df = (spark.readStream.format("kafka")
             .option("kafka.bootstrap.servers", "localhost:9092")
             .option("subscribe", "orders").load()
             .select(F.from_json(F.col("value").cast("string"), schema).alias("o"))
             .select("o.*"))
(sales_by_category(stream_df).writeStream
    .outputMode("complete").format("console").start())

Structured Streaming groups events into micro-batches (typically every few hundred milliseconds to seconds) and runs each with the batch engine. The trade-offs: latency is seconds rather than milliseconds, and event-time correctness needs explicit watermark configuration, but teams already on Spark add streaming with little new to learn. Good when you have Spark batch pipelines, need MLlib, or second-level latency is fine. See Spark architecture.

Apache Beam: a portability layer

Beam is a unified programming model that runs on several engines (Flink, Spark, Google Cloud Dataflow and others): write the pipeline once in the Beam SDK and run it on Dataflow in GCP, Flink on-premises or Spark on EMR. The appeal is avoiding lock-in and standardizing one API across environments; the cost is abstraction, since engine-specific features (Flink's state backends, Spark's ML integration) may not be fully exposed.

Greenfield low-latency streaming usually points to Flink; heavy Spark shops get the gentlest path with Structured Streaming; multi-cloud enterprises value Beam. Unified processing means one codebase, one test suite, easier backfills (same code over historical data) and the same semantics for bounded or unbounded input: batch vs stream becomes an input setting, not an architecture decision.