Stream Processing: Windows, Watermarks and Late Data

Stream processing computes results continuously over an unbounded flow of events, rather than over a finished dataset. Its hard part isn't speed but time: events arrive late, out of order, and from devices with wrong clocks. Windows group events into finite chunks to aggregate, event time says when something actually happened, and watermarks tell the processor how long to wait for stragglers before emitting a result.

Event time vs processing time

A phone that loses signal in a tunnel might send a purchase event ten minutes late. Counted in processing time, the purchase lands in the wrong minute's revenue. Counted in event time, it lands in the right one, as long as the processor is still waiting for that window.

Window types

Window types in depth

You can't aggregate an infinite stream, so windows cut it into finite chunks: collect events over an interval, aggregate, emit.

Tumbling windows: fixed, non-overlapping

Counting page views per minute: 10:00:00–10:00:59, then 10:01:00–10:01:59, and so on; when a minute ends, emit its count and start the next. A dashboard's "requests per minute" chart is one tumbling-window count per point. Strengths: simple, memory-efficient (one window of state at a time), every event counted exactly once. Limitation: activity near a boundary is split; a session from 10:59 to 11:01 lands in two windows.

Sliding (hopping) windows: overlapping

You choose a size and a slide. Rate limiting is the classic case: "requests in the last 60 seconds". With tumbling windows, a burst at 10:00:59 and another at 10:01:01 fall into different windows and slip through; a continuous sliding window (or one with a small slide) has some window covering both, though a coarse slide can still split them. Strengths: smoother metrics, catches patterns that straddle boundaries, good for moving averages. Limitation: more memory; with a 1-minute window sliding every 10 seconds, each event is in up to 6 windows. See rate limiting.

Session windows: activity-based

No fixed length: events within the gap timeout (say 30 minutes) extend the session; 30 quiet minutes close it and emit. Analyzing browsing on a shop, user A visits for 5 minutes and user B for 2 hours; fixed windows would chop both arbitrarily, while session windows match real behavior. Strengths: follows natural activity, ideal per user or entity. Limitations: output waits for the gap; per-key session state is more complex; very long sessions accumulate unbounded state.

The incomplete window problem

When should a window close? "When the clock shows 10:01:00, emit the 10:00 window" fails if an event stamped 10:00:45 arrives at 10:01:02 after a network delay. Drop it (lose accuracy)? Reopen the window and re-emit (update results)? Count it in the current window (wrong time)? Answering that requires event time and watermarks.

Event time, in depth

Event time is when the event happened, stamped at the source (the user's device says 10:00:00). Processing time is when your system handles it: 10:00:05 normally, or 10:05:00 if the system fell behind during an outage. Mobile apps buffer events offline and send them in bursts; remote IoT devices can take minutes to reach the data center.

• Processing-time windows count clicks by when they were processed. A 5-minute backlog pushes 10:00 clicks into the 10:05 window, and a consumer restart can double-count replayed events. The same data gives different answers depending on system timing.
• Event-time windows count clicks by when users clicked. The same input always gives the same counts: deterministic reprocessing, so you can replay and get identical output. Reproducible analytics need event time.

The catch is out-of-order arrival: user A's 10:00:55 event goes through a slow cell tower while user B's 10:01:02 event comes over fast Wi-Fi, so you see 10:01:02 first. Close the 10:00 window on the first 10:01 event and you miss A's click; but you can't wait forever either.

Watermarks and late data

A watermark is the processor's estimate that "no more events older than time T will arrive". A common heuristic is the maximum event time seen minus an allowed delay, for example two minutes. When the watermark passes the end of a window, the window can be finalised and emitted.

• Too short a delay: results are fast but many events arrive "late" and are dropped or require corrections.
• Too long a delay: results are complete but slow, and state for open windows grows.
• Allowed lateness: keep a finished window's state a little longer and emit updated results for stragglers, then discard it.
• Side outputs: send events that are too late to a separate stream or table, so they're counted in a correction job instead of silently lost.
• Idle sources: a partition with no traffic holds the watermark back; processors need idleness detection or per-source timeouts.

Downstream systems must accept updated results for a window (upserts keyed by window and key) if you emit corrections. If they can only append, you'll double count.

How watermarks are generated

• Perfect watermarks: possible when you control the source and know everything has been seen, e.g. reading a bounded, ordered log file.
• Heuristic watermarks: estimated from observed lateness. If 99% of events arrive within 30 seconds, keep the watermark 30 seconds behind the newest event time.
• Source-based watermarks: with partitioned input (Kafka partitions), track event-time progress per partition and take the minimum: the slowest partition bounds the watermark.

Handling late events

Latency vs completeness is unavoidable. Aggressive watermarks (fast, little lateness) emit sooner but miss late events; conservative ones are more complete but slower and hold more state. A real-time active-users dashboard can drop a few late events; a billing system should wait for the right answer rather than undercharge and reconcile; fraud detection often scores with aggressive watermarks for speed and runs a separate batch reconciliation for accuracy. Event time makes results reproducible, watermarks let event time advance despite disorder, and the trade-off becomes an explicit choice instead of random distortion from processing delays.

State, checkpoints and exactly-once

Windows, joins and deduplication keep state per key. Engines such as Apache Flink and Spark Structured Streaming store that state locally and checkpoint it, together with the input offsets, to durable storage. After a crash, the job restores the last checkpoint and replays input from the matching offsets, so each event affects the state exactly once.

That guarantee is about internal state. For the output to be exactly-once too, the sink must be transactional (commit output with the checkpoint) or idempotent (upserts keyed by window), as explained in delivery guarantees. Otherwise replay after a crash can produce duplicates downstream.

Design checklist

1. Stamp events with event time at the source, and keep a unique event ID for deduplication.
2. Choose windows from the business question, and the watermark delay from measured lateness (for example the 99th percentile delay).
3. Decide what happens to late data: update results, send to a side output, or drop with a metric.
4. Partition the input stream by the aggregation key so state stays local, and watch for hot keys.
5. Use checkpoints and an idempotent or transactional sink for correct results after failures.
6. Reconcile periodically with a batch job over the same raw data. See batch vs stream processing.