What Is Stream Processing? Unbounded Data, Brokers and State

A nightly fraud job flags suspicious accounts by morning, but a fraudster who steals card details at 9 AM has all day to drain them. Stream processing handles data continuously as it arrives instead of waiting for it to pile up, so results come in seconds rather than hours.

Bounded vs unbounded data

Batch jobs read bounded data: a finite dataset with a start and an end ("yesterday's logs", "Q3 transactions"); they read it all, process it and stop. Stream processors read unbounded data: clicks, sensor readings and payments that never stop arriving, so the processor runs indefinitely. It's counting fish in a bucket versus counting fish swimming past a point in a river: the river never ends, so you report rolling totals. See batch vs stream processing.

Message brokers: the input layer

Streams usually come from a message broker (event bus or streaming platform): Apache Kafka most often, or AWS Kinesis, Apache Pulsar and others. Producers (your services) write events to topics such as user-signups, order-placed or sensor-readings; stream processors subscribe and receive them in order. See Kafka.

• Decoupling: producers don't know who consumes; consumers subscribe to the topics they need.
• Durability: events are kept for a retention period (hours, days or forever). A crashed consumer resumes where it left off, and a new consumer can replay history.
• Ordering: events keep their order within a partition of a topic.
• Scalability: topics are split into partitions across brokers, and consumer instances divide the partitions between them.

The stream processing loop

1. Read events from input topics.
2. Process each one: filter, transform, enrich, aggregate.
3. Produce output: write to output topics, update databases, trigger alerts.

Unlike a batch job, the processor never finishes; it waits for the next event. The hard part is state: counting page views needs a running count, and detecting "three failed logins within 5 minutes" needs recent attempts per user. Stream processors keep this state, usually in an embedded store (such as RocksDB) backed up to durable storage so it survives restarts.

# Stateful stream processing: alert on 3 failed logins for one user within 5 minutes
import json
from collections import defaultdict, deque
from kafka import KafkaConsumer, KafkaProducer

WINDOW_S, LIMIT = 300, 3
consumer = KafkaConsumer("login-attempts", bootstrap_servers="localhost:9092",
                         group_id="login-guard", value_deserializer=json.loads)
producer = KafkaProducer(bootstrap_servers="localhost:9092",
                         value_serializer=lambda v: json.dumps(v).encode())
failures = defaultdict(deque)            # state: user_id -> recent failure timestamps

for msg in consumer:                     # runs forever
    e = msg.value                        # {"user_id": "u1", "ok": false, "ts": 1760950000}
    if e["ok"]:
        failures.pop(e["user_id"], None)
        continue
    recent = failures[e["user_id"]]
    recent.append(e["ts"])
    while recent and recent[0] <= e["ts"] - WINDOW_S:   # drop attempts older than 5 min
        recent.popleft()
    if len(recent) >= LIMIT:
        producer.send("security-alerts", {"user_id": e["user_id"], "failures": len(recent)})
        recent.clear()
# Production engines (Flink, Kafka Streams) keep this state fault-tolerant and partitioned by key.

How streaming differs from batch

When stream processing fits

• Real-time decisions: fraud scoring before the transaction completes; recommendations while the user is still browsing.
• Live dashboards and monitoring: requests per second, error rates and active users now, not in an hourly batch that's useless during an incident.
• Event-driven architectures: an order placed triggers inventory reservation, then payment, then a shipping notification, each step reacting to the previous step's events. See event-driven architecture.
• Threshold alerts: page the on-call engineer when errors exceed 5% now, not when the nightly job notices.

Streaming brings new problems batch never had: aggregating "per hour" over a stream that never ends (windows), events that arrive late or out of order (event time and watermarks), and processing each event exactly once despite failures (delivery semantics).