Event Streaming and Kafka: Log-Based Message Brokers

Apache Kafka and similar systems (Amazon Kinesis, Apache Pulsar, Redpanda) are log-based message brokers. Queues such as Redis Queue keep messages in memory and delete them once processed: fast, but with no long-term storage. A log-based broker instead appends events to a durable, ordered log on disk and keeps them for a retention period. Because messages stay on disk, one message can be consumed by many consumers, independently and at their own pace, and history can be replayed. That makes these systems the backbone of event-driven architectures and data pipelines.

For example, a hotel booking service writes each booking to a log; the order processing, customer notification and analytics services each read the same booking event and do their own work. Every consumer sees every event, nothing is lost if one of them is down, and adding a new consumer requires no change to the producer.

Components of a log-based message queue

1. Log: an ordered, append-only sequence of records (messages). Each message has a unique, ever-increasing sequence number called its offset. Logs can be partitioned and spread across machines for horizontal scaling and parallel processing.
2. Producers: services that send messages. They append to the end of the log, and each appended message gets the next offset, giving a total order within its partition.
3. Partitions: a topic's log is split into partitions, each its own ordered, append-only sequence. Partitions live on different nodes, which adds parallelism and throughput, and each can be replicated to other nodes for fault tolerance (just like database replication).
4. Consumers: services that read messages. Each consumer keeps its own offset (its position in the log), reads sequentially from it, and advances it as it processes. Many consumers can read the same partition independently without affecting each other.
5. Consumer groups: consumers that share the work. Each consumer in a group gets a distinct set of partitions; when a consumer fails or joins, partitions are reassigned among the group, giving both scalability and fault tolerance.

In the strip above, highlighted (orange) cells are where consumers currently are: one consumer is at offset 4 of topic 1 partition 1 and offset 2 of topic 2 partition 0; another is at offset 8 of topic 2 partition 1. Each consumer resumes from its saved offset if it disconnects, exactly like a follower resuming from its position in a database's replication log.

What happens when a message arrives

1. Production: a producer builds a message (key, metadata, topic or partition, payload) and sends it.
2. Serialization: the producer turns it into bytes for efficient storage and transfer, using JSON, Avro or Protocol Buffers.
3. Partition selection: if the producer didn't name a partition, the system picks one, round-robin or by hashing the message key, so load spreads evenly and same-key messages land together.
4. Appending: the message is appended to the end of the chosen partition. This is a sequential disk write, so it's very fast, and the message gets the next offset.
5. Consumption: consumers subscribed to the topic eventually read it, sequentially from their current offset, and advance their offset. Consumers in the same group split the partitions between them.

In a typical setup one partition is read by only one consumer in a group at a time. If one message is slow to process, everything behind it in that partition waits. That's why, when individual messages are expensive to process (encoding a video, calling a slow partner API), a traditional queue such as RabbitMQ or SQS, where any free worker can take the next message, is usually the better fit. See message queues.

Kafka's building blocks

Kafka's architecture closely follows the general log-based design above:

• Event: a record that something happened, with the state of the system or an entity, encoded as JSON, Avro or Protobuf. Kafka stores events as key-value pairs.
• Topic: like a table in a database, a named category of events (orders, page-views). Each topic is made of partitions holding append-only logs. Logs are immutable and simple to reason about; they're read by offset (scanning forward, no random access by content); they're durable files on disk, kept forever or for a retention period; and they replicate easily.
• Partition: lets a topic spread across many machines. The key decides the partition: no key means round-robin; with a key, hash(key) picks the partition. Messages with the same key always go to the same partition and, because the log is append-only, stay in order. There is no ordering across partitions.
• Broker: a machine running Kafka. Brokers host partitions, handle replication, and serve reads and writes.
• Replication: every partition is copied to several brokers using leader-based replication: one leader takes writes and N−1 followers copy it. It's automatic, and producers can tune how many copies must confirm a write.
• Producer: written with the Producer / ProducerRecord API; chooses the partition from the event's key.
• Consumer: written with the Consumer / ConsumerRecord API, usually in a consumer group that subscribes to a topic; the topic's partitions are divided among the group's consumers. A group should have no more consumers than partitions, since extra consumers sit idle.
• Kafka Connect: an integration framework that moves data between Kafka and other systems, acting as a producer or consumer. Ready-made connectors exist for Elasticsearch, relational databases and many more, so you don't write boilerplate; the Datagen connector generates demo data.
• Schema Registry: a separate service holding a database (itself a Kafka topic) of message schemas, cached for speed. Producers and consumers check messages against it so formats stay compatible as they evolve. See data contracts and schema evolution.

Consumer groups

Consumers with the same group ID share the work: each partition is assigned to exactly one consumer in the group. Different groups each get the full stream. So one topic can feed billing, analytics and search indexing, each scaling independently.

• Parallelism within a group is capped by the number of partitions: 12 partitions means at most 12 active consumers in a group. Choose partition counts with growth in mind.
• When consumers join or leave, partitions are rebalanced among the group.
• Consumers commit offsets after processing. On restart they resume from the last commit, so events since then are reprocessed: at-least-once by default. See delivery guarantees.

Use cases

• Service-to-service communication: like a traditional queue, a link between services, with the bonus that many services can consume the same events.
• Event sourcing: store the sequence of events that changed application state, and rebuild the current state by replaying them in order. See event sourcing and CQRS.
• Stream processing: analyze real-time streams with Kafka Streams or Apache Flink: aggregate data from many sources, compute over time windows, and trigger actions. See stream processing.
• Data pipelines and change data capture: move database changes and activity data into search indexes, caches and warehouses. See change data capture.

Popular log-based systems

Why a log is powerful

• Replay: reset a consumer group's offset to reprocess history after fixing a bug, or to build a brand-new service from past events.
• Many independent readers without the broker tracking per-message state for each.
• High throughput: sequential disk writes, batching, compression and zero-copy reads let a cluster handle millions of events per second.
• Log compaction: for topics keyed by entity, Kafka can keep only the latest event per key, turning the topic into a durable table of current state.

Durability settings

Kafka vs a traditional queue

Hands-on: partitions, keys and ordering

This exercise shows how Kafka spreads keyed messages across partitions and what ordering you get. We'll create a topic with 3 partitions, run a producer that sends increasing integers, run 3 consumers each reading one partition, and watch the output.

1. Start Kafka. Setting Kafka up by hand is more involved than Redis: you need a JDK, the Kafka distribution and Python. The easiest route is the official Docker image, which runs a single-node broker in KRaft mode (no ZooKeeper):

docker run -d --name kafka -p 9092:9092 apache/kafka:3.8.0
pip install kafka-python

2. Create a topic with 3 partitions and describe it:

docker exec kafka /opt/kafka/bin/kafka-topics.sh --create --topic multi_partition_ordered \
  --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
Created topic multi_partition_ordered.

docker exec kafka /opt/kafka/bin/kafka-topics.sh --describe --topic multi_partition_ordered --bootstrap-server localhost:9092
Topic: multi_partition_ordered  PartitionCount: 3  ReplicationFactor: 1
    Topic: multi_partition_ordered  Partition: 0  Leader: 1  Replicas: 1  Isr: 1
    Topic: multi_partition_ordered  Partition: 1  Leader: 1  Replicas: 1  Isr: 1
    Topic: multi_partition_ordered  Partition: 2  Leader: 1  Replicas: 1  Isr: 1

3. The producer, kafka_producer.py. It sends 0, 1, 2, … once a second, with key key-{n % 3}. Kafka's default partitioner hashes the key, so every message with the same key lands on the same partition:

import time
from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers="localhost:9092")

number = 0
while True:
    key = f"key-{number % 3}"            # three keys -> (usually) three partitions
    producer.send("multi_partition_ordered", key=key.encode(), value=str(number).encode())
    number += 1
    time.sleep(1)

Run python kafka_producer.py and leave it running (it prints nothing).

4. The consumers, kafka_consumer.py. Three threads, each assigned one partition of the topic:

import threading, time
from kafka import KafkaConsumer, TopicPartition

TOPIC = "multi_partition_ordered"

def consume_partition(partition_id):
    consumer = KafkaConsumer(bootstrap_servers="localhost:9092",
                             group_id="ordered_partition_group",
                             auto_offset_reset="earliest")
    partitions = None
    while not partitions:                           # wait until the topic exists
        partitions = consumer.partitions_for_topic(TOPIC)
        if not partitions:
            time.sleep(1)
    consumer.assign([TopicPartition(TOPIC, sorted(partitions)[partition_id])])
    print(f"Consuming partition {partition_id}")
    for msg in consumer:
        print(f"Partition: {msg.partition} - Offset: {msg.offset} - Key: {msg.key} - Value: {msg.value}")

if __name__ == "__main__":
    for i in range(3):                              # consumers must be <= partitions
        threading.Thread(target=consume_partition, args=(i,)).start()

5. Run the consumers in a second terminal with python kafka_consumer.py:

Consuming partition 2
Consuming partition 0
Consuming partition 1
Partition: 1 - Offset: 0 - Key: b'key-0' - Value: b'0'
Partition: 1 - Offset: 1 - Key: b'key-0' - Value: b'3'
Partition: 1 - Offset: 2 - Key: b'key-0' - Value: b'6'
Partition: 1 - Offset: 3 - Key: b'key-0' - Value: b'9'
Partition: 1 - Offset: 4 - Key: b'key-0' - Value: b'12'
...
Partition: 1 - Offset: 23 - Key: b'key-0' - Value: b'69'

Key key-0 (values 0, 3, 6, …) hashed to partition 1, and within that partition the values are always increasing, each at the next offset. Messages for other keys appear on other partitions, interleaved in whatever order the consumers happen to print them. That's the guarantee in one picture: order within a partition, none across partitions. (With only three keys, two may hash to the same partition, leaving one consumer idle, which is also why the number of distinct keys should comfortably exceed the number of partitions.)

Pitfalls

• Hot partitions: a skewed key (one huge customer) overloads one partition and its consumer. Choose keys with even distribution. See hot partitions.
• Slow consumers block their partition: one slow message delays everything behind it in that partition. Keep processing fast; push slow work to a separate queue.
• Too few partitions limit scaling; increasing them later changes key-to-partition mapping and breaks per-key ordering during the change.
• Consumer lag is the key metric: how far behind the latest offset each group is. Alert on it.