Publish-Subscribe Messaging
Publish-subscribe explained: topics, fan-out to many subscribers, push vs pull delivery, and how pub/sub differs from a work queue.
Publish-subscribe (pub/sub) is a messaging pattern where publishers send messages to a topic without knowing who will receive them, and every subscriber to that topic gets its own copy. Where a work queue splits work between consumers, pub/sub broadcasts events to many independent consumers.
How it works
Each subscriber typically gets its own queue or subscription attached to the topic, so it can consume at its own pace, retry independently and be down without affecting the others. Within one subscription, several instances of the same service can share the work (competing consumers).
Why it's useful
- Loose coupling: the publisher doesn't call each downstream service, so it doesn't break or slow down when one of them does.
- Extensibility: new features (fraud checks, a recommendation update, an audit log) subscribe to existing events without touching the publisher.
- Fan-out: one event triggers many reactions: notifications, cache invalidation across servers, real-time updates to connected clients.
Push vs pull delivery
| Push | Pull | |
|---|---|---|
| Delivery | The broker calls subscribers (HTTP endpoints, functions) | Subscribers poll the broker for messages |
| Flow control | Broker must back off when subscribers are slow | Subscribers take only what they can handle |
| Good for | Webhooks, serverless functions, low volume | High throughput, batch processing |
| Examples | SNS → HTTP or Lambda, Google Pub/Sub push | Kafka consumers, Google Pub/Sub pull, SQS |
Durability differs a lot
- Fire-and-forget pub/sub (Redis pub/sub, plain MQTT QoS 0): only currently connected subscribers receive messages; anything published while a subscriber is disconnected is lost. Fine for cache invalidation signals and live UI updates.
- Durable pub/sub (SNS + SQS, Google Pub/Sub, RabbitMQ fanout exchanges with durable queues): messages are stored per subscription until acknowledged.
- Log-based (Kafka, Pulsar, Kinesis): events are retained for days regardless of who read them, so new subscribers can even start from the beginning. See event streaming and Kafka.
Design considerations
- Event design: publish facts about what happened (
OrderPlacedwith the order data), not commands to specific services. Include an event ID, type, version and timestamp. - Schema evolution: subscribers you don't know about depend on your events. Add fields compatibly; use a schema registry for Avro or Protobuf events.
- Delivery is at-least-once for durable systems, and order across subscribers isn't coordinated, so subscribers must be idempotent.
- Publishing reliably: if you update the database and then publish, a crash between the two loses the event. Use the transactional outbox.
- Visibility: with many subscribers, tracing an event's effects needs good observability (correlation IDs on every event).
Where it’s used
A common pattern: publish once to an SNS topic, with an SQS queue per consuming service for durable, independent processing.
Chat gateways subscribe to channels (Redis pub/sub, NATS) to receive messages for the users connected to them. See WebSockets.
When data changes, an invalidation message is broadcast so every app server drops its in-memory copy.
Devices publish telemetry to topics like building/7/floor/2/temp and dashboards subscribe with wildcards.
In an interview
- Use pub/sub when one event must reach several services; use a work queue when one worker should handle each job.
- Give each subscriber its own durable subscription so failures stay isolated.
- Publish domain events (what happened), versioned and with IDs, and make consumers idempotent.
- Mention the outbox pattern to avoid losing events between the database write and the publish.