Message Queues

A message queue lets one part of a system hand work to another without waiting for it to finish. A producer adds a message to the queue; a consumer takes it off, processes it, and acknowledges it so it is removed. Think of it as email between services: the sender doesn't need the receiver to be available right now. The queue decouples them in time and speed: the producer stays fast, and work survives even if consumers are slow, busy or down.

Asynchronous communication and messages

Synchronous calls make the caller wait. Asynchronous communication shares data between components without blocking, and messaging is its most common form.

A message is the unit of communication. It has two parts:

• Header: key-value metadata such as a message ID, timestamp, sender, type or priority.
• Body: the payload, as text or binary. Its format (JSON, Protobuf, XML) isn't fixed by the queue; the system's designers decide it.

+-----------------------------------------------+
| Header                                        |
| message_id: 12345, timestamp: 1678901234567   |
+-----------------------------------------------+
| Body                                          |
| { "order_id": "A456", "status": "confirmed" } |
+-----------------------------------------------+

Here (for example in RabbitMQ) the header carries a unique message_id for tracking and the creation timestamp; the body carries an order ID and its status, a typical way for an e-commerce system to tell the inventory service about a confirmed order without calling it directly.

Messaging styles

One-way notification

A sender sends a message to one recipient and expects no reply: fire and forget. Used for logging, alerts and background jobs.

One-way publish/subscribe

A publisher sends messages to a topic and every subscriber receives them. Common in event-driven systems where many components react to the same event. See pub/sub.

Two-way asynchronous request–response

The sender sends a request message and later receives a reply message, which needs two channels: one for requests and one for responses (with a correlation ID linking them). It's less common than the first two.

Message brokers

The message channel is usually a message broker: an intermediary that every message flows through. The sender writes to the broker and the broker delivers to the receiver. It ensures reliable delivery, manages queues, and supports patterns such as point-to-point and publish–subscribe. Each broker implements the channel idea differently:

Why use a message queue

1. It buffers between components that run at different speeds. Services that write to a database are usually slower than stateless web services working in memory. With a queue, producers send at their own pace without overwhelming slower consumers. In a photo-sharing app, users upload photos far faster than the image processor and database updater can handle them; on a video platform like YouTube, the frontend accepts comments faster than the database updater can store them. The queue lets the consumers catch up without losing anything.
2. It decouples producer and consumer, which adds fault tolerance. If a consumer goes down, its messages stay in the queue and are processed when it comes back. If the photo processor or the database updater crashes, nothing is lost.

A concrete case. On Black Friday a store gets 10,000 orders a minute. The order service calls the payment service, which takes 2 seconds per order, and waits for each one: orders pile up and users see timeouts. Then the payment service crashes for 30 seconds; the 5,000 orders that arrive meanwhile all fail, with nothing to retry them, and the revenue is gone. With a queue, the order service publishes each order and replies to the user at once; the payment service works through the queue at its own pace; if it crashes, orders wait in the queue and processing resumes when it recovers. No order is lost.

How delivery works

1. Producers publish messages, for example {"order_id": 123, "user_id": 456, "total": 99.99}.
2. The queue stores them durably: if the queue server crashes, messages persist on disk. They wait until a consumer is ready.
3. A consumer receives a message. The queue hides it from other consumers for a visibility timeout (say 30 seconds) but doesn't delete it yet.
4. The consumer processes it and acknowledges it; the queue then deletes it.
5. If the consumer crashes or the timeout passes without an ack, the message becomes visible again and is redelivered, possibly to another consumer.

This gives at-least-once delivery: a message is never lost, but it can be processed more than once (for example if a consumer finishes the work but crashes before acknowledging). Consumers must therefore be idempotent. See delivery guarantees and idempotency.

Two families of message queues

AMQP/JMS-style (traditional) brokers

Built on the Advanced Message Queuing Protocol (AMQP) or the Java Message Service (JMS) standard, originally for passing messages inside large enterprise systems. They support publish/subscribe, point-to-point and request/reply.

• Messages are usually held in memory and deleted once consumed. The emphasis is on routing, filtering and transforming messages.
• Duplicates are possible, and ordering is either not guaranteed or guaranteed only per queue (never globally).
• Examples: RabbitMQ, ActiveMQ, Azure Service Bus, IBM MQ.

Log-based brokers

Kafka, AWS Kinesis and similar systems store messages in a distributed, append-only log. All messages in a partition go to the same consumer, always in the same order; parallelism comes from having many partitions; consumers record their progress by checkpointing the offset of the last message they processed. Because messages stay on disk, a consumer can jump back and re-read old ones. These are often called event streaming platforms: the difference from a queue is that a stream is persisted and read with an iterator, while a queue's messages are processed independently and deleted when done.

• Messages are persisted on disk, enabling long retention, reprocessing and strong fault tolerance.
• Built for high throughput and low latency: data streaming and real-time analytics.
• Strong durability guarantees at very high volumes.
• Messages within a partition are totally ordered, each with a unique offset.
• Examples: Apache Kafka, AWS Kinesis, Google Cloud Pub/Sub, Azure Event Hubs. See event streaming and Kafka.

As with databases, the line blurs: IBM MQ added message deduplication (from version 9.2.3), and Amazon SQS is a hybrid: queue-centric with acknowledgment and deletion, but also distributed with message retention.

Patterns: one-to-one and publish/subscribe

• One-to-one (point-to-point): each message is processed by exactly one consumer, and producers usually know which queue they're sending to. Amazon SQS works this way: a message sent to a queue is processed by one consumer.
• Publish/subscribe: each message can be processed by many consumers (the fan-out pattern). Producers publish to a central topic without knowing who's listening; each subscriber gets its own copy, delivered immediately. A common AWS setup combines SNS and SQS: SNS is the topic, and each subscribing SQS queue receives a copy of every message, so several independent consumers can process it.

Use case 1: absorbing traffic between tiers

A hotel booking site: customer traffic passes a load balancer to a pool of web servers. When someone books, the web servers call a second tier of booking microservices behind another load balancer, and the booking service writes to a relational database.

Most traffic is people browsing offers; relatively few actually book. Then a big promotion launches and bookings arrive faster than the system was built for. The many web servers forward a flood of requests to the booking tier, which can't keep up; the database starts timing out and dropping requests, and the errors flow back to customers.

The fix: replace the second load balancer with a queue. The queue absorbs the burst, and the booking services pull bookings from the backlog at a rate they and the database can handle. The capacity mismatch between the tiers can no longer cause an overload.

Use case 2: notifying other services

After a booking completes, the booking service could call the other services directly: payments, the notification service (mobile push), external partner integrations (sending the voucher to the hotel) and more. But each of those differs in speed, call frequency and traffic spikes; any one being slow or down slows or breaks bookings.

Instead, the booking service publishes one "new booking completed" message to a booking topic, and every interested service subscribes. The booking service no longer knows or cares who listens, and adding a new consumer requires no change to it.

Use case 3: background tasks

After a successful booking, the service must do several things:

Done one after another, the user waits 30 + 1,000 + 500 + 1,000 = 2,530 ms.

def synchronous_tasks():
    update_booking_status()
    create_pdf_voucher()
    send_confirmation_email()
    notify_other_services()        # the user waits for all four: about 2.5 s

Only the status update has to happen before responding. Running the rest in background threads cuts the response to about 30 ms:

from threading import Thread

def asynchronous_tasks():
    update_booking_status()                          # 30 ms, before responding
    for task in (create_pdf_voucher, send_confirmation_email, notify_other_services):
        Thread(target=task, daemon=True).start()     # don't wait for them

That's fast, but fragile: if the process crashes or a thread fails, the work is silently lost and nobody knows whether it ran. The better approach is a task queue: put the background tasks on a queue and let separate worker processes run them. A worker removes a task only when it's done, so tasks survive crashes, can be retried, and their status can be checked. With Python's Redis Queue (RQ):

import time
from redis import Redis
from rq import Queue

queue = Queue(connection=Redis())

def update_booking_status():   time.sleep(0.03); print("Booking status updated.")
def create_pdf_voucher():      time.sleep(1.0);  print("PDF voucher created.")
def send_confirmation_email(): time.sleep(0.5);  print("Confirmation email sent.")
def notify_other_services():   time.sleep(1.0);  print("Other services notified.")

if __name__ == "__main__":
    update_booking_status()                       # synchronous, before responding
    jobs = [queue.enqueue(task) for task in (create_pdf_voucher, send_confirmation_email, notify_other_services)]
    print([job.id for job in jobs])               # keep the IDs to check status later
    # Run workers in separate processes:  rq worker

In production, put the task functions in their own module so workers can import them, add retries (Retry(max=3, interval=[10, 30, 60]) in RQ), and make every task idempotent.

Dead-letter queues

A dead-letter queue (DLQ) stores messages that couldn't be processed or delivered: the consumer kept crashing on them, their format was invalid, or they exceeded the maximum number of delivery attempts. (The name comes from undeliverable postal mail.) A message that fails every time is called a poison message. DLQs help in four ways:

1. Error handling: a failed message is moved aside instead of being lost or retried forever.
2. Monitoring and debugging: developers inspect the DLQ to find bugs in the processing logic or the messages themselves.
3. Isolation: bad messages no longer block or slow the valid ones behind them.
4. Reprocessing: once the bug is fixed, messages can be moved back to the main queue and processed.

Alert on DLQ depth, and give operators a way to inspect and replay messages.

When a consumer fails

What to do depends on why it failed and what the application needs:

1. Retry with backoff and jitter: wait and try again, for transient failures like a network blip. Increase the delay each time and randomize it so consumers don't all retry at once (the thundering-herd problem). See retries.
2. Send it to a dead-letter queue: instead of putting it back on the main queue, move it to the DLQ so the main queue keeps flowing, usually after a fixed number of attempts.
3. Discard it: for non-critical messages it can be acceptable to drop failures.

Also keep the visibility timeout longer than processing time, or extend it with heartbeats for long jobs; otherwise a slow job becomes visible again and is processed twice in parallel.

Ordering

Standard queues with many consumers don't guarantee order: messages are processed in parallel and retries reorder them. FIFO (first in, first out) queues guarantee order, so order messages for the same user are processed in sequence, at the cost of throughput; standard queues allow out-of-order processing for higher throughput. To keep order without losing parallelism, use ordered groups: SQS FIFO message groups, or Kafka partitions keyed by entity ID, which keep order per key while different keys proceed in parallel. See event streaming.

Watch the queue depth

A queue hides overload instead of removing it. If producers outpace consumers for long, the backlog grows, latency climbs from seconds to hours, and memory or retention limits are hit. Monitor queue depth and age of the oldest message, autoscale consumers on them, and apply backpressure or load shedding when the backlog can't be cleared.

Common implementations

Hands-on: a task queue with Redis Queue

Let's queue a slow task (downloading an image) and have a worker process it. The flow: main.py (the producer) enqueues a job in Redis; an rq worker process picks it up and runs download_image.

1. Set up. Start Redis (for example docker run -d --name redis -p 6379:6379 redis:7) and install the Python libraries from requirements.txt:

rq==2.6.0
requests==2.32.5
redis==6.4.0

pip install -r requirements.txt
redis-cli KEYS '*'        # (empty array): Redis starts clean

2. The task, in image_downloader.py. It downloads an image and saves it under its file name:

import requests

def download_image(url):
    response = requests.get(url, timeout=30)
    response.raise_for_status()
    filename = url.rstrip("/").split("/")[-1] or "image"
    with open(filename, "wb") as f:
        f.write(response.content)
    print(f"Image downloaded and saved as {filename}")

3. The producer, in main.py. It connects to Redis, creates a queue and enqueues one job:

from redis import Redis
from rq import Queue
from image_downloader import download_image

queue = Queue(connection=Redis())
job = queue.enqueue(download_image, "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png")
print("Enqueued job", job.id)

4. Run the producer and look inside Redis. The job is waiting in the queue:

$ python main.py
Enqueued job 27148712-d3c5-4762-bc8d-8af71001320f

$ redis-cli KEYS '*'
1) "rq:job:27148712-d3c5-4762-bc8d-8af71001320f"
2) "rq:queue:default"
3) "rq:queues"

5. Start a worker. It listens on the default queue, picks up the job and runs it:

$ rq worker
09:52:39 Worker 4f16…: started with PID 32, version 2.6.0
09:52:39 *** Listening on default...
09:52:39 default: image_downloader.download_image('https://…/PNG_transparency_demonstration_1.png') (27148712-…)
Image downloaded and saved as PNG_transparency_demonstration_1.png
09:52:41 Successfully completed image_downloader.download_image(...) job in 0:00:02.107133s
09:52:41 default: Job OK (27148712-…)
09:52:41 Result is kept for 500 seconds

Stop the worker and ls the directory: the image is there. You've just seen the whole lifecycle: enqueue, wait, dequeue, process, acknowledge. From here, try enqueueing heavier tasks (hashing or converting a video), starting several workers in parallel to see them share the queue, or making the task fail to see retries and the failed-job registry. The RQ documentation at python-rq.org covers scheduling, retries and job dependencies.