Message Queues & Event Streaming

By the end of this lesson you will be able to explain why systems put a queue between services, compare a classic queue with a Kafka-style log, reason about partitions, ordering and consumer groups, and handle failures with retries and dead-letter queues.

Why put a queue in the middle?

When an order is placed on a food app, many things must happen: charge the customer, notify the restaurant, find a rider, send an SMS, update analytics. If the order service calls each one directly and waits, the user waits too, and if the SMS service is down, the order fails for no good reason.

A message queue lets one service drop a message and move on. Other services pick messages up when they are ready. The sender is the producer, the reader is the consumer, and the system holding messages in between is the broker.

Think of the order spike at a dabba service. Kitchens drop tiffins at the collection point and go back to cooking. Delivery staff pick them up in batches and deliver at their own pace. The collection point is the queue. Kitchens and delivery staff no longer need to meet face to face.

Queues give you:

• Decoupling: the producer does not need to know who consumes, or whether they are up right now.
• Buffering: a sudden spike waits in the queue instead of crushing the consumers.
• Asynchronous work: the user gets a fast reply while slow work happens in the background.
• Fan-out: one event can feed many independent consumers.

Absorbing a spike

Step 1: model a flash sale. Orders arrive at up to 900 per second for a few seconds, but the payment-confirmation workers can process only 500 per second.

arrivals = [200, 300, 900, 900, 900, 900, 400, 200, 200, 200, 200, 200]   # per second
capacity = 500

backlog = 0
for second, incoming in enumerate(arrivals):
    backlog += incoming
    done = min(backlog, capacity)
    backlog -= done
    wait = backlog / capacity                     # rough seconds a new message waits
    bar = "#" * (backlog // 100)
    print(f"t={second:2}s in {incoming:3} done {done:3} backlog {backlog:5} wait ~{wait:3.1f}s {bar}")

t= 0s in 200 done 200 backlog     0 wait ~0.0s 
t= 1s in 300 done 300 backlog     0 wait ~0.0s 
t= 2s in 900 done 500 backlog   400 wait ~0.8s ####
t= 3s in 900 done 500 backlog   800 wait ~1.6s ########
t= 4s in 900 done 500 backlog  1200 wait ~2.4s ############
t= 5s in 900 done 500 backlog  1600 wait ~3.2s ################
t= 6s in 400 done 500 backlog  1500 wait ~3.0s ###############
t= 7s in 200 done 500 backlog  1200 wait ~2.4s ############
t= 8s in 200 done 500 backlog   900 wait ~1.8s #########
t= 9s in 200 done 500 backlog   600 wait ~1.2s ######
t=10s in 200 done 500 backlog   300 wait ~0.6s ###
t=11s in 200 done 500 backlog     0 wait ~0.0s 

Step 2: read the result. Without a queue, 1,600 requests would have failed. With it, they waited a few seconds and were all processed. The backlog (also called consumer lag) is the key number to monitor: if it keeps growing, add consumers.

Two models: queues and logs

Classic message queues (such as RabbitMQ or Amazon SQS) hand each message to one consumer. When the consumer confirms it has finished (an acknowledgement, or ack), the message is deleted. If the consumer crashes before acking, the message becomes visible again and another consumer gets it. This is perfect for a pool of workers sharing jobs.

Event logs (such as Apache Kafka, which LinkedIn engineers described in a 2011 paper, and similar systems like Redpanda and Apache Pulsar) work differently. Messages are appended to a durable, ordered log and kept for a set retention period, whether or not anyone has read them. Each consumer remembers its own position, called an offset. Many independent consumers can read the same log, and a consumer can rewind and replay old events, for example to rebuild a search index.

Kafka 4.0, released in March 2025, runs entirely without Apache ZooKeeper, using its own built-in consensus mode called KRaft.

Partitions, keys and consumer groups

A Kafka topic (a named stream, such as orders) is split into partitions. Each partition is an ordered log on a broker. Partitions are how a topic scales: different partitions live on different machines and are read in parallel.

When producing, you choose a key. Messages with the same key go to the same partition, so they stay in order. Use the order ID or user ID as the key when order matters for that entity.

A consumer group is a set of consumers sharing the work. Each partition is read by exactly one consumer in the group at a time. So the number of partitions caps how many consumers in one group can work in parallel.

import hashlib

class Topic:
    def __init__(self, partitions):
        self.logs = [[] for _ in range(partitions)]

    def produce(self, key, value):
        p = int(hashlib.md5(key.encode()).hexdigest(), 16) % len(self.logs)
        self.logs[p].append((key, value))
        return p

def assign(partitions, consumers):
    plan = {c: [] for c in consumers}
    for p in range(partitions):
        plan[consumers[p % len(consumers)]].append(p)
    return plan

orders = Topic(partitions=4)
events = [("order-11", "PLACED"), ("order-12", "PLACED"), ("order-11", "PAID"),
          ("order-13", "PLACED"), ("order-11", "DELIVERED"), ("order-12", "CANCELLED")]
for key, status in events:
    orders.produce(key, status)

for p, log in enumerate(orders.logs):
    print(f"partition {p}: {log}")

print("\n2 consumers:", assign(4, ["c1", "c2"]))
print("3 consumers:", assign(4, ["c1", "c2", "c3"]))
print("5 consumers:", assign(4, ["c1", "c2", "c3", "c4", "c5"]))

partition 0: [('order-12', 'PLACED'), ('order-12', 'CANCELLED')]
partition 1: []
partition 2: [('order-13', 'PLACED')]
partition 3: [('order-11', 'PLACED'), ('order-11', 'PAID'), ('order-11', 'DELIVERED')]

2 consumers: {'c1': [0, 2], 'c2': [1, 3]}
3 consumers: {'c1': [0, 3], 'c2': [1], 'c3': [2]}
5 consumers: {'c1': [0], 'c2': [1], 'c3': [2], 'c4': [3], 'c5': []}

All events for order-11 are in one partition, in order: placed, paid, delivered. With 5 consumers and 4 partitions, c5 sits idle. Choose enough partitions for the parallelism you will need, because adding partitions later changes which partition a key maps to.

Delivery guarantees

Networks and consumers fail, so a broker can promise one of three things:

• At most once: a message may be lost but is never processed twice (ack before processing).
• At least once: a message is never lost but may be processed twice (ack after processing). This is the most common choice.
• Exactly once: each message affects the result once. Kafka offers exactly-once semantics for reading from and writing to Kafka within its transactions, but once you call an outside system such as a payment gateway or an SMS provider, you need your own protection.

The practical rule: assume at-least-once delivery and make consumers idempotent, so handling the same message twice is harmless. The next lesson, Async Processing & Idempotency, shows how.

Poison messages and dead-letter queues

Some messages always fail, for example an order with a corrupt field. If the consumer keeps retrying it, it blocks everything behind it. The fix is a dead-letter queue (DLQ): after a few failed attempts, move the message aside for a human or a repair job, and carry on.

from collections import deque

def process(msg):
    if msg["amount"] < 0:
        raise ValueError("negative amount")
    return f"charged Rs {msg['amount']} for {msg['id']}"

queue = deque([{"id": "o1", "amount": 499, "tries": 0},
               {"id": "o2", "amount": -50, "tries": 0},
               {"id": "o3", "amount": 1299, "tries": 0}])
dead_letters, MAX_TRIES = [], 3

while queue:
    msg = queue.popleft()
    try:
        print(process(msg))
    except ValueError as err:
        msg["tries"] += 1
        if msg["tries"] >= MAX_TRIES:
            dead_letters.append((msg["id"], str(err)))
            print(f"{msg['id']} moved to DLQ after {msg['tries']} tries")
        else:
            queue.append(msg)              # retry later, after the others
print("DLQ:", dead_letters)

charged Rs 499 for o1
charged Rs 1299 for o3
o2 moved to DLQ after 3 tries
DLQ: [('o2', 'negative amount')]

Common mistakes

• Assuming exactly-once delivery and charging a customer twice when a message is redelivered.
• Too few partitions, capping how many consumers can work in parallel.
• Random keys when order matters, so "cancelled" is processed before "placed".
• No monitoring of consumer lag, so a stuck consumer goes unnoticed for hours.
• No dead-letter queue, letting one bad message block a whole partition.
• Using a queue where the user needs an immediate answer, such as "is this seat free?".

Key points

• Queues decouple services, buffer spikes and move slow work out of the request path.
• Classic queues delete messages after ack; Kafka-style logs keep them and let consumers replay.
• Messages with the same key go to the same partition and stay in order.
• Within a consumer group, each partition has one consumer, so partitions limit parallelism.
• Plan for at-least-once delivery with idempotent consumers, and use dead-letter queues for poison messages.

arrivals = [200, 300, 900, 900, 900, 900, 400, 200, 200, 200, 200, 200]

def max_backlog(capacity):
    backlog, worst = 0, 0
    for incoming in arrivals:
        backlog = max(0, backlog + incoming - capacity)
        worst = max(worst, backlog)
    return worst

capacity = 500
while max_backlog(capacity) > 600:
    capacity += 50
print(capacity, "msgs/s, worst backlog", max_backlog(capacity))

750 msgs/s, worst backlog 600