Async Processing & Idempotency

By the end of this lesson you will be able to move slow work into background jobs, make retries safe with idempotency keys, write consumers that can see a message twice without harm, and keep a database and a message queue in step with the outbox pattern.

Synchronous vs asynchronous work

In synchronous processing, the client waits until all the work is done. In asynchronous processing, the server accepts the request, replies quickly, and finishes the work in the background.

Think of a tailor in Jaipur. For a button, he stitches it while you wait (synchronous). For a full sherwani, he takes your measurements, gives you a receipt with a number, and says "come on Friday". You can call with the receipt number to ask about progress (asynchronous).

An asynchronous API often answers 202 Accepted with a link to check status:

POST /v1/reports/gst-summary HTTP/1.1
Host: api.ledger.example
Content-Type: application/json

{"month": "2026-10"}

HTTP/1.1 202 Accepted
Location: /v1/jobs/job_5f81
Content-Type: application/json

{"job_id": "job_5f81", "status": "QUEUED"}

The client then polls GET /v1/jobs/job_5f81, or the server calls a webhook (a URL the client registered) when the job finishes. Use async for anything slow or unreliable: reports, video processing, sending emails and SMS, calling partner systems.

The retry problem

Networks fail in an awkward way. When a request times out, the client cannot tell which of these happened:

1. The request never reached the server.
2. The server did the work, but the reply was lost.

If the client retries in case 2, the work happens twice. For "fetch my profile" that is harmless. For "pay ₹1,299" it means a double charge, an angry customer and a support ticket.

The solution is to make the operation idempotent: doing it many times has the same effect as doing it once. GET, PUT and DELETE are naturally idempotent (see API Design). For POST, you add an idempotency key.

Idempotency keys

The client generates a unique key (usually a UUID) for each logical action and sends it with every retry of that action, typically in an Idempotency-Key header. Payment providers such as Stripe document this pattern in their public API docs, and the IETF has a draft standard for the header.

The server keeps a table of keys it has seen:

1. New key: record it as "in progress", do the work, save the result against the key.
2. Key seen and finished: do not repeat the work; return the saved result.
3. Key seen and still in progress: return 409 Conflict so the client retries a little later.
4. Same key, different request body: reject it with an error; it is a client bug.

Step 1: run the simulation. The first reply is "lost", and the client retries.

import hashlib, json

class PaymentAPI:
    def __init__(self, use_keys):
        self.use_keys = use_keys
        self.charges = []                     # what really happened at the bank
        self.keys = {}                        # key -> (request hash, saved response)

    def charge(self, body, key=None):
        if self.use_keys:
            fingerprint = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
            if key in self.keys:
                saved_hash, saved_response = self.keys[key]
                if saved_hash != fingerprint:
                    return {"status": 422, "error": "key reused with a different body"}
                return saved_response         # replay: no new charge
        self.charges.append(body["amount"])
        response = {"status": 201, "charge_id": f"ch_{len(self.charges)}"}
        if self.use_keys:
            self.keys[key] = (fingerprint, response)
        return response

order = {"user": "isha", "amount": 1299}
for use_keys in [False, True]:
    api = PaymentAPI(use_keys)
    api.charge(order, key="k-7c1e")           # reply lost on the network
    second = api.charge(order, key="k-7c1e")  # client retries
    print(f"keys={use_keys}: retry got {second}, bank charged {api.charges}")

print(api.charge({"user": "isha", "amount": 5000}, key="k-7c1e"))

keys=False: retry got {'status': 201, 'charge_id': 'ch_2'}, bank charged [1299, 1299]
keys=True: retry got {'status': 201, 'charge_id': 'ch_1'}, bank charged [1299]
{'status': 422, 'error': 'key reused with a different body'}

Step 2: compare. Without keys, Isha paid twice. With keys, the retry returns the original result and the bank is charged once. Keys are usually kept for a limited time, such as 24 hours or a few days, which covers any realistic retry window.

In a real system the key table lives in the database, and a unique constraint on the key makes two simultaneous requests with the same key impossible to both succeed:

CREATE TABLE idempotency_keys (
  idem_key      VARCHAR(64) PRIMARY KEY,
  request_hash  CHAR(64)    NOT NULL,
  status        VARCHAR(12) NOT NULL,   -- IN_PROGRESS or DONE
  response_json TEXT,
  created_at    TIMESTAMP   NOT NULL
);

Idempotent consumers

Message queues usually deliver at least once (Message Queues), so a consumer can receive the same message twice. The fix is the same idea: remember which message IDs you have processed, and record that in the same database transaction as the work itself. Python's built-in sqlite3 is enough to show it:

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE processed (msg_id TEXT PRIMARY KEY)")
db.execute("CREATE TABLE wallet (user TEXT PRIMARY KEY, balance INTEGER)")
db.execute("INSERT INTO wallet VALUES ('rohan', 0)")

def handle(msg):
    try:
        with db:                                   # one transaction
            db.execute("INSERT INTO processed VALUES (?)", (msg["id"],))
            db.execute("UPDATE wallet SET balance = balance + ? WHERE user = ?",
                       (msg["cashback"], msg["user"]))
        return "applied"
    except sqlite3.IntegrityError:
        return "duplicate, skipped"                # the ID was already processed

deliveries = [{"id": "m-1", "user": "rohan", "cashback": 50},
              {"id": "m-2", "user": "rohan", "cashback": 25},
              {"id": "m-1", "user": "rohan", "cashback": 50}]   # redelivered
for m in deliveries:
    print(m["id"], handle(m))
print("balance:", db.execute("SELECT balance FROM wallet").fetchone()[0])

m-1 applied
m-2 applied
m-1 duplicate, skipped
balance: 75

Because the "processed" insert and the wallet update commit together, a crash between them cannot leave one done without the other.

The dual-write problem and the outbox pattern

A common bug: a service saves an order to its database, then publishes an OrderPlaced event to a queue. If the process crashes between the two steps, the order exists but nobody hears about it. If you publish first and the database write fails, others act on an order that does not exist. Writing to two systems without a shared transaction is the dual-write problem.

The transactional outbox fixes it:

1. In one database transaction, insert the order and a row in an outbox table describing the event.
2. A separate relay process reads new outbox rows and publishes them to the queue.
3. After publishing, it marks the rows as sent.

If the relay crashes after publishing but before marking, it publishes again, which is fine, because consumers are idempotent. Some teams use change data capture (reading the database's own change log) as the relay.

Sagas: transactions across services

When one business action spans several services (reserve stock, charge payment, book a rider), a single database transaction is not possible. A saga runs a sequence of local steps, and if one fails, runs compensating actions to undo the earlier ones: release the stock, refund the payment.

def run_saga(steps):
    done = []
    for name, action, compensate in steps:
        try:
            action()
            done.append((name, compensate))
            print(f"ok    {name}")
        except RuntimeError as err:
            print(f"FAIL  {name}: {err}")
            for prev_name, undo in reversed(done):
                undo()
                print(f"undo  {prev_name}")
            return "order cancelled"
    return "order confirmed"

def no_rider():
    raise RuntimeError("no rider within 5 km")

steps = [
    ("reserve stock",  lambda: None, lambda: None),
    ("charge Rs 640",  lambda: None, lambda: None),   # undo = refund
    ("assign rider",   no_rider,     lambda: None),
]
print(run_saga(steps))

ok    reserve stock
ok    charge Rs 640
FAIL  assign rider: no rider within 5 km
undo  charge Rs 640
undo  reserve stock
order cancelled

Compensations must themselves be idempotent and retried until they succeed, because a refund that silently fails is worse than no saga at all.

Common mistakes

• Retrying POST requests without idempotency keys, causing double orders and double charges.
• Checking "already processed?" and doing the work in separate transactions, leaving a gap for duplicates.
• Saving to the database and publishing to a queue as two independent steps (the dual-write problem).
• Generating a new idempotency key on each retry, which defeats the whole purpose.
• Sagas without compensation for every step, leaving money taken for orders that never happen.

Key points

• Move slow or unreliable work to background jobs; reply 202 Accepted with a status link.
• A timeout does not tell you whether the work happened, so retries must be safe.
• Idempotency keys let the server return the saved result for repeated requests.
• Idempotent consumers record processed message IDs in the same transaction as the work.
• The outbox pattern keeps database changes and published events in step; sagas undo partial work with compensations.

import hashlib, json

class PaymentAPI:
    def __init__(self):
        self.charges, self.keys = [], {}      # key -> (hash, response or None)

    def _fp(self, body):
        return hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()

    def start(self, key, body):
        self.keys[key] = (self._fp(body), None)

    def charge(self, body, key):
        fp = self._fp(body)
        if key in self.keys:
            saved_fp, saved = self.keys[key]
            if saved_fp != fp:
                return {"status": 422, "error": "key reused with a different body"}
            if saved is None:
                return {"status": 409, "error": "request in progress"}
            return saved
        self.charges.append(body["amount"])
        response = {"status": 201, "charge_id": f"ch_{len(self.charges)}"}
        self.keys[key] = (fp, response)
        return response

api = PaymentAPI()
body = {"user": "meera", "amount": 899}
api.start("k-42", body)
print(api.charge(body, "k-42"))

{'status': 409, 'error': 'request in progress'}