Design a Payment System

By the end of this lesson you will have designed the payment part of an e-commerce or ticketing app: an API that never charges twice, a state machine that survives timeouts and late webhooks, a double-entry ledger that always balances, and a reconciliation job that catches what everything else missed.

Step 1: requirements

Our app (say, an online bookstore in Lucknow) does not move money itself. It works with a payment service provider (PSP), also called a payment gateway or aggregator, which talks to banks, card networks and UPI. Our system must:

• Create a payment for an order and send the customer to pay by UPI, card or net banking.
• Know the final result reliably, even if the customer closes the app mid-payment.
• Never charge a customer twice for one order, and never mark an unpaid order as paid.
• Support refunds.
• Keep an accurate record of every rupee for accounting, audits and disputes.

Non-functional: correctness beats speed. A payment page that takes one extra second is acceptable; a double charge is not.

Think of buying a train ticket at a counter with cash. The clerk writes the amount in a register, gives you a receipt with a number, and at the end of the day counts the cash drawer against the register. If they do not match, someone investigates before going home. Payment systems automate that same discipline: unique receipts, a careful register and a daily count.

Step 2: estimates

orders_per_day = 400_000
avg_order_rupees = 650
peak_factor = 5                         # sale days and evenings
webhooks_per_payment = 2                # e.g. "authorised" and "captured"

tps = orders_per_day / 86_400
print(f"Payments: {tps:.1f}/s average, {tps * peak_factor:.0f}/s at peak")
print(f"Webhooks: {tps * webhooks_per_payment * peak_factor:.0f}/s at peak")
print(f"Daily volume: Rs {orders_per_day * avg_order_rupees:,}")
print(f"Ledger rows per year: {orders_per_day * 365 * 2:,} (two entries per payment)")

Payments: 4.6/s average, 23/s at peak
Webhooks: 46/s at peak
Daily volume: Rs 260,000,000
Ledger rows per year: 292,000,000 (two entries per payment)

The load is small. A well-indexed relational database with ACID transactions handles it easily. The difficulty is correctness, not scale, which is why this design is about states, idempotency and records.

Step 3: the payment flow

1. The customer taps "Pay ₹650" for order ord_5521.
2. Our payment service creates a payment record (CREATED) and calls the PSP's "create payment" API with an idempotency key tied to this payment.
3. The PSP returns a payment ID and a link or UPI intent; the app sends the customer there.
4. The customer approves in their UPI app or bank page.
5. The PSP sends us a webhook (an HTTP call to our server) saying the payment succeeded or failed. The app also polls our status API.
6. We update the payment to SUCCEEDED, write ledger entries, and mark the order paid, all in one transaction.

Any step can fail: our call to the PSP can time out, the customer can vanish, the webhook can arrive twice, late or not at all.

Step 4: a state machine

A state machine lists the allowed states and the allowed moves between them. Anything else is rejected. This stops bugs such as a late "failed" webhook overwriting a payment that already succeeded.

ALLOWED = {
    "CREATED":   {"PENDING", "FAILED"},
    "PENDING":   {"SUCCEEDED", "FAILED"},
    "SUCCEEDED": {"REFUNDED"},
    "FAILED":    set(),
    "REFUNDED":  set(),
}

class Payment:
    def __init__(self, pid):
        self.pid, self.state, self.seen_events = pid, "CREATED", set()

    def apply(self, event_id, new_state):
        if event_id in self.seen_events:
            return f"{event_id}: duplicate ignored"
        self.seen_events.add(event_id)
        if new_state == self.state:
            return f"{event_id}: already {new_state}"
        if new_state not in ALLOWED[self.state]:
            return f"{event_id}: {self.state} -> {new_state} not allowed, flag for review"
        self.state = new_state
        return f"{event_id}: now {new_state}"

p = Payment("pay_901")
for event_id, state in [("ev1", "PENDING"), ("ev2", "SUCCEEDED"), ("ev2", "SUCCEEDED"),
                        ("ev3", "FAILED"), ("ev4", "REFUNDED")]:
    print(p.apply(event_id, state))

ev1: now PENDING
ev2: now SUCCEEDED
ev2: duplicate ignored
ev3: SUCCEEDED -> FAILED not allowed, flag for review
ev4: now REFUNDED

The duplicate webhook is ignored, and the impossible "failed after succeeded" event is flagged for a human instead of silently corrupting the record.

Step 5: timeouts and unknown results

When our call to the PSP times out, we do not know whether the payment was created. Never assume failure and never blindly create a second payment. Instead:

• Retry with the same idempotency key, so the PSP returns the original payment if it exists (Async Processing & Idempotency).
• Keep the payment PENDING and query the PSP's status API on a schedule (after 30 seconds, 2 minutes, 10 minutes...) until it is final.
• Tell the customer honestly: "Payment is being confirmed. If money was debited, it will be confirmed or refunded automatically."

UPI payments in particular can stay pending for a while, so the "query until final" loop is essential, not optional.

Step 6: webhooks you can trust

A webhook is an HTTP request from the internet to your server, so anyone could fake one: "payment for order 5521 succeeded". PSPs therefore sign webhooks with a shared secret, and you must verify the signature before acting.

import hashlib, hmac, json

WEBHOOK_SECRET = b"demo-secret-load-real-one-from-a-secrets-manager"

def psp_sends(payload):                     # what the PSP does
    body = json.dumps(payload, separators=(",", ":")).encode()
    sig = hmac.new(WEBHOOK_SECRET, body, hashlib.sha256).hexdigest()
    return body, sig

def we_receive(body, signature):            # what our endpoint does
    expected = hmac.new(WEBHOOK_SECRET, body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        return 401, "bad signature, ignored"
    event = json.loads(body)
    return 200, f"verified {event['event_id']}: {event['payment_id']} -> {event['status']}"

body, sig = psp_sends({"event_id": "ev2", "payment_id": "pay_901", "status": "SUCCEEDED"})
print(we_receive(body, sig))
fake = body.replace(b"pay_901", b"pay_999")
print(we_receive(fake, sig))

(200, 'verified ev2: pay_901 -> SUCCEEDED')
(401, 'bad signature, ignored')

Verify over the raw request body exactly as received, before parsing. Then apply the event through the state machine, and return 200 quickly; slow processing makes the PSP retry, causing more duplicates. For extra safety, confirm important events by calling the PSP's status API.

Step 7: the ledger

The ledger is the permanent record of money movements. Payment systems use double-entry bookkeeping: every movement is recorded as two entries, a debit in one account and a credit in another, so the total of all entries is always zero. If the books do not balance, something is wrong, and you know immediately.

Ledger rows are append-only. You never edit or delete a row; a refund is new rows that reverse the original.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE ledger (
    id INTEGER PRIMARY KEY, txn TEXT, account TEXT, amount_paise INTEGER)""")

def record(txn, entries):
    assert sum(amount for _, amount in entries) == 0, "entries must balance"
    with db:                                         # all entries or none
        db.executemany("INSERT INTO ledger (txn, account, amount_paise) VALUES (?, ?, ?)",
                       [(txn, acct, amt) for acct, amt in entries])

record("pay_901", [("psp_receivable", 65_000), ("merchant_sales", -65_000)])
record("pay_902", [("psp_receivable", 120_000), ("merchant_sales", -120_000)])
record("ref_901", [("merchant_sales", 65_000), ("psp_receivable", -65_000)])   # refund

for account, total in db.execute(
        "SELECT account, SUM(amount_paise) FROM ledger GROUP BY account ORDER BY account"):
    print(f"{account:15} Rs {total / 100:>9,.2f}")
print("books balance:", db.execute("SELECT SUM(amount_paise) FROM ledger").fetchone()[0] == 0)

merchant_sales  Rs -1,200.00
psp_receivable  Rs  1,200.00
books balance: True

Store money as integers in the smallest unit (paise), never as floating-point numbers, which cannot represent values like 0.1 exactly.

Step 8: reconciliation, design and trade-offs

Even with all of the above, records can drift: a webhook lost for good, a bug, a bank reversal. Reconciliation is a daily job that compares our records with the PSP's settlement report, line by line, and raises every mismatch for investigation.

App --> Order service --> Payment service --(create, idempotency key)--> PSP --> banks / UPI
                              |      ^                                     |
                              |      +----- signed webhooks ---------------+
                              |      +----- status polling (pending) ------+
                              v
                 Payments DB (state machine) + Ledger (append-only, double-entry)
                              |
                              v  events: PaymentSucceeded (outbox)
                 Queue --> order fulfilment, emails, analytics

Daily: PSP settlement file --> Reconciliation job --> mismatch queue for finance team

Trade-offs and rules to state in an interview:

• Relational database with ACID transactions for payments and the ledger; scale is modest and correctness is everything.
• Synchronous replication for the payments database, because an RPO of zero matters here (Reliability).
• Never store card numbers. Card data security is governed by the PCI DSS standard, and RBI rules on card tokenisation stop merchants from storing card numbers; let the PSP's hosted page or SDK collect card details, and keep only tokens and references.
• Events via the outbox pattern, so "payment succeeded" and the event announcing it can never disagree.

Common mistakes

• Treating a timeout as a failure and creating a second payment.
• Trusting unsigned webhooks or verifying after parsing and re-serialising the body.
• Updating payment status without a state machine, letting late events overwrite final states.
• Floats for money, or editing ledger rows instead of adding reversing entries.
• No reconciliation, so small errors pile up unnoticed for months.

Key points

• Payments are about correctness, not scale: ACID transactions, idempotency and careful states.
• Use idempotency keys with the PSP, and on timeouts keep the payment pending and poll for the real result.
• A state machine accepts only valid transitions and ignores duplicate events.
• Verify webhook signatures over the raw body, respond fast, and process idempotently.
• A double-entry, append-only ledger in paise always balances; daily reconciliation catches the rest.

ours = {"pay_1": 65000, "pay_2": 120000, "pay_3": 49900}
psp = {"pay_1": 65000, "pay_2": 112000, "pay_4": 30000}

for pid in sorted(psp.keys() - ours.keys()):
    print(f"{pid}: paid at PSP, not marked succeeded by us")
for pid in sorted(ours.keys() - psp.keys()):
    print(f"{pid}: succeeded with us, missing at PSP")
for pid in sorted(ours.keys() & psp.keys()):
    if ours[pid] != psp[pid]:
        print(f"{pid}: amount differs, ours Rs {ours[pid] / 100:.2f}, PSP Rs {psp[pid] / 100:.2f}")

pay_4: paid at PSP, not marked succeeded by us
pay_3: succeeded with us, missing at PSP
pay_2: amount differs, ours Rs 1200.00, PSP Rs 1120.00