Reading A/B Test Results

By the end of this lesson you will be able to read an A/B test result table, calculate the lift, check whether the difference is likely to be real with a p-value and a confidence interval, catch a broken test with a sample ratio check, and write the result in plain words for a manager. The maths stays light; the Statistics course explains the theory in full.

An A/B test (also called a split test) shows two versions of something to two random groups of users and compares the results. Version A is usually the current one, called the control. Version B is the new one, called the variant or treatment. Think of a sweet shop owner in Lucknow who puts the old price label on half the boxes and a new label on the other half, at random, and counts which half sells better. Because the halves were chosen at random, any big difference is likely caused by the label.

Step 1: read the result table

Most testing tools give you a table like this. A food delivery app tested a new "Order again" button on the home screen.

results = {
    "A (control)": {"users": 20000, "orders": 1640},
    "B (new button)": {"users": 20100, "orders": 1810},
}

for group, r in results.items():
    rate = r["orders"] / r["users"] * 100
    print(f'{group:15} users {r["users"]:6,}  orders {r["orders"]:5,}  conversion {rate:.2f}%')

a = results["A (control)"]["orders"] / results["A (control)"]["users"]
b = results["B (new button)"]["orders"] / results["B (new button)"]["users"]
print(f"Absolute lift: {(b - a) * 100:+.2f} percentage points")
print(f"Relative lift: {(b - a) / a * 100:+.1f}%")

A (control)     users 20,000  orders 1,640  conversion 8.20%
B (new button)  users 20,100  orders 1,810  conversion 9.00%
Absolute lift: +0.80 percentage points
Relative lift: +9.8%

What you should see: conversion of 8.20% for A and about 9.00% for B. There are two ways to describe the difference, and people mix them up all the time:

• Absolute lift: 9.00% − 8.20% = about 0.8 percentage points.
• Relative lift: 0.8 ÷ 8.20 = about 10% better than control.

Both are correct. Always say which one you mean. "Conversion went up 10%" can sound like "from 8% to 18%" to a busy reader.

Step 2: is the difference real?

Even if the button did nothing, two random groups would never convert at exactly the same rate. The question is whether the difference is bigger than random chance usually produces. Two tools answer this:

• The p-value is the probability of seeing a difference at least this big if the new version truly made no difference. A small p-value (commonly below 0.05) means "chance alone rarely produces this".
• The 95% confidence interval is a range of plausible values for the true difference. If the whole range is above zero, B is very likely better.

Here is the standard two-proportion test, written with Python's math module only:

import math

users_a, orders_a = 20000, 1640
users_b, orders_b = 20100, 1810
pa, pb = orders_a / users_a, orders_b / users_b

pooled = (orders_a + orders_b) / (users_a + users_b)
se_pooled = math.sqrt(pooled * (1 - pooled) * (1 / users_a + 1 / users_b))
z = (pb - pa) / se_pooled
p_value = math.erfc(abs(z) / math.sqrt(2))          # two-sided

se_diff = math.sqrt(pa * (1 - pa) / users_a + pb * (1 - pb) / users_b)
low, high = (pb - pa) - 1.96 * se_diff, (pb - pa) + 1.96 * se_diff

print(f"z = {z:.2f}, p-value = {p_value:.4f}")
print(f"95% CI for the difference: {low * 100:+.2f} to {high * 100:+.2f} percentage points")

z = 2.87, p-value = 0.0041
95% CI for the difference: +0.26 to +1.35 percentage points

What you should see: a p-value well below 0.05, and a confidence interval that is entirely above zero. In plain words: "The new button very likely increases conversion. Our best estimate is +0.8 points, and the true effect is probably between about +0.3 and +1.4 points."

Step 3: check that the test is not broken

Before trusting any result, check the split. If the tool was meant to send 50% of users to each version, the group sizes should be close to equal. A big imbalance, called a sample ratio mismatch (SRM), usually means a bug: for example, the new version crashed for some phones, and those users were never counted. Then the groups are no longer comparable, and the result cannot be trusted, however good it looks.

import math

def srm_check(users_a, users_b, expected_share_a=0.5):
    total = users_a + users_b
    exp_a, exp_b = total * expected_share_a, total * (1 - expected_share_a)
    chi2 = (users_a - exp_a) ** 2 / exp_a + (users_b - exp_b) ** 2 / exp_b
    p = math.erfc(math.sqrt(chi2 / 2))               # chi-square test with 1 degree of freedom
    return p

for a, b in [(20000, 20100), (20000, 21200)]:
    p = srm_check(a, b)
    verdict = "looks fine" if p >= 0.001 else "MISMATCH: investigate before reading results"
    print(f"A={a:,}  B={b:,}  p={p:.4f}  -> {verdict}")

A=20,000  B=20,100  p=0.6175  -> looks fine
A=20,000  B=21,200  p=0.0000  -> MISMATCH: investigate before reading results

What you should see: the first split passes, and the second fails even though 20,000 against 21,200 does not look dramatic. With large groups, a gap of 1,200 users is very unlikely to be chance. Teams usually use a strict threshold such as 0.001 for this check, because a false alarm only costs an investigation.

Step 4: was the test big enough?

A test that is too small cannot detect a real but modest improvement. A well-known rule of thumb for a conversion test (5% significance, 80% chance of detecting the effect) is:

users needed per group ≈ 16 × p × (1 − p) ÷ (smallest difference worth detecting)²

Here p is the current conversion rate. This is an approximation; use a proper calculator or the statistics course for real planning.

baseline = 0.082          # current conversion 8.2%
for diff in (0.002, 0.005, 0.010):
    n = 16 * baseline * (1 - baseline) / diff ** 2
    print(f"To detect +{diff * 100:.1f} points: about {n:,.0f} users per group")

To detect +0.2 points: about 301,104 users per group
To detect +0.5 points: about 48,177 users per group
To detect +1.0 points: about 12,044 users per group

What you should see: detecting a tiny change needs hundreds of thousands of users per group, while a 1-point change needs about 12,000. That is why small apps often cannot test small tweaks, and should test bolder changes instead.

A checklist for reading any A/B test

1. Check the split (sample ratio) before anything else.
2. Check the duration: at least one full week, so weekdays and weekends are both included.
3. Do not stop early just because the result looks good on day 3. Checking again and again and stopping at the first "significant" moment, called peeking, produces many false wins.
4. Read lift with its interval, not a single number.
5. Check guardrail metrics: did the new version hurt something else, such as cancellations, refunds or app crashes?
6. Ask whether it matters: a statistically real +0.05 points may not be worth the engineering cost. This is practical significance.

Common mistakes

• Mixing absolute and relative lift. Say "percentage points" for absolute differences.
• Reading p-values as the chance B is better. They are not.
• Ignoring a sample ratio mismatch. A broken split invalidates the whole test.
• Stopping early on a good day. Fix the duration before the test starts.

Key points

• An A/B test compares a control and a variant shown to random groups.
• Report both absolute lift (percentage points) and relative lift (%).
• A small p-value and a confidence interval above zero suggest a real improvement.
• Always check sample ratio, duration and guardrail metrics before trusting a result.
• Small effects need very large samples; statistical and practical significance are different.

import math

ua, oa, ub, ob = 15200, 610, 15050, 655
pa, pb = oa / ua, ob / ub
pooled = (oa + ob) / (ua + ub)
z = (pb - pa) / math.sqrt(pooled * (1 - pooled) * (1 / ua + 1 / ub))
p_value = math.erfc(abs(z) / math.sqrt(2))
se = math.sqrt(pa * (1 - pa) / ua + pb * (1 - pb) / ub)
print(f"A {pa:.2%}  B {pb:.2%}  abs {(pb - pa) * 100:+.2f} pts  rel {(pb - pa) / pa:+.1%}")
print(f"p = {p_value:.3f}, 95% CI {(pb - pa - 1.96 * se) * 100:+.2f} to {(pb - pa + 1.96 * se) * 100:+.2f} pts")