Hypothesis Testing Basics

Private school students in students.csv average a few marks more in maths than government school students. Is that a real difference, or could it be luck in who ended up in the data? Hypothesis testing is the standard way to answer that question with numbers instead of gut feeling.

In this lesson you will learn the logic of a test, what a p-value really means, and how to run the most common tests in scipy.stats: t-tests for comparing means, the Mann-Whitney test for skewed data, and the chi-square test for categories.

The logic of a test

Every test follows the same steps:

1. State the null hypothesis (H0): the boring explanation, "there is no difference" or "there is no relationship".
2. State the alternative hypothesis (H1): what you suspect, "the means are different".
3. Choose a significance level, usually α = 0.05, before looking at the result.
4. Compute a test statistic from the data and its p-value.
5. If p < α, reject H0: the data would be surprising if H0 were true. Otherwise, you fail to reject H0. That is not the same as proving it true.

The p-value is the probability of getting a result at least as extreme as yours if the null hypothesis were true. A small p-value says "chance alone would rarely produce this". It is not the probability that H0 is true, and it says nothing about how large or important the difference is.

One-sample t-test: is the mean different from a target?

A coaching centre claims its students average 65 in English. Test that claim with the practice data.

import pandas as pd
from scipy import stats

students = pd.read_csv("students.csv").drop_duplicates()
english = students["english"]

print("Sample mean:", round(english.mean(), 2), " n =", len(english))
res = stats.ttest_1samp(english, popmean=65)
print(f"t = {res.statistic:.2f}, p = {res.pvalue:.2g}")
print("Reject H0 at 0.05?", res.pvalue < 0.05)

Sample mean: 62.11  n = 480
t = -4.46, p = 1e-05
Reject H0 at 0.05? True

H0 is "the true mean is 65". The sample mean is about 3 marks lower, and with 480 students that difference is far too large to blame on chance: the p-value is tiny, so you reject H0. The t statistic measures how many standard errors the sample mean is from 65; the negative sign means it is below.

Two-sample t-test: comparing two groups

stats.ttest_ind() compares the means of two independent groups. Pass equal_var=False to use Welch's t-test, which does not assume both groups have the same spread. It is the safer default.

import pandas as pd
from scipy import stats

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
school = students["school_type"].str.strip().str.lower().str.rstrip(".")
students["school"] = school.map({"government": "Government", "govt": "Government", "private": "Private"})

gov = students.loc[students["school"] == "Government", "maths"].dropna()
pvt = students.loc[students["school"] == "Private", "maths"].dropna()
print(f"Government: mean {gov.mean():.1f} (n={len(gov)})   Private: mean {pvt.mean():.1f} (n={len(pvt)})")

res = stats.ttest_ind(pvt, gov, equal_var=False)
print(f"Welch t = {res.statistic:.2f}, p = {res.pvalue:.4f}")

ci = res.confidence_interval(0.95)
print(f"95% CI for the difference: {ci.low:.1f} to {ci.high:.1f} marks")

pooled_sd = ((gov.var() + pvt.var()) / 2) ** 0.5
print("Effect size (Cohen's d):", round((pvt.mean() - gov.mean()) / pooled_sd, 2))

Government: mean 55.8 (n=240)   Private: mean 60.4 (n=238)
Welch t = 2.80, p = 0.0053
95% CI for the difference: 1.4 to 7.7 marks
Effect size (Cohen's d): 0.26

The p-value is well below 0.05, so the difference is statistically significant: chance alone is an unlikely explanation. The confidence interval says the true difference is probably a few marks. Cohen's d, the difference in units of standard deviation, puts it in perspective: around 0.2 is usually called small, 0.5 medium and 0.8 large. So the gap is real but small. Remember too that this is observational data: private school families may differ in many ways, so the test shows a difference, not its cause.

Paired t-test: before and after

When the same people are measured twice, such as marks before and after a revision camp, the two measurements are linked. Use stats.ttest_rel(), which tests the average of the differences.

import numpy as np
from scipy import stats

rng = np.random.default_rng(8)
before = rng.normal(55, 12, size=25).round()
after = (before + rng.normal(4, 4, size=25)).round()     # a camp with a small average gain

print("Average gain:", round((after - before).mean(), 2))
paired = stats.ttest_rel(after, before)
print(f"Paired test:   p = {paired.pvalue:.4f}")
unpaired = stats.ttest_ind(after, before, equal_var=False)
print(f"Unpaired test: p = {unpaired.pvalue:.4f}")

Average gain: 3.08
Paired test:   p = 0.0034
Unpaired test: p = 0.3955

The paired test detects the gain clearly, while the unpaired test, which ignores that each "after" belongs to a "before", sees only two overlapping clouds of marks and misses it. Choosing the right test for the design of your data matters as much as the calculation.

Mann-Whitney U: when data is skewed

t-tests compare means and work best when the data is roughly normal or the samples are large. For small or strongly skewed samples, the Mann-Whitney U test compares the two groups using ranks instead of raw values.

import pandas as pd
from scipy import stats

sales = pd.read_csv("sales.csv", parse_dates=["order_date"]).drop_duplicates()
sales = sales[sales["quantity"].between(1, 20)]
sales["value"] = sales["quantity"] * sales["unit_price"] * (1 - sales["discount"] / 100)
weekend = sales["order_date"].dt.dayofweek >= 5

wk, wd = sales.loc[weekend, "value"], sales.loc[~weekend, "value"]
print(f"Median order: weekend ₹{wk.median():.0f}, weekday ₹{wd.median():.0f}")

res = stats.mannwhitneyu(wk, wd, alternative="two-sided")
print(f"Mann-Whitney p = {res.pvalue:.3f}")

Median order: weekend ₹1098, weekday ₹1099
Mann-Whitney p = 0.952

Order values are right-skewed, so a rank-based test is a sensible choice. The p-value is large, so you fail to reject H0: there is no evidence that weekend orders are bigger or smaller than weekday ones. The shop gets more orders at weekends, as you saw in Dates & Time Series, but not bigger ones.

Chi-square test: are two categories related?

For two categorical variables, the chi-square test of independence checks whether they are related. It compares the counts in a crosstab with the counts you would expect if there were no relationship.

import pandas as pd
from scipy import stats

emp = pd.read_csv("employees.csv").drop_duplicates()

table = pd.crosstab(emp["rating"], emp["left"])
print(table)

res = stats.chi2_contingency(table)
print(f"chi2 = {res.statistic:.1f}, p = {res.pvalue:.2g}, dof = {res.dof}")
print("Smallest expected count:", res.expected_freq.min().round(1))
print(pd.crosstab(emp["rating"], emp["left"], normalize="index").round(2)["Yes"])

left     No  Yes
rating          
1         6    4
2        30    6
3       106   27
4        82   10
5        29    0
chi2 = 13.7, p = 0.0085, dof = 4
Smallest expected count: 1.6
rating
1    0.40
2    0.17
3    0.20
4    0.11
5    0.00
Name: Yes, dtype: float64

H0 is "whether an employee left does not depend on their rating". The p-value is below 0.01, so you reject it. The share who left falls as the rating rises, from 40% at rating 1 to nobody at rating 5. The chi-square test needs reasonably large counts: a common rule of thumb is an expected count of at least 5 in every cell, and res.expected_freq shows them. Here the smallest is below 5, because only 10 employees have rating 1, so a careful analyst would merge ratings 1 and 2 into one group and test again before relying on the result.

Errors and good practice

Two kinds of mistake are possible:

Common mistakes

• Saying "p = 0.03 means a 3% chance H0 is true". The p-value assumes H0 is true; it is the chance of data this extreme under that assumption.
• Treating "not significant" as "no difference". A small sample may simply be too weak to detect it.
• Ignoring effect size. With huge samples, tiny, useless differences become significant. Report the difference and its confidence interval.
• Using an unpaired test on paired data, or a t-test on very skewed small samples.

Key points

• A test starts from a null hypothesis of no effect and asks how surprising the data would be if it were true.
• If p < α (usually 0.05), reject H0; otherwise you fail to reject it, which does not prove it.
• ttest_1samp, ttest_ind(equal_var=False) and ttest_rel compare means; mannwhitneyu uses ranks for skewed data.
• chi2_contingency tests whether two categorical variables are related, using a crosstab.
• Report effect sizes and confidence intervals, and beware of running many tests.

import pandas as pd
from scipy import stats

emp = pd.read_csv("employees.csv").drop_duplicates().dropna(subset=["salary"])
remote = emp.loc[emp["work_mode"] == "Remote", "salary"]
office = emp.loc[emp["work_mode"] == "Office", "salary"]

print(f"Remote mean ₹{remote.mean():,.0f}, Office mean ₹{office.mean():,.0f}")
res = stats.ttest_ind(remote, office, equal_var=False)
print(f"p = {res.pvalue:.3f}")
print("Significant difference" if res.pvalue < 0.05 else "No significant difference at 0.05")

chi = stats.chi2_contingency(pd.crosstab(emp["work_mode"], emp["left"]))
print(f"work_mode vs left: p = {chi.pvalue:.3f}")

Remote mean ₹806,535, Office mean ₹883,492
p = 0.127
No significant difference at 0.05
work_mode vs left: p = 0.889