Chi-Square Tests

t-tests compare means of numbers. Many questions, though, are about counts in categories: is a die fair, do customers really pay by UPI as often as the shop thinks, does the way people commute depend on the city they live in? The chi-square test (χ², pronounced "kai-square") handles these questions by comparing the counts you observed with the counts you would expect if H0 were true.

In this lesson you will run the chi-square goodness-of-fit test and the chi-square test of independence, compute expected counts and the χ² statistic by hand, check the results with scipy.stats, measure the strength of a relationship with Cramér's V, and learn what to do when counts are small.

The chi-square statistic

Both tests use the same statistic:

χ² = Σ (observed - expected)² / expected

Each cell contributes its squared gap between observed and expected counts, scaled by the expected count. If H0 is true, observed and expected counts are close and χ² is small. A large χ² means the data does not fit H0. The p-value comes from the chi-square distribution with the right degrees of freedom (df), using stats.chi2.sf().

Goodness of fit: is the die fair?

The goodness-of-fit test checks whether one categorical variable follows a claimed distribution. A die is rolled 120 times (invented results). If it is fair, each face should appear about 120 / 6 = 20 times.

import numpy as np
from scipy import stats

observed = np.array([25, 17, 15, 23, 24, 16])     # 120 rolls of a die (invented)
expected = np.full(6, observed.sum() / 6)
chi2 = ((observed - expected) ** 2 / expected).sum()
df = observed.size - 1
print("Expected:", expected)
print(f"By hand: chi2 = {chi2:.2f}, df = {df}, p = {stats.chi2.sf(chi2, df):.4f}")
res = stats.chisquare(observed)
print(f"scipy:   chi2 = {res.statistic:.2f}, p = {res.pvalue:.4f}")

Expected: [20. 20. 20. 20. 20. 20.]
By hand: chi2 = 5.00, df = 5, p = 0.4159
scipy:   chi2 = 5.00, p = 0.4159

By hand: (25-20)²/20 + (17-20)²/20 + (15-20)²/20 + (23-20)²/20 + (24-20)²/20 + (16-20)²/20 = (25 + 9 + 25 + 9 + 16 + 16) / 20 = 5.0. For goodness of fit, df = number of categories - 1 = 5, because once five counts are known the sixth is fixed by the total. The p-value 0.42 is large: counts this uneven are common for a fair die, so there is no evidence of a loaded die.

Goodness of fit with unequal expected shares

The claimed distribution need not be equal. A Rohtak shop owner believes that 50% of payments are by UPI, 30% by cash and 20% by card. In 200 payments (invented) she counts 118 UPI, 46 cash and 36 card.

import numpy as np
from scipy import stats

modes = ["UPI", "Cash", "Card"]
observed = np.array([118, 46, 36])                 # 200 payments at a Rohtak shop (invented)
claimed = np.array([0.50, 0.30, 0.20])
expected = claimed * observed.sum()
print("Expected:", expected)
res = stats.chisquare(observed, f_exp=expected)
print(f"chi2 = {res.statistic:.3f}, p = {res.pvalue:.4f}")
print("Contributions:", ((observed - expected) ** 2 / expected).round(2))

Expected: [100.  60.  40.]
chi2 = 6.907, p = 0.0316
Contributions: [3.24 3.27 0.4 ]

With p = 0.032 < 0.05 the owner's belief does not fit the data. The contributions show where: more UPI and less cash than she thought, while card is close to the claim. The expected counts passed to f_exp must add up to the same total as the observed counts.

Test of independence

The test of independence checks whether two categorical variables are related, using a two-way table. H0 says they are independent: knowing one tells you nothing about the other.

If H0 is true, the expected count for each cell comes from the multiplication rule for independent events:

expected = row total x column total / grand total
df = (rows - 1) x (columns - 1)

Here is an invented survey of 300 commuters in three cities:

import numpy as np
import pandas as pd
from scipy import stats

table = pd.DataFrame(
    {"Bus": [40, 30, 50], "Metro": [45, 25, 10], "Two-wheeler": [35, 45, 20]},
    index=["Pune", "Kochi", "Lucknow"],
)
obs = table.to_numpy()
row_totals = obs.sum(axis=1, keepdims=True)
col_totals = obs.sum(axis=0, keepdims=True)
expected = row_totals * col_totals / obs.sum()
chi2 = ((obs - expected) ** 2 / expected).sum()
df = (obs.shape[0] - 1) * (obs.shape[1] - 1)
print(pd.DataFrame(expected, index=table.index, columns=table.columns))
print(f"By hand: chi2 = {chi2:.2f}, df = {df}, p = {stats.chi2.sf(chi2, df):.2e}")

res = stats.chi2_contingency(table)
print(f"scipy:   chi2 = {res.statistic:.2f}, df = {res.dof}, p = {res.pvalue:.2e}")
print(f"Cramer's V = {stats.contingency.association(obs, method='cramer'):.3f}")

          Bus      Metro  Two-wheeler
Pune     48.0  32.000000    40.000000
Kochi    40.0  26.666667    33.333333
Lucknow  32.0  21.333333    26.666667
By hand: chi2 = 31.74, df = 4, p = 2.16e-06
scipy:   chi2 = 31.74, df = 4, p = 2.16e-06
Cramer's V = 0.230

Pune has 120 of the 300 commuters and 120 use the bus, so if city and mode were unrelated, Pune would have 120 x 120 / 300 = 48 bus users. It has 40. The biggest gaps are Pune's 45 metro users (32 expected) and Lucknow's 50 bus users (32 expected). The p-value of about 2 in a million says the mode of travel clearly depends on the city. keepdims=True keeps the totals as a column and a row, so multiplying them gives the full table of expected counts.

A p-value says whether there is a relationship, not how strong it is. Cramér's V measures strength on a 0 to 1 scale (0 means no association). A value of 0.23 is a moderate association.

2 x 2 tables and Yates' correction

For a 2 x 2 table (df = 1), chi2_contingency() applies Yates' continuity correction by default, which makes the test a little more cautious. Test whether tuition depends on gender in students.csv:

import pandas as pd
from scipy import stats

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
students["gender"] = students["gender"].str.upper()
table = pd.crosstab(students["gender"], students["tuition"])
print(table)
with_yates = stats.chi2_contingency(table)
without = stats.chi2_contingency(table, correction=False)
print(f"With Yates correction:    chi2 = {with_yates.statistic:.3f}, p = {with_yates.pvalue:.4f}")
print(f"Without correction:       chi2 = {without.statistic:.3f}, p = {without.pvalue:.4f}")

tuition   No  Yes
gender           
F        118  101
M        156  105
With Yates correction:    chi2 = 1.454, p = 0.2279
Without correction:       chi2 = 1.686, p = 0.1942

In the frequency tables lesson you saw that 46.1% of girls and 40.2% of boys take tuition. Either way, the p-value is around 0.2, so that gap could easily be chance. With large counts the correction makes little difference; with small counts it matters more. When you compare two proportions, the uncorrected χ² equals the square of the two-proportion z statistic you will use in A/B testing.

When counts are small: Fisher's exact test

The chi-square p-value is an approximation that works when expected counts are large enough. A common rule is that every expected count should be at least 5. For small 2 x 2 tables, use Fisher's exact test, which computes the p-value exactly.

from scipy import stats

small = [[7, 1], [2, 6]]
print("Expected counts:", stats.chi2_contingency(small).expected_freq.round(1).tolist())
print(f"Fisher's exact p = {stats.fisher_exact(small).pvalue:.4f}")

Expected counts: [[4.5, 3.5], [4.5, 3.5]]
Fisher's exact p = 0.0406

Every expected count is below 5, so the chi-square approximation is unreliable here; Fisher's exact test is the right choice. For larger tables with small counts, merge sparse categories (such as rating 1 and rating 2) before testing.

Assumptions

• Counts, not percentages. The test needs raw frequencies. Turning 118 out of 200 into 59% throws away the sample size.
• Independent observations. Each person or item is counted once, in one cell.
• Large enough expected counts, usually at least 5 in every cell.

Common mistakes

• Running the test on percentages or means. χ² only works with counts.
• Ignoring small expected counts. Check res.expected_freq.
• Reading a small p-value as a strong relationship. With thousands of rows, tiny associations are significant; report Cramér's V or the row percentages.
• Treating independence as causation. City and commuting mode are related, but the test does not say why.

Key points

• χ² = Σ (observed - expected)² / expected compares observed counts with the counts H0 predicts.
• Goodness of fit: one variable against claimed shares, df = categories - 1 (stats.chisquare).
• Independence: two variables in a table, expected = row total x column total / grand total, df = (r - 1)(c - 1) (stats.chi2_contingency).
• Expected counts should be at least 5; for small 2 x 2 tables use stats.fisher_exact.
• Cramér's V measures the strength of an association; the p-value does not.

import pandas as pd
from scipy import stats

emp = pd.read_csv("employees.csv").drop_duplicates()
table = pd.crosstab(emp["work_mode"], emp["left"])
print(table)
res = stats.chi2_contingency(table)
print(res.expected_freq.round(1))
print(f"chi2 = {res.statistic:.3f}, df = {res.dof}, p = {res.pvalue:.3f}")
print(f"Cramer's V = {stats.contingency.association(table.to_numpy(), method='cramer'):.3f}")

left        No  Yes
work_mode          
Hybrid     112   22
Office     103   19
Remote      38    6
[[113.   21. ]
 [102.9  19.1]
 [ 37.1   6.9]]
chi2 = 0.195, df = 2, p = 0.907
Cramer's V = 0.026