ANOVA

A t-test compares two groups. What if you have three teaching methods, five cities or six departments? Running a t-test for every pair seems natural, but it quietly inflates your chance of a false alarm. Analysis of variance (ANOVA) tests all the groups at once with a single question: are all the group means equal, or is at least one different?

In this lesson you will see why many t-tests are a problem, compute a one-way ANOVA by hand by splitting the variation in the data, run it with scipy.stats.f_oneway, find which groups differ with Tukey's HSD test, and check the assumptions with Levene's test and the rank-based Kruskal-Wallis test.

Why not just run several t-tests?

With 3 groups there are 3 pairs to compare; with 5 groups, 10 pairs. Each test at α = 0.05 has a 5% false-alarm rate, and the chances add up. If the tests were independent, the chance of at least one false alarm in 3 tests would be 1 - 0.95³ = 0.143, and in 10 tests 1 - 0.95¹⁰ = 0.401. ANOVA keeps the overall false-alarm rate at α by asking one question first:

H0: μ1 = μ2 = ... = μk      (all group means are equal)
H1: at least one mean is different

Splitting the variation

The idea behind ANOVA is to split the total variation in the data into two parts:

• Between groups: how far the group means are from the overall (grand) mean.
• Within groups: how far individual values are from their own group mean.

If the group means are truly equal, the between-group variation should be about as large as you would expect from the within-group noise. If it is much larger, the groups differ.

SST (total)   = Σ (x - grand mean)²
SSB (between) = Σ n_j x (group mean_j - grand mean)²
SSW (within)  = Σ (x - own group mean)²            SST = SSB + SSW

MSB = SSB / (k - 1)        MSW = SSW / (N - k)
F   = MSB / MSW            df = (k - 1, N - k)

k is the number of groups and N the total number of observations. The F statistic is a ratio of two variance estimates; under H0 it is close to 1, and the p-value comes from the F distribution.

One-way ANOVA by hand

Eighteen students (invented data) were taught a topic by three methods: lectures only, video lessons, or practice problems, 6 students each.

import numpy as np
from scipy import stats

groups = {
    "Lecture":  np.array([62, 58, 65, 60, 55, 61]),
    "Videos":   np.array([66, 70, 64, 68, 63, 69]),
    "Practice": np.array([72, 68, 75, 70, 74, 71]),
}
everyone = np.concatenate(list(groups.values()))
grand = everyone.mean()
k, N = len(groups), everyone.size

ssb = sum(g.size * (g.mean() - grand) ** 2 for g in groups.values())
ssw = sum(((g - g.mean()) ** 2).sum() for g in groups.values())
msb, msw = ssb / (k - 1), ssw / (N - k)
F = msb / msw
for name, g in groups.items():
    print(f"{name:<9} mean {g.mean():.2f}")
print(f"Grand mean {grand:.2f}")
print(f"SSB = {ssb:.2f}, SSW = {ssw:.2f}, SST = {((everyone - grand) ** 2).sum():.2f}")
print(f"MSB = {msb:.2f}, MSW = {msw:.2f}, F = {F:.2f}, p = {stats.f.sf(F, k - 1, N - k):.2e}")
res = stats.f_oneway(*groups.values())
print(f"f_oneway: F = {res.statistic:.2f}, p = {res.pvalue:.2e}")
print(f"Eta squared = {ssb / (ssb + ssw):.3f}")

Lecture   mean 60.17
Videos    mean 66.67
Practice  mean 71.67
Grand mean 66.17
SSB = 399.00, SSW = 131.50, SST = 530.50
MSB = 199.50, MSW = 8.77, F = 22.76, p = 2.86e-05
f_oneway: F = 22.76, p = 2.86e-05
Eta squared = 0.752

Check the split: 399.00 + 131.50 = 530.50, so SSB + SSW = SST. The between-group mean square (199.5) is about 23 times the within-group mean square (8.77), and p = 0.00003 is tiny, so you reject H0: the methods do not all give the same average mark. stats.f_oneway(*groups.values()) takes each group as a separate argument; the * unpacks the dictionary's arrays.

Eta squared (η² = SSB / SST) is the share of the total variation explained by the groups, an effect size for ANOVA. Here 75% of the variation in marks is explained by teaching method, a very large effect (real data is rarely this clean).

Which groups differ? Tukey's HSD

A significant ANOVA says that some means differ, not which ones. A post-hoc test compares every pair while keeping the overall false-alarm rate at α. The most common is Tukey's honestly significant difference (HSD) test.

import numpy as np
from scipy import stats

groups = {
    "Lecture":  np.array([62, 58, 65, 60, 55, 61]),
    "Videos":   np.array([66, 70, 64, 68, 63, 69]),
    "Practice": np.array([72, 68, 75, 70, 74, 71]),
}
res = stats.tukey_hsd(*groups.values())
print(res)

Pairwise Group Comparisons (95.0% Confidence Interval)
Comparison  Statistic  p-value  Lower CI  Upper CI
 (0 - 1)     -6.500     0.005   -10.940    -2.060
 (0 - 2)    -11.500     0.000   -15.940    -7.060
 (1 - 0)      6.500     0.005     2.060    10.940
 (1 - 2)     -5.000     0.027    -9.440    -0.560
 (2 - 0)     11.500     0.000     7.060    15.940
 (2 - 1)      5.000     0.027     0.560     9.440

The groups are numbered in the order you passed them: 0 = Lecture, 1 = Videos, 2 = Practice. Each pair appears twice with opposite signs. Every difference is significant: videos beat lectures by 6.5 marks (p = 0.005), practice beats lectures by 11.5 marks, and practice beats videos by 5 marks (p = 0.027). None of the confidence intervals include 0. The result also has res.pvalue and res.confidence_interval() if you need the numbers in code.

ANOVA on real-looking data, with assumption checks

One-way ANOVA assumes:

1. Independent observations.
2. Roughly normal values within each group (less important with larger groups).
3. Equal variances across groups (homogeneity of variance).

Levene's test (stats.levene) checks the third assumption: a small p-value suggests unequal variances. The Kruskal-Wallis test (stats.kruskal) is the rank-based alternative when normality or equal variances are doubtful. Here are maths marks across the five cities in students.csv:

import pandas as pd
from scipy import stats

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
students = students.dropna(subset=["maths"])
by_city = {city: g["maths"] for city, g in students.groupby("city")}
print(students.groupby("city")["maths"].agg(["count", "mean", "std"]).round(2))

print(f"Levene (equal variances?): p = {stats.levene(*by_city.values()).pvalue:.3f}")
res = stats.f_oneway(*by_city.values())
print(f"One-way ANOVA: F = {res.statistic:.3f}, p = {res.pvalue:.3f}")
kw = stats.kruskal(*by_city.values())
print(f"Kruskal-Wallis: H = {kw.statistic:.3f}, p = {kw.pvalue:.3f}")

         count   mean    std
city                        
Jaipur     115  56.70  17.41
Kochi       79  60.01  17.00
Lucknow     92  58.78  19.15
Pune       103  57.81  18.41
Rohtak      89  57.89  17.57
Levene (equal variances?): p = 0.831
One-way ANOVA: F = 0.444, p = 0.777
Kruskal-Wallis: H = 1.241, p = 0.871

Levene's p = 0.83 gives no reason to doubt equal variances, so ANOVA is appropriate. Its F of 0.44 is below 1 and p = 0.78: the city averages (56.7 to 60.0) differ by no more than chance would produce. Kruskal-Wallis agrees. When the overall test is not significant, you normally stop there and do not go hunting for pairs.

Common mistakes

• Running every pairwise t-test instead of ANOVA. The false-alarm rate grows with the number of pairs.
• Stopping at a significant F. It only says some means differ; use Tukey's HSD to find which.
• Searching for pairs after a non-significant F. That is a fishing trip.
• Ignoring very unequal variances or strong skew. Check with Levene's test and a box plot; use Kruskal-Wallis if needed.

Key points

• One-way ANOVA tests H0: all group means are equal, with a single test.
• Total variation splits into between-group and within-group parts: SST = SSB + SSW.
• F = MSB / MSW with df (k - 1, N - k); stats.f_oneway(*groups) computes it.
• Eta squared = SSB / SST measures effect size; stats.tukey_hsd(*groups) shows which pairs differ.
• Check assumptions with stats.levene, and use stats.kruskal as the rank-based alternative.

import pandas as pd
from scipy import stats

emp = pd.read_csv("employees.csv").drop_duplicates().dropna(subset=["salary"])
emp["department"] = emp["department"].str.strip().str.title().replace({"Hr": "HR"})
print(emp.groupby("department")["salary"].agg(["count", "mean"]).round(0))

groups = [g["salary"] for _, g in emp.groupby("department")]
res = stats.f_oneway(*groups)
print(f"ANOVA: F = {res.statistic:.2f}, p = {res.pvalue:.2e}")
print(f"Levene p = {stats.levene(*groups).pvalue:.2e}")
print(f"Welch ANOVA p = {stats.f_oneway(*groups, equal_var=False).pvalue:.2e}")
print(f"Kruskal-Wallis p = {stats.kruskal(*groups).pvalue:.2e}")

              count       mean
department                    
Data Science     38  1177921.0
Engineering      94  1070457.0
Finance          29   762690.0
HR               24   823292.0
Sales            59   783322.0
Support          51   544843.0
ANOVA: F = 42.90, p = 5.41e-33
Levene p = 2.09e-07
Welch ANOVA p = 7.96e-29
Kruskal-Wallis p = 3.24e-30