Sampling & the Central Limit Theorem

You rarely get to measure everyone. A survey asks 1,000 voters, not the whole country; a quality check opens 50 boxes, not the whole truck. A sample is a smaller set of data taken from a larger population, and statistics is largely about what a sample can tell you about the population, and how sure you can be.

In this lesson you will draw samples with pandas, see how a badly chosen sample misleads, watch the central limit theorem appear in a simulation, and measure the uncertainty of an estimate with the standard error and a bootstrap confidence interval.

Population, sample and statistic

• The population is everything you want to learn about: all orders of the year, all Class 10 students in a state.
• A sample is the part you actually observe.
• A parameter is a true number about the population, such as the true average order value. A statistic is the same number calculated from a sample, used as an estimate of the parameter.

In this lesson the 1,200 orders in sales.csv play the role of the population. Because you can see all of them, you can check how close each sample gets to the truth.

Random samples with pandas

df.sample() draws random rows. Use n for a number of rows or frac for a fraction, and random_state to make the sample repeatable.

import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()
sales = sales[sales["quantity"].between(1, 20)]
sales["value"] = sales["quantity"] * sales["unit_price"] * (1 - sales["discount"] / 100)

print("Population mean:", round(sales["value"].mean(), 1))
for seed in [1, 2, 3]:
    s = sales.sample(n=50, random_state=seed)
    print(f"Sample {seed} (n=50) mean:", round(s["value"].mean(), 1))

big = sales.sample(frac=0.5, random_state=1)
print("Half the data, mean:", round(big["value"].mean(), 1))

Population mean: 1537.9
Sample 1 (n=50) mean: 1827.1
Sample 2 (n=50) mean: 1554.3
Sample 3 (n=50) mean: 1160.2
Half the data, mean: 1610.9

Each sample of 50 gives a different estimate, sometimes well above and sometimes well below the true value. This sampling variation is unavoidable. The larger sample, half the data, lands much closer. Two lessons follow: a single sample is never exactly right, and bigger samples are usually closer.

Stratified samples and sampling bias

A stratified sample takes a fixed share from each group (stratum), so every group is represented in the right proportion. With pandas, sample within groups using groupby().sample().

import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()

print(sales["category"].value_counts(normalize=True).round(3))

strat = sales.groupby("category").sample(frac=0.1, random_state=4)
print(strat["category"].value_counts(normalize=True).round(3))

first_100 = sales.head(100)
print("Share of October orders in first 100 rows:", (first_100["order_date"].str[5:7] == "10").mean())
print("Share of October orders overall:", round((sales["order_date"].str[5:7] == "10").mean(), 3))

category
Electronics       0.249
Fashion           0.225
Home & Kitchen    0.178
Groceries         0.155
Sports            0.101
Books             0.093
Name: proportion, dtype: float64
category
Electronics       0.248
Fashion           0.223
Home & Kitchen    0.182
Groceries         0.157
Sports            0.099
Books             0.091
Name: proportion, dtype: float64
Share of October orders in first 100 rows: 0.0
Share of October orders overall: 0.14

The stratified sample keeps each category's share almost exactly. The last two lines show sampling bias: taking the first 100 rows is convenient, but the file is sorted by date, so those rows are all from early in the year and contain no October orders at all. A biased sample does not get better when it gets bigger. Common real-world examples are online polls (only people who see them reply) and asking only customers who visit on weekdays.

The central limit theorem

Order values are right-skewed: most orders are small, a few are large. Yet if you take many samples and calculate each sample's mean, those means form a bell curve. This is the central limit theorem (CLT): for large enough samples, the distribution of sample means is close to normal, whatever the shape of the original data, and it is centred on the true mean.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

sales = pd.read_csv("sales.csv").drop_duplicates()
sales = sales[sales["quantity"].between(1, 20)]
values = (sales["quantity"] * sales["unit_price"] * (1 - sales["discount"] / 100)).to_numpy()

rng = np.random.default_rng(0)
means = np.array([rng.choice(values, size=40).mean() for _ in range(5000)])

fig, axes = plt.subplots(1, 2, figsize=(10, 3.5), layout="constrained")
axes[0].hist(values, bins=40, color="#94a3b8")
axes[0].set_title("Order values (skewed)")
axes[1].hist(means, bins=40, color="#c2410c")
axes[1].set_title("Means of 5,000 samples of 40")
plt.show()

print("Population mean:", round(values.mean(), 1), " mean of sample means:", round(means.mean(), 1))
print("Population SD:", round(values.std(), 1), " SD of sample means:", round(means.std(), 1))
print("Predicted SD of means (SD / sqrt(40)):", round(values.std() / np.sqrt(40), 1))

Population mean: 1537.9  mean of sample means: 1538.3
Population SD: 1419.8  SD of sample means: 222.6
Predicted SD of means (SD / sqrt(40)): 224.5

The left histogram is lopsided: a tall pile of small orders and a long tail to the right. The right histogram, made of 5,000 sample means, is a neat, nearly symmetric bell centred on the true mean. Its spread is much smaller, and it matches the prediction: the SD of sample means equals the population SD divided by the square root of the sample size.

Standard error: how uncertain is an estimate?

That spread of sample means has a name: the standard error (SE). In practice you have only one sample, so you estimate it as the sample SD divided by √n. A rough 95% confidence interval for the mean is then the sample mean ± 1.96 × SE.

import numpy as np
import pandas as pd
from scipy import stats

students = pd.read_csv("students.csv").drop_duplicates()
sample = students["english"].sample(n=60, random_state=11)

mean = sample.mean()
se = sample.std() / np.sqrt(len(sample))
print(f"Sample mean {mean:.1f}, standard error {se:.2f}")
print(f"Rough 95% interval: {mean - 1.96 * se:.1f} to {mean + 1.96 * se:.1f}")

low, high = stats.t.interval(0.95, df=len(sample) - 1, loc=mean, scale=stats.sem(sample))
print(f"t-based 95% interval: {low:.1f} to {high:.1f}")
print("True mean of all students:", round(students["english"].mean(), 1))

Sample mean 62.3, standard error 1.99
Rough 95% interval: 58.4 to 66.2
t-based 95% interval: 58.4 to 66.3
True mean of all students: 62.1

From 60 students, the interval runs a few marks either side of the sample mean, and the true mean of all 480 students falls inside it. "95% confidence" describes the method: if you repeated the sampling many times, about 95% of the intervals built this way would contain the true mean. stats.sem() computes the standard error, and the t distribution gives a slightly wider, more accurate interval for small samples.

The bootstrap

What about the uncertainty of a median, or of something with no simple formula? The bootstrap answers it by resampling: draw many new samples from your one sample, with replacement, compute the statistic each time, and use the middle 95% of the results as the interval.

import numpy as np
import pandas as pd

emp = pd.read_csv("employees.csv").drop_duplicates().dropna(subset=["salary"])
salaries = emp["salary"].to_numpy()

rng = np.random.default_rng(5)
boot = np.array([np.median(rng.choice(salaries, size=len(salaries), replace=True)) for _ in range(5000)])

low, high = np.percentile(boot, [2.5, 97.5])
print(f"Median salary: ₹{np.median(salaries):,.0f}")
print(f"Bootstrap 95% interval: ₹{low:,.0f} to ₹{high:,.0f}")

Median salary: ₹819,000
Bootstrap 95% interval: ₹785,000 to ₹856,000

"With replacement" means the same employee can be drawn more than once in a resample, which mimics drawing fresh samples from the population. The bootstrap needs no formula, works for almost any statistic, and is easy with NumPy. SciPy also offers scipy.stats.bootstrap() for the same job.

Common mistakes

• Taking the first rows as a sample. Files are often sorted; use sample() with a random_state.
• Believing a bigger biased sample fixes bias. It only makes the wrong answer more precise.
• Reporting an estimate without its uncertainty. Give a standard error or confidence interval.
• Misreading "95% confidence". It does not mean a 95% chance that this particular interval contains the truth; it describes how often the method succeeds.

Key points

• A sample statistic estimates a population parameter and varies from sample to sample.
• df.sample(n=..., random_state=...) draws random rows; groupby().sample() draws stratified samples.
• Biased sampling, such as taking the first rows of a sorted file, gives wrong answers at any size.
• The central limit theorem: means of large enough samples are roughly normal, with spread SD / √n.
• The standard error and confidence intervals measure uncertainty; the bootstrap works for any statistic.

import numpy as np
import pandas as pd

weather = pd.read_csv("weather.csv")
delhi = weather.loc[(weather["city"] == "Delhi") & (weather["temp_max"] < 60), "temp_max"]
sample = delhi.sample(n=40, random_state=3)

mean = sample.mean()
se = sample.std() / np.sqrt(len(sample))
low, high = mean - 1.96 * se, mean + 1.96 * se
print(f"Sample mean {mean:.2f}, SE {se:.2f}, interval {low:.2f} to {high:.2f}")
print("True mean", round(delhi.mean(), 2), "inside:", low <= delhi.mean() <= high)

Sample mean 31.37, SE 1.01, interval 29.39 to 33.35
True mean 30.92 inside: True