Sampling Methods & Bias

A sample is only useful if it looks like the population it came from. How you choose the sample matters far more than how big it is. A carefully chosen sample of 1,000 people can describe a whole country well, while a badly chosen sample of a million can be completely wrong.

In this lesson you will learn the main probability sampling methods and code each one in Python, meet the common non-probability shortcuts, and see with simulations how bias creeps in and why adding more people does not remove it.

Probability and non-probability sampling

In probability sampling, every member of the population has a known, non-zero chance of being chosen, and chance (not a person) makes the choice. This is what allows the formulas of inferential statistics to work.

In non-probability sampling, someone's judgement or convenience decides who is in the sample. It is cheaper and faster, but you cannot measure how far its results are from the truth.

To pick a probability sample you usually need a sampling frame: a list of the population, such as a school's roll register, a voter list or a customer database. If the frame misses people, so will your sample.

Simple random sampling

In a simple random sample (SRS), every possible group of n members is equally likely to be chosen. It is like writing every name on a slip, mixing the slips in a box and drawing n of them.

import numpy as np

rng = np.random.default_rng(5)
roll_numbers = np.arange(1, 121)          # 120 students in Class 12
chosen = rng.choice(roll_numbers, size=10, replace=False)
print(np.sort(chosen))

[  3   7  34  54  60  74  75  91  93 117]

replace=False makes sure no roll number is picked twice. Notice that 74 and 75, and 91 and 93, were both chosen while there is a big gap from 7 to 34. That is normal for random choice; random does not mean evenly spread.

Systematic sampling

Systematic sampling picks every k-th member from the list after a random start. The interval is k = N / n. It is quick to do by hand, for example checking every 12th file in a cabinet.

import numpy as np

rng = np.random.default_rng(9)
N, n = 120, 10
k = N // n                           # the sampling interval
start = int(rng.integers(1, k + 1))  # random start from 1 to k
chosen = np.arange(start, N + 1, k)
print("k =", k, " start =", start)
print(chosen)

k = 12  start = 6
[  6  18  30  42  54  66  78  90 102 114]

rng.integers(1, k + 1) returns a whole number from 1 to 12 (the upper limit is excluded). Systematic sampling is fine when the list is in a random or alphabetical order. It goes wrong if the list has a repeating pattern that matches k: if a housing society lists 12 flats per building with the corner flat always first, every 12th flat might be a corner flat.

Stratified sampling

In stratified sampling you split the population into groups called strata (such as cities or school types) and take a random sample from each group, usually in proportion to its size. This guarantees that every group is represented, and it often gives more precise estimates than a simple random sample of the same size.

With pandas, groupby(...).sample() does it in one line.

import pandas as pd

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
strat = students.groupby("city").sample(frac=0.1, random_state=4)
simple = students.sample(n=len(strat), random_state=4)

table = pd.DataFrame({
    "all": students["city"].value_counts(normalize=True),
    "stratified": strat["city"].value_counts(normalize=True),
    "simple": simple["city"].value_counts(normalize=True),
}).round(3)
print("Sample size:", len(strat))
print(table)

Sample size: 48
           all  stratified  simple
city                              
Jaipur   0.240       0.250   0.250
Kochi    0.165       0.167   0.083
Lucknow  0.192       0.188   0.271
Pune     0.215       0.208   0.188
Rohtak   0.190       0.188   0.208

Both samples have 48 students. The stratified sample matches the city shares of the file almost exactly, because 10% was taken from each city. The simple random sample, by chance, has only half the right share of Kochi students and too many from Lucknow. If Kochi students differ in some way, the simple sample would give a less accurate picture.

Cluster and multistage sampling

In cluster sampling you randomly choose whole groups (clusters), such as schools, villages or housing societies, and then measure everyone inside the chosen clusters. It is cheaper because surveyors travel to only a few places.

import numpy as np

rng = np.random.default_rng(3)
schools = [f"School {c}" for c in "ABCDEFGHIJKL"]
picked = rng.choice(schools, size=3, replace=False)
print("Survey every student in:", picked.tolist())

Survey every student in: ['School A', 'School C', 'School I']

The difference from stratified sampling: strata are all sampled from, while only some clusters are chosen. Students in the same school tend to be similar, so a cluster sample is usually less precise than a simple random sample of the same number of students. Large national surveys often combine methods in multistage sampling: choose districts, then villages or wards inside them, then households inside those.

Non-probability sampling

These methods can be useful for early exploration or for hard-to-reach groups, but do not use their results to make precise claims about a population.

Bias: when the method itself is wrong

Bias is a systematic error: the method tends to miss the truth in the same direction every time. The main kinds are:

• Selection bias: the way people are chosen favours some groups (a phone survey misses people without phones).
• Undercoverage: part of the population is not in the sampling frame at all.
• Non-response bias: chosen people who do not reply differ from those who do.
• Voluntary response bias: people decide for themselves whether to take part.
• Response bias: the question or situation pushes answers one way ("You do agree that our service is excellent?").

This simulation shows voluntary response bias in reviews. Unhappy customers are made six times as likely to write a review as others.

import numpy as np

rng = np.random.default_rng(21)
# Invented: satisfaction score (1-10) of 20,000 customers of a food-delivery app
scores = rng.integers(1, 11, size=20_000)
# Unhappy customers are far more likely to bother writing a review
p_review = np.where(scores <= 3, 0.30, 0.05)
reviewed = scores[rng.random(scores.size) < p_review]

print(f"True average:   {scores.mean():.2f} ({scores.size} customers)")
print(f"Review average: {reviewed.mean():.2f} ({reviewed.size} reviews)")

True average:   5.52 (20000 customers)
Review average: 3.37 (2488 reviews)

Nearly 2,500 reviews sounds like a lot of evidence, yet their average is more than two points below the truth.

A bigger sample does not fix bias

Here a survey list was taken from a parking-permit database, so car owners are three times as likely to be picked as they should be. Watch what happens as the sample grows.

import numpy as np

rng = np.random.default_rng(8)
# Invented town: 30% of households own a car, but the survey list has
# three times as many car owners as it should (it came from a parking-permit database)
own_car = rng.random(50_000) < 0.30
weight = np.where(own_car, 3.0, 1.0)
prob = weight / weight.sum()

print("True share:", round(own_car.mean(), 3))
for n in [100, 1_000, 10_000]:
    idx = rng.choice(own_car.size, size=n, replace=False, p=prob)
    print(f"n = {n:>6}: estimated share {own_car[idx].mean():.3f}")

True share: 0.299
n =    100: estimated share 0.530
n =   1000: estimated share 0.538
n =  10000: estimated share 0.536

The estimate settles down as n grows, but it settles on the wrong answer: about 54% instead of 30%. More data made the result more confident, not more correct. A famous real case is the 1936 US presidential poll by the Literary Digest magazine: it collected over two million replies and still predicted the wrong winner. Its mailing lists leaned on telephone directories and car registrations, which at that time over-represented richer households, and only the people who chose to reply were counted.

Common mistakes

• Calling a convenience sample "random". Asking people who walk past your college gate is not random sampling.
• Believing size beats method. A big biased sample gives a precise wrong answer.
• Ignoring non-response. If 70% of the people you chose did not reply, your sample is no longer the one you planned.
• Forgetting the sampling frame. An online form cannot reach people who are not online.

Key points

• Probability sampling lets chance choose, so you can measure the uncertainty of your results.
• Simple random: every group of n equally likely. Systematic: every k-th after a random start.
• Stratified: sample within every group; cluster: sample whole groups and measure everyone in them.
• Convenience, voluntary response, quota and snowball samples are not suitable for precise population claims.
• Bias is a systematic error; a larger sample does not reduce it.

import pandas as pd

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
strat = students.groupby("tuition").sample(frac=0.1, random_state=7)
print("Share with tuition, all:   ", round((students["tuition"] == "Yes").mean(), 3))
print("Share with tuition, sample:", round((strat["tuition"] == "Yes").mean(), 3))

print("Mean maths, all students:  ", round(students["maths"].mean(), 2))
print("Mean maths, tuition only:  ", round(students.loc[students["tuition"] == "Yes", "maths"].mean(), 2))
# Surveying only tuition students is selection bias (undercoverage of everyone else).

Share with tuition, all:    0.429
Share with tuition, sample: 0.438
Mean maths, all students:   58.11
Mean maths, tuition only:   58.79