Range, IQR, Variance & Standard Deviation

Two batters both average 40 runs. One scores 35 to 45 every match; the other scores either a duck or a big hundred. Their averages are equal, but you would pick them for very different reasons. The average alone does not describe the data: you also need to know how spread out the values are.

In this lesson you will compute the range, the interquartile range (IQR), the variance and the standard deviation (SD) by hand and in Python. You will learn the difference between the population formula (divide by n) and the sample formula (divide by n - 1), see a simulation that shows why n - 1 is used, and meet the coefficient of variation.

Range and IQR

The range is the simplest measure of spread:

range = maximum - minimum

It is easy to compute but depends on only two values, so a single extreme value changes it completely.

The interquartile range uses the quartiles instead. Q1 has 25% of the data below it and Q3 has 75% below it, so:

IQR = Q3 - Q1      (the spread of the middle 50% of the data)

Because it ignores the top and bottom quarters, the IQR is not affected by a few extreme values. It pairs naturally with the median.

Variance and standard deviation by hand

The variance measures the average squared distance of the values from their mean. The standard deviation is its square root, which brings the units back to the units of the data (runs, rupees, marks).

deviation        = x - mean
population variance  σ² = Σ(x - μ)² / N
sample variance      s² = Σ(x - x̄)² / (n - 1)
standard deviation   = √variance

Why square the deviations? Because the deviations themselves always add up to zero (the values above the mean balance those below), so their plain average is useless. Squaring makes them all positive and gives extra weight to values far from the mean.

Here are both batters, step by step:

For batter A, the population variance is 58 / 5 = 11.6 and the sample variance is 58 / 4 = 14.5. The code does the same for both batters.

import math

a = [40, 42, 38, 45, 35]     # Batter A: steady
b = [0, 95, 5, 80, 20]       # Batter B: all or nothing

for name, runs in [("A", a), ("B", b)]:
    n = len(runs)
    mean = sum(runs) / n
    squares = [(x - mean) ** 2 for x in runs]
    pop_var = sum(squares) / n
    samp_var = sum(squares) / (n - 1)
    print(f"Batter {name}: mean {mean}, range {max(runs) - min(runs)}")
    print("  squared deviations:", squares)
    print(f"  population: variance {pop_var}, SD {math.sqrt(pop_var):.2f}")
    print(f"  sample:     variance {samp_var}, SD {math.sqrt(samp_var):.2f}")

Batter A: mean 40.0, range 10
  squared deviations: [0.0, 4.0, 4.0, 25.0, 25.0]
  population: variance 11.6, SD 3.41
  sample:     variance 14.5, SD 3.81
Batter B: mean 40.0, range 95
  squared deviations: [1600.0, 3025.0, 1225.0, 1600.0, 400.0]
  population: variance 1570.0, SD 39.62
  sample:     variance 1962.5, SD 44.30

Same mean, very different spread: batter A's SD is under 4 runs while batter B's is about 40 runs. A rough way to read an SD: a typical value is about one SD away from the mean.

Sample or population: n or n - 1?

Use the population formula (divide by N) when your data is the whole group you care about, for example all 5 innings if you only want to describe those 5 innings. Use the sample formula (divide by n - 1) when your data is a sample and you want to estimate the spread of the larger population, which is the usual situation.

The libraries do not agree on a default, so always check:

import statistics
import numpy as np
import pandas as pd

b = [0, 95, 5, 80, 20]
print("statistics.pvariance:", statistics.pvariance(b), " pstdev:", round(statistics.pstdev(b), 2))
print("statistics.variance: ", statistics.variance(b), " stdev:", round(statistics.stdev(b), 2))
print("np.var (default):    ", np.var(b), " np.std:", round(np.std(b), 2))
print("np.var(ddof=1):      ", np.var(b, ddof=1))
print("pandas .var():       ", pd.Series(b).var())

statistics.pvariance: 1570  pstdev: 39.62
statistics.variance:  1962.5  stdev: 44.3
np.var (default):     1570.0  np.std: 39.62
np.var(ddof=1):       1962.5
pandas .var():        1962.5

The statistics module has separate functions: pvariance/pstdev for a population and variance/stdev for a sample. NumPy uses the population formula by default; pass ddof=1 ("delta degrees of freedom") to divide by n - 1. pandas uses ddof=1 by default. SciPy functions such as stats.sem() also default to ddof=1, as you will see later.

Why n - 1? A simulation

The sample variance measures distances from the sample mean x̄. But x̄ is calculated from the same values, so it sits closer to them than the true mean μ does. As a result, squared distances from x̄ are a little too small on average, and dividing by n underestimates the true variance. Dividing by n - 1 (called Bessel's correction) exactly fixes this on average.

You can check it: draw many small samples from a population whose variance you know, and average the two versions.

import numpy as np

rng = np.random.default_rng(1)
# 100,000 samples of size 5 from a population whose variance is exactly 100 (SD 10)
samples = rng.normal(50, 10, size=(100_000, 5))

var_n = samples.var(axis=1, ddof=0)
var_n1 = samples.var(axis=1, ddof=1)
print("True variance:              100")
print(f"Average, divisor n:         {var_n.mean():.1f}")
print(f"Average, divisor n - 1:     {var_n1.mean():.1f}")

True variance:              100
Average, divisor n:         79.8
Average, divisor n - 1:     99.8

Dividing by n gives about 80 on average, which is 100 x 4/5: too small by a factor of (n - 1)/n. Dividing by n - 1 gives about 100, the right answer on average. With large samples the difference becomes tiny, but with small samples it matters. The number n - 1 is called the degrees of freedom: once you know the mean, only n - 1 of the deviations are free to vary, because they must add up to zero.

Quartiles can be calculated in different ways

There is more than one accepted rule for quartiles when they fall between two values, and different tools use different rules.

import statistics
import numpy as np
import pandas as pd
from scipy import stats

marks = [35, 41, 48, 52, 55, 58, 63, 67, 72, 90]

q1, q3 = np.percentile(marks, [25, 75])
print("NumPy (linear):  Q1", q1, " Q3", q3, " IQR", q3 - q1)
print("pandas:         ", pd.Series(marks).quantile([0.25, 0.75]).tolist())
print("stats.iqr:      ", stats.iqr(marks))
print("statistics.quantiles:", statistics.quantiles(marks, n=4))
print("  inclusive:         ", statistics.quantiles(marks, n=4, method="inclusive"))
print("NumPy 'weibull':     ", np.percentile(marks, [25, 75], method="weibull"))

NumPy (linear):  Q1 49.0  Q3 66.0  IQR 17.0
pandas:          [49.0, 66.0]
stats.iqr:       17.0
statistics.quantiles: [46.25, 56.5, 68.25]
  inclusive:          [49.0, 56.5, 66.0]
NumPy 'weibull':      [46.25 68.25]

NumPy, pandas and SciPy use the "linear" method by default: Q1 is at position 0.25 x (n - 1) = 2.25 in the sorted list (counting from 0), a quarter of the way from 48 to 52, which gives 49. Python's statistics.quantiles() uses the "exclusive" method by default, which uses position 0.25 x (n + 1) and gives 46.25. NumPy calls that method "weibull". Neither is wrong; just use one method consistently and say which. With large datasets the differences shrink.

Comparing spread across different scales

A standard deviation of ₹300 is large for a chai stall but tiny for a jewellery shop. The coefficient of variation (CV) compares spread relative to the mean:

CV = SD / mean

The median absolute deviation (MAD), the median of the distances from the median, is a spread measure that resists outliers, just like the median itself.

import numpy as np
from scipy import stats

chai = np.array([2400, 2650, 2200, 2900, 2500, 2750, 3100])                     # ₹ per day
jeweller = np.array([85_000, 40_000, 1_20_000, 60_000, 95_000, 30_000, 2_10_000])

for name, x in [("Chai stall", chai), ("Jeweller", jeweller)]:
    sd = x.std(ddof=1)
    mad = stats.median_abs_deviation(x)
    print(f"{name:<10}  mean ₹{x.mean():>8,.0f}  SD ₹{sd:>7,.0f}  CV {sd / x.mean():.2f}  MAD ₹{mad:,.0f}")

Chai stall  mean ₹   2,643  SD ₹    306  CV 0.12  MAD ₹250
Jeweller    mean ₹  91,429  SD ₹ 61,013  CV 0.67  MAD ₹35,000

The chai stall's daily sales vary by about 12% of its mean; the jeweller's vary by 67%. The CV only makes sense for ratio data with a true zero (rupees, kilograms), not for temperatures in Celsius.

Common mistakes

• Mixing ddof settings. np.std(x) and pd.Series(x).std() give different answers for the same data. Decide which you need.
• Reporting variance in a report. Variance is in squared units (runs²); readers understand the SD better.
• Using the range for data with outliers. Use the IQR or the MAD.
• Comparing SDs of things measured on different scales. Use the CV.

Key points

• Range = max - min; IQR = Q3 - Q1 and ignores the extreme quarters.
• Variance is the average squared deviation from the mean; SD is its square root, in the data's units.
• Population formulas divide by N (ddof=0, NumPy's default); sample formulas divide by n - 1 (ddof=1, pandas' default).
• Dividing by n - 1 corrects the underestimate that comes from measuring around the sample mean.
• Quartile methods differ between tools; CV = SD / mean compares spread across scales.

import math
import numpy as np
from scipy import stats

bowlers = {"Rohan": [6, 8, 7, 9, 5, 8, 7, 6], "Farhan": [2, 14, 3, 12, 1, 15, 4, 5]}
for name, runs in bowlers.items():
    n = len(runs)
    mean = sum(runs) / n
    sd = math.sqrt(sum((x - mean) ** 2 for x in runs) / (n - 1))
    print(f"{name}: mean {mean}, SD by hand {sd:.3f}, NumPy {np.std(runs, ddof=1):.3f}, IQR {stats.iqr(runs)}")

Rohan: mean 7.0, SD by hand 1.309, NumPy 1.309, IQR 2.0
Farhan: mean 7.0, SD by hand 5.707, NumPy 5.707, IQR 9.75