Descriptive Statistics

Descriptive statistics are numbers that summarise a dataset: where its centre is, how spread out it is, and what shape it has. They are the first thing anyone asks for ("What is the average salary?") and the easiest thing to get subtly wrong ("Average" in which sense?).

In this lesson you will compute measures of centre, spread and shape with pandas, NumPy and SciPy, see when the mean misleads, and learn the one setting (ddof) that makes pandas and NumPy disagree about the standard deviation.

Measures of centre: mean, median and mode

• The mean is the sum divided by the count, the everyday "average".
• The median is the middle value when the data is sorted. Half the values are below it, half above.
• The mode is the most frequent value. It is the only one of the three that works for categories.

import pandas as pd

emp = pd.read_csv("employees.csv").drop_duplicates()
salary = emp["salary"]

print("Mean:  ", round(salary.mean()))
print("Median:", salary.median())
print("Mode of city:", emp["city"].mode().tolist())
print("Mode of rating:", emp["rating"].mode().tolist())

top10 = salary.nlargest(10).sum() / salary.sum() * 100
print(f"Top 10 earners take {top10:.1f}% of the total salary bill")

Mean:   885641
Median: 819000.0
Mode of city: ['Hyderabad']
Mode of rating: [3]
Top 10 earners take 6.9% of the total salary bill

The mean salary is clearly higher than the median, because a group of senior people with very high salaries pulls the mean up while the median stays with the typical employee. This happens with incomes, house prices, order values and rainfall: data with a long tail of large values. In such cases, report the median, or both. mode() returns a Series because there can be ties.

Measures of spread

Two classes can have the same average mark and still be very different: one where everyone scores 60, one where half score 30 and half score 90. Measures of spread capture that difference.

import pandas as pd

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
students = students[students["science"] <= 100]

for subject in ["maths", "hindi"]:
    s = students[subject]
    q1, q3 = s.quantile([0.25, 0.75])
    print(f"{subject:<6} mean {s.mean():.1f}  sd {s.std():.1f}  range {s.max() - s.min():.0f}  IQR {q3 - q1:.0f}")

print(students["maths"].quantile([0.1, 0.5, 0.9]))
print("Coefficient of variation, maths:", round(students["maths"].std() / students["maths"].mean(), 2))

maths  mean 58.2  sd 17.8  range 82  IQR 26
hindi  mean 66.1  sd 13.4  range 73  IQR 18
0.1    35.0
0.5    58.0
0.9    79.0
Name: maths, dtype: float64
Coefficient of variation, maths: 0.31

Maths has a lower mean and a larger standard deviation than Hindi: marks are more spread out, so the class is more divided in maths. The 10th and 90th percentiles say that the weakest 10% scored at or below the first value and the strongest 10% at or above the last. The coefficient of variation (SD divided by mean) compares spread across measures with different averages.

The ddof trap: pandas vs NumPy

Here is a surprise that confuses many people. pandas and NumPy give different standard deviations for the same numbers.

import numpy as np
import pandas as pd

marks = [62, 75, 58, 91, 70]

print("pandas std:", round(pd.Series(marks).std(), 3))
print("NumPy std: ", round(np.std(marks), 3))
print("NumPy with ddof=1:", round(np.std(marks, ddof=1), 3))
print("pandas with ddof=0:", round(pd.Series(marks).std(ddof=0), 3))

pandas std: 12.911
NumPy std:  11.548
NumPy with ddof=1: 12.911
pandas with ddof=0: 11.548

The difference is the divisor. NumPy divides the squared distances by n (ddof=0), which describes these exact numbers as a complete population. pandas divides by n - 1 (ddof=1), the sample standard deviation, which gives a better estimate of the spread of a larger population when your data is only a sample of it. With large datasets the two are almost equal; with five values they differ noticeably. Pick one deliberately and say which you used.

Shape: skewness and kurtosis

Skewness measures whether a distribution has a longer tail on one side. Positive skew means a long tail to the right (a few very large values); negative skew means a long tail to the left; near zero means roughly symmetric. Kurtosis measures how heavy the tails are compared with a bell curve; pandas reports excess kurtosis, where a bell curve scores 0.

import pandas as pd
from scipy import stats

weather = pd.read_csv("weather.csv")
emp = pd.read_csv("employees.csv").drop_duplicates()
students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()

rain = weather.loc[weather["rainfall_mm"] > 0, "rainfall_mm"]
print("Skew of rainfall on rainy days:", round(rain.skew(), 2))
print("Skew of salary:", round(emp["salary"].skew(), 2))
print("Skew of english marks:", round(students["english"].skew(), 2))

summary = stats.describe(students["english"])
print(summary.nobs, round(summary.mean, 2), round(summary.variance, 2), round(summary.skewness, 3))

Skew of rainfall on rainy days: 2.33
Skew of salary: 0.92
Skew of english marks: -0.15
480 62.11 202.37 -0.154

Rainfall is strongly right-skewed: most rainy days bring a little rain and a few bring a lot. Salaries are also right-skewed. English marks are close to symmetric, so their mean and median are close, and the mean is a fair summary. SciPy's stats.describe() returns several statistics at once; note that its variance uses ddof=1, like pandas, while its skewness uses a slightly different formula from pandas, so the last digits can differ.

Summaries by group

The most useful descriptive statistics are often comparisons between groups. groupby() combined with describe() or agg() produces them in one step.

import pandas as pd

weather = pd.read_csv("weather.csv")
weather = weather[weather["temp_max"] < 60]

table = weather.groupby("city")["temp_max"].describe()
print(table[["mean", "std", "min", "50%", "max"]].round(1))

sales = pd.read_csv("sales.csv").drop_duplicates()
sales = sales[sales["quantity"].between(1, 20)]
sales["value"] = sales["quantity"] * sales["unit_price"]
avg_price = (sales["unit_price"] * sales["quantity"]).sum() / sales["quantity"].sum()
print("Average price per item sold:", round(avg_price, 2))
print("Average of product prices:  ", round(sales["unit_price"].mean(), 2))

         mean  std   min   50%   max
city                                
Chennai  32.6  3.3  24.2  32.8  39.9
Delhi    30.9  6.3  16.3  32.2  42.5
Jaipur   32.2  5.7  19.3  32.7  43.0
Kochi    30.1  2.3  25.5  30.2  36.8
Mumbai   31.5  2.4  24.6  31.9  37.1
Average price per item sold: 1079.86
Average of product prices:   1120.33

The temperature table shows that Jaipur and Delhi have the largest standard deviations, big swings between winter and summer, while Kochi and Mumbai have small ones. The last two lines show a weighted mean: the average price per item sold weights each price by its quantity, and it differs from the plain average over orders because cheaper grocery items are often bought several at a time. Always ask "average of what?".

Common mistakes

• Reporting only the mean for skewed data. For salaries, prices and rainfall, the median is more typical.
• Ignoring spread. Two groups with equal means can be very different; report the SD or IQR too.
• Mixing ddof conventions. NumPy defaults to ddof=0, pandas to ddof=1.
• Summarising before cleaning. Outliers such as a 99.9 °C sensor error distort the mean, SD and skewness.

Key points

• Mean, median and mode measure the centre; the median resists outliers, the mode works for categories.
• Range, variance, standard deviation, percentiles and IQR measure spread.
• pandas std() uses ddof=1 (sample); NumPy np.std() uses ddof=0 (population).
• Skewness shows which side has the long tail; right-skewed data has a mean above the median.
• Use groupby().describe() to compare groups, and weighted means when rows count unequally.

import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()
sales = sales[sales["quantity"].between(1, 20)]
value = sales["quantity"] * sales["unit_price"] * (1 - sales["discount"] / 100)

print("Mean:  ", round(value.mean(), 2))
print("Median:", round(value.median(), 2))
print("SD:    ", round(value.std(), 2))
print("P90:   ", round(value.quantile(0.9), 2))
print("Skew:  ", round(value.skew(), 2))
# The median: order values are right-skewed, so a few big orders pull the mean up.

Mean:   1537.92
Median: 1099.0
SD:     1420.42
P90:    2999.0
Skew:   2.57