Correlation

Do students who study more score higher? Do hotter days have lower humidity? Do bigger discounts sell more items? Each question asks whether two numeric variables move together. Correlation puts a single number on that relationship.

In this lesson you will calculate correlations with pandas and SciPy, read a correlation matrix, choose between Pearson and Spearman correlation, and learn the limits of the number, including the famous rule that correlation does not prove causation.

What the correlation coefficient means

The correlation coefficient r is a number between -1 and +1:

The sign gives the direction and the size gives the strength. Rough labels such as "weak" (below 0.3), "moderate" and "strong" (above 0.7) are common, but what counts as strong depends on the field. In human behaviour, 0.4 is already a meaningful relationship.

import pandas as pd

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
students = students[(students["attendance"] <= 100) & (students["science"] <= 100)]

print(round(students["study_hours"].corr(students["maths"]), 3))
print(round(students["attendance"].corr(students["maths"]), 3))
print(round(students["maths"].corr(students["science"]), 3))

0.493
0.388
0.84

Series.corr() gives the Pearson correlation by default. Study hours and attendance both have a moderate positive relationship with maths marks. Maths and science marks are strongly related, which makes sense: a student who does well in one usually does well in the other.

The correlation matrix

DataFrame.corr() correlates every numeric column with every other one, giving a correlation matrix. The diagonal is always 1, because each column correlates perfectly with itself, and the matrix is symmetric.

import pandas as pd
import matplotlib.pyplot as plt

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
students = students[(students["attendance"] <= 100) & (students["science"] <= 100)]
cols = ["attendance", "study_hours", "maths", "science", "english", "hindi"]

matrix = students[cols].corr()
print(matrix.round(2))

fig, ax = plt.subplots(figsize=(6, 5))
im = ax.imshow(matrix, cmap="RdBu_r", vmin=-1, vmax=1)
ax.set_xticks(range(len(cols)), cols, rotation=45, ha="right")
ax.set_yticks(range(len(cols)), cols)
for i in range(len(cols)):
    for j in range(len(cols)):
        ax.text(j, i, f"{matrix.iloc[i, j]:.2f}", ha="center", va="center", fontsize=8)
fig.colorbar(im)
ax.set_title("Correlation matrix")
plt.show()

             attendance  study_hours  maths  science  english  hindi
attendance         1.00         0.06   0.39     0.37     0.38   0.37
study_hours        0.06         1.00   0.49     0.50     0.48   0.47
maths              0.39         0.49   1.00     0.84     0.80   0.78
science            0.37         0.50   0.84     1.00     0.80   0.78
english            0.38         0.48   0.80     0.80     1.00   0.72
hindi              0.37         0.47   0.78     0.78     0.72   1.00

The heat map colours each cell from blue (negative) through white (zero) to red (positive), with the value written in each cell. All the mark columns are warm red with each other; attendance and study hours are paler, a moderate relationship with every subject; and attendance and study hours are only weakly related to each other. A diverging colour map with a white middle (RdBu_r) and fixed limits of -1 and +1 keeps zero neutral, so the colours are honest. If a DataFrame has text columns, pass numeric_only=True to corr().

Pearson vs Spearman

Pearson correlation measures how close the points are to a straight line, and outliers affect it strongly. Spearman correlation first replaces values by their ranks (1st, 2nd, 3rd...) and then correlates the ranks. It measures whether the relationship goes consistently in one direction, even if it curves, and it resists outliers.

import numpy as np
import pandas as pd

rng = np.random.default_rng(3)
x = np.arange(1, 31)
curve = pd.DataFrame({"x": x, "y": np.exp(x / 5) + rng.normal(0, 5, 30)})
print("Curved data   pearson:", round(curve.corr().iloc[0, 1], 3),
      " spearman:", round(curve.corr(method="spearman").iloc[0, 1], 3))

weather = pd.read_csv("weather.csv")
kochi = weather[weather["city"] == "Kochi"][["temp_max", "humidity"]]
print("With 99.9 error pearson:", round(kochi.corr().iloc[0, 1], 3),
      " spearman:", round(kochi.corr(method="spearman").iloc[0, 1], 3))
clean = kochi[kochi["temp_max"] < 60]
print("Cleaned       pearson:", round(clean.corr().iloc[0, 1], 3),
      " spearman:", round(clean.corr(method="spearman").iloc[0, 1], 3))

Curved data   pearson: 0.823  spearman: 0.951
With 99.9 error pearson: -0.311  spearman: -0.642
Cleaned       pearson: -0.611  spearman: -0.646

For the curved data, y always rises with x, so Spearman is almost 1, while Pearson is lower because the points do not follow a straight line. For Kochi, a single 99.9 °C sensor error noticeably weakens the Pearson correlation between temperature and humidity, while Spearman barely moves. After cleaning, both show a clear negative relationship: on Kochi's hotter days the air is drier, and on cooler, rainy monsoon days it is more humid. A third option, method="kendall", is another rank-based measure.

Is the correlation real? p-values with SciPy

With only a few data points, a correlation can appear by pure chance. scipy.stats.pearsonr() and spearmanr() return the coefficient and a p-value: roughly, how surprising a correlation this large would be if there were really no relationship at all. A small p-value (for example below 0.05) means chance alone is an unlikely explanation. You will learn exactly how to read p-values in Hypothesis Testing Basics.

import pandas as pd
from scipy import stats

emp = pd.read_csv("employees.csv").drop_duplicates().dropna(subset=["salary"])

res = stats.pearsonr(emp["experience_years"], emp["salary"])
print(f"experience vs salary: r = {res.statistic:.3f}, p = {res.pvalue:.2g}")

res = stats.spearmanr(emp["rating"], emp["salary"])
print(f"rating vs salary:     rho = {res.statistic:.3f}, p = {res.pvalue:.3f}")

few = emp.head(8)
res = stats.pearsonr(few["rating"], few["age"])
print(f"8 rows, rating vs age: r = {res.statistic:.3f}, p = {res.pvalue:.3f}")

experience vs salary: r = 0.637, p = 6.3e-35
rating vs salary:     rho = 0.030, p = 0.605
8 rows, rating vs age: r = 0.431, p = 0.287

Experience and salary have a fairly strong correlation with a p-value so small that chance is ruled out. Rating and salary show almost no relationship, and the large p-value says even that tiny correlation could be chance. The last line is a warning: with only 8 rows, even a correlation that looks sizeable can have a large p-value, meaning it could easily be a fluke.

Correlation is not causation

A correlation tells you that two variables move together, not why. Three common explanations other than "A causes B":

• A third variable causes both. Ice cream sales and AC repairs in Delhi both rise in May. Neither causes the other; the heat causes both. This hidden third variable is called a confounder.
• The cause runs the other way. Students who score well may enjoy the subject and therefore study more, rather than studying causing the marks.
• Coincidence. Test enough pairs of columns and some will correlate by chance.

To show cause and effect you need a controlled experiment (for example, randomly assigning students to an extra study programme) or careful methods beyond this course. Until then, say "is associated with", not "causes".

Common mistakes

• Only looking at the number. Draw a scatter plot; curves, clusters and outliers hide behind r.
• Using Pearson with outliers or curved data. Use Spearman, or clean the outliers first.
• Claiming causation. "Students with tuition score higher" is a correlation, not proof that tuition works.
• Trusting small samples. Check the p-value or collect more data.

Key points

• The correlation coefficient runs from -1 to +1; the sign is the direction and the size the strength.
• s1.corr(s2) and df.corr() compute Pearson by default; use method="spearman" for ranks.
• Spearman handles curved but consistent relationships and resists outliers.
• scipy.stats.pearsonr() and spearmanr() also return p-values to judge whether a correlation could be chance.
• Correlation does not imply causation; look for confounders, reverse causes and coincidence.

import pandas as pd

weather = pd.read_csv("weather.csv")
weather = weather[weather["temp_max"] < 60]

result = {}
for city, g in weather.groupby("city"):
    result[city] = g["temp_max"].corr(g["humidity"])

print(pd.Series(result).sort_values().round(3))
# The city at the top of the list has the strongest (most negative) relationship.

Mumbai    -0.654
Kochi     -0.611
Chennai   -0.595
Delhi     -0.387
Jaipur    -0.245
dtype: float64