Overfitting & the Bias-Variance Trade-off

You have already met the two ways a model can fail. A model that is too simple misses the pattern (underfitting). A model that is too flexible learns the noise in its training data and fails on new data (overfitting). Almost every decision in machine learning, from choosing max_depth to setting alpha, is about steering between the two.

In this lesson you will measure the two kinds of error behind this, bias and variance, and use two diagnostic tools, the validation curve and the learning curve, to tell which problem your model has and what to do about it.

Reading train and test scores

Comparing the training score with a validation or test score tells you a lot:

The gap between the two scores is the key signal. A small gap with low scores means the model is too simple; a large gap means it is too flexible for the amount of data.

Bias and variance

Imagine training the same kind of model many times, each time on a fresh sample of data, and looking at its prediction for one particular input:

• Bias is how far the average prediction is from the truth. A straight line fitted to a curve is wrong in the same direction every time, whatever sample it gets. That is high bias.
• Variance is how much the prediction jumps around from one training sample to the next. An unlimited decision tree follows every noisy point, so it gives a different answer for every sample. That is high variance.

The expected error of a model can be split into bias squared, plus variance, plus irreducible noise that no model can remove. Making a model more flexible usually lowers bias but raises variance, and making it simpler does the opposite: this is the bias-variance trade-off. We can measure both directly with the car mileage curve from earlier lessons by training each model on 30 different random samples.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.tree import DecisionTreeRegressor

def true_mileage(speed):
    return 22 - 0.005 * (speed - 55) ** 2

models = {
    "Straight line": lambda: LinearRegression(),
    "Degree-2 curve": lambda: make_pipeline(PolynomialFeatures(2), LinearRegression()),
    "Unlimited tree": lambda: DecisionTreeRegressor(random_state=42),
}
predictions = {name: [] for name in models}
rng = np.random.default_rng(42)
for trial in range(30):                      # 30 different training sets
    speed = rng.uniform(20, 100, 60)
    mileage = true_mileage(speed) + rng.normal(0, 1, 60)
    for name, make in models.items():
        model = make().fit(speed.reshape(-1, 1), mileage)
        predictions[name].append(model.predict([[40]])[0])

truth = true_mileage(40)
print(f"True mileage at 40 km/h: {truth:.2f}")
for name, preds in predictions.items():
    preds = np.array(preds)
    print(f"{name:<15} average {preds.mean():6.2f}  bias {preds.mean() - truth:+.2f}  spread {preds.std():.2f}")

True mileage at 40 km/h: 20.88
Straight line   average  20.17  bias -0.71  spread 0.47
Degree-2 curve  average  20.88  bias +0.01  spread 0.15
Unlimited tree  average  20.72  bias -0.16  spread 0.95

The straight line is consistently wrong (large bias) but steady (small spread). The unlimited tree is right on average but its individual answers scatter widely (high variance). The degree-2 curve, which matches the true shape, has both low bias and low variance: that is the sweet spot we are looking for.

Validation curves: finding the right complexity

A validation curve shows training and cross-validated scores as one hyperparameter changes. scikit-learn's validation_curve runs the cross-validation for every value for you.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
depths = [1, 2, 3, 4, 6, 8, 12]
train_scores, val_scores = validation_curve(
    DecisionTreeClassifier(random_state=42), X, y,
    param_name="max_depth", param_range=depths, cv=5)

for d, tr, va in zip(depths, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    print(f"max_depth={d:<3} train {tr:.3f}   validation {va:.3f}   gap {tr - va:.3f}")

max_depth=1   train 0.924   validation 0.900   gap 0.025
max_depth=2   train 0.952   validation 0.928   gap 0.024
max_depth=3   train 0.974   validation 0.919   gap 0.055
max_depth=4   train 0.985   validation 0.921   gap 0.064
max_depth=6   train 0.996   validation 0.921   gap 0.076
max_depth=8   train 1.000   validation 0.916   gap 0.084
max_depth=12  train 1.000   validation 0.917   gap 0.083

As depth grows, the training score climbs towards 1.0 and the gap widens. The validation score rises at first (less bias), then stops improving or drops (more variance). The best depth is where the validation score peaks; beyond it, extra flexibility only buys memorisation.

Learning curves: would more data help?

A learning curve shows scores as the amount of training data grows. It answers a very practical question: is it worth collecting more data?

• High bias: training and validation scores meet quickly at a low level. More data will not help; you need a more flexible model or better features.
• High variance: a big gap that shrinks slowly as data grows. More data, or more regularisation, will help.

import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharey=True)
for ax, depth in zip(axes, [1, None]):
    sizes, train_scores, val_scores = learning_curve(
        DecisionTreeClassifier(max_depth=depth, random_state=42), X, y,
        train_sizes=[0.1, 0.3, 0.5, 0.75, 1.0], cv=5, shuffle=True, random_state=42)
    tr, va = train_scores.mean(axis=1), val_scores.mean(axis=1)
    print(f"max_depth={depth}: sizes {sizes.tolist()}")
    print(f"   train {tr.round(3).tolist()}")
    print(f"   valid {va.round(3).tolist()}")
    ax.plot(sizes, tr, "o-", label="training")
    ax.plot(sizes, va, "o-", label="validation")
    ax.set_title(f"Tree, max_depth={depth}")
    ax.set_xlabel("Training samples")
axes[0].set_ylabel("Accuracy")
axes[0].legend()
plt.show()

max_depth=1: sizes [45, 136, 227, 341, 455]
   train [0.969, 0.938, 0.929, 0.924, 0.924]
   valid [0.896, 0.902, 0.902, 0.902, 0.9]
max_depth=None: sizes [45, 136, 227, 341, 455]
   train [1.0, 1.0, 1.0, 1.0, 1.0]
   valid [0.888, 0.925, 0.907, 0.912, 0.917]

The depth-1 tree (a single question) shows the high-bias pattern: training and validation scores sit close together and do not improve with more data. The unlimited tree shows the high-variance pattern: a perfect training score with a gap to validation that only slowly narrows as data grows. scikit-learn also offers LearningCurveDisplay and ValidationCurveDisplay, which draw these plots in one line.

What to do about it

Bagging methods such as random forests mainly reduce variance by averaging many high-variance trees, while boosting mainly reduces bias by adding small trees step by step. Whatever you change, judge it by cross-validated scores, never by the training score.

Common mistakes

• Judging a model by its training score. It always rewards more flexibility. Look at validation scores and the gap.
• Collecting more data for a high-bias model. If the learning curves have already met at a low level, more rows will not help; change the model or features.
• Tuning on the test set. Use validation curves and cross-validation on training data; keep the test set for the end.
• Thinking overfitting only happens with big models. A small dataset can be overfitted by a modest model; always compare train and validation scores.

Key points

• Underfitting means high bias: the model is too simple and wrong in a consistent way.
• Overfitting means high variance: the model changes a lot with the training sample and memorises noise.
• More flexibility lowers bias and raises variance; good models balance the two.
• A validation curve shows scores across a hyperparameter; pick the value where the validation score peaks.
• A learning curve shows whether more data would help: a large shrinking gap says yes, two low converged lines say no.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import validation_curve
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
ks = [1, 3, 7, 15, 31, 63, 127]
train_scores, val_scores = validation_curve(
    make_pipeline(StandardScaler(), KNeighborsClassifier()), X, y,
    param_name="kneighborsclassifier__n_neighbors", param_range=ks, cv=5)

for k, tr, va in zip(ks, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    print(f"k={k:<4} train {tr:.3f}   validation {va:.3f}")

k=1    train 1.000   validation 0.954
k=3    train 0.982   validation 0.960
k=7    train 0.972   validation 0.970
k=15   train 0.967   validation 0.961
k=31   train 0.956   validation 0.953
k=63   train 0.949   validation 0.944
k=127  train 0.920   validation 0.917