Random Forests & Gradient Boosting

A single decision tree is easy to read but unstable and quick to overfit. The fix is surprisingly simple: build many trees and combine them. A model made of many models is called an ensemble, and tree ensembles are among the most successful methods for tabular data: spreadsheets, transaction logs, customer records.

In this lesson you will learn the two main ways to combine trees, random forests (many independent trees that vote) and gradient boosting (trees built one after another, each fixing the last one's mistakes), and use scikit-learn's fast HistGradientBoostingClassifier.

Random forests: many trees vote

Ask one cricket fan to predict a match and you get one opinion, possibly a strange one. Ask a hundred fans who watched different matches and take the majority, and individual quirks cancel out. A random forest does the same with trees, using two sources of randomness so that the trees are different from each other:

• Bootstrap samples: each tree trains on a random sample of the rows, drawn with replacement, so some rows appear twice and others not at all. Training on bootstrap samples and averaging is called bagging (bootstrap aggregating).
• Random features: at each split, a tree may only choose among a random subset of the features (max_features="sqrt" by default for classifiers, meaning the square root of the number of features).

Each tree overfits in its own way, but their mistakes point in different directions, so the vote is much more stable and accurate than any single tree.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
print(f"Single tree        test {tree.score(X_test, y_test):.3f}")

for n in [20, 40, 80]:
    forest = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=42)
    forest.fit(X_train, y_train)
    print(f"Forest of {n:>2} trees  test {forest.score(X_test, y_test):.3f}   OOB {forest.oob_score_:.3f}")

Single tree        test 0.923
Forest of 20 trees  test 0.958   OOB 0.955
Forest of 40 trees  test 0.951   OOB 0.955
Forest of 80 trees  test 0.951   OOB 0.960

Every forest beats the single tree. Beyond a few dozen trees the score moves only by a test sample or two; extra trees never cause overfitting, they only cost time. The default is n_estimators=100. (With very few trees, scikit-learn warns that some rows have no OOB score yet, because every tree happened to see them.)

oob_score=True gives a free validation score. Each tree never saw about a third of the training rows (the ones left out of its bootstrap sample), so every row can be scored by the trees that did not train on it. This out-of-bag (OOB) score is close to the test score without using the test set.

Which features matter? Permutation importance

Forests have feature_importances_ like single trees, but these impurity-based numbers can mislead, as mentioned in Decision Trees. Permutation importance is a more honest check: shuffle one feature's column in the test set and measure how much the score drops. If shuffling a feature hurts a lot, the model relies on it.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

data = load_diabetes()
rng = np.random.default_rng(42)
X = np.column_stack([data.data, rng.normal(size=len(data.target))])  # add a pure-noise column
names = list(data.feature_names) + ["noise"]
X_train, X_test, y_train, y_test = train_test_split(X, data.target, test_size=0.25, random_state=42)

forest = RandomForestRegressor(n_estimators=60, random_state=42).fit(X_train, y_train)
result = permutation_importance(forest, X_test, y_test, n_repeats=5, random_state=42)
print("feature  R² drop  impurity")
for i in np.argsort(result.importances_mean)[::-1]:
    print(f"{names[i]:<8} {result.importances_mean[i]:7.3f}  {forest.feature_importances_[i]:8.3f}")

feature  R² drop  impurity
bmi        0.272     0.326
s5         0.258     0.212
bp         0.035     0.097
s6         0.020     0.061
noise      0.013     0.065
s3         0.012     0.041
s4         0.010     0.028
sex        0.009     0.009
age        0.007     0.054
s2        -0.001     0.058
s1        -0.021     0.048

Forests do regression too: RandomForestRegressor averages the trees' predictions instead of voting. Both measures agree that bmi and s5 matter most. Now look at the noise column, which we filled with random numbers. Impurity importance gives it 6.5%, more than the real feature s6, because a deep tree can always find splits on random values that happen to fit the training rows. Permutation importance, measured on the test set, shows that shuffling it costs only about 0.01 R². Adding a noise column like this is a handy trick: any real feature that scores no better than the noise is not clearly useful.Permutation importance works with any model, not only trees, and it measures what matters on unseen data.

Gradient boosting: trees that learn from mistakes

Boosting builds trees one after another. The first small tree makes rough predictions. The second tree is trained to predict the errors of the first, the third to predict what is still wrong, and so on. Each new tree's contribution is multiplied by a learning rate (such as 0.1), so the model improves in small, careful steps. The name "gradient" comes from the maths used to decide what the next tree should fix.

scikit-learn has two versions. GradientBoostingClassifier is the classic one. HistGradientBoostingClassifier (and HistGradientBoostingRegressor) first groups each feature's values into at most 255 bins, which makes it dramatically faster on large data. It is the one to use today, and it has several practical features:

• Missing values (NaN) are handled natively: no imputer needed. (Recent versions of decision trees and RandomForestClassifier accept NaN too.)
• Categorical features can be used directly: by default (categorical_features="from_dtype"), pandas columns with the category dtype are treated as categories.
• Early stopping: with the default early_stopping="auto", it holds back part of the training data and stops adding trees when the score stops improving, once the dataset has more than 10,000 samples.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
rng = np.random.default_rng(42)
X = X.copy()
X[rng.random(X.shape) < 0.10] = np.nan     # knock out 10% of all values
print("Missing values:", np.isnan(X).sum())

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
for rate in [0.3, 0.1, 0.03]:
    hgb = HistGradientBoostingClassifier(learning_rate=rate, max_iter=100, random_state=42)
    hgb.fit(X_train, y_train)
    print(f"learning_rate={rate:<5} test accuracy {hgb.score(X_test, y_test):.3f}")

Missing values: 1653
learning_rate=0.3   test accuracy 0.965
learning_rate=0.1   test accuracy 0.951
learning_rate=0.03  test accuracy 0.965

Even with 10% of the values missing, boosting stays accurate without any imputation. On this small dataset the learning rate makes little difference. In general, a smaller rate needs more rounds (max_iter) but often generalises a little better, so in practice you set a small rate, allow many rounds, and let early stopping or cross-validation decide when to stop.

Popular boosting libraries

Outside scikit-learn, three libraries are widely used in industry and in competitions: XGBoost, LightGBM and CatBoost (installed with pip install xgboost, lightgbm or catboost). They follow the same idea as HistGradientBoostingClassifier and offer scikit-learn compatible classes such as XGBClassifier, so they fit into the same pipelines and grid searches. Start with scikit-learn's version; switch only if you need a specific feature or more speed.

Common mistakes

• Expecting more trees to fix everything. After a point, extra trees add time, not accuracy. Tune depth and features too.
• Running boosting for too many rounds with a large learning rate. It can overfit; use early stopping or cross-validation.
• Trusting impurity importances as truth. Confirm with permutation_importance on held-out data.
• Scaling features for tree ensembles. It is unnecessary, although it does no harm.

Key points

• Ensembles combine many models; tree ensembles are top performers on tabular data.
• A random forest trains trees on bootstrap samples with random feature subsets and lets them vote.
• oob_score=True gives a free validation score from rows each tree did not see.
• Gradient boosting adds shallow trees one by one, each correcting the previous errors, scaled by a learning rate.
• HistGradientBoostingClassifier is fast, handles missing values natively, supports categorical features and stops early on large data.
• Use permutation importance to see which features a model really relies on.

from sklearn.datasets import load_wine
from sklearn.ensemble import (GradientBoostingClassifier, HistGradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_wine(return_X_y=True)
models = {
    "Decision tree": DecisionTreeClassifier(random_state=42),
    "Random forest": RandomForestClassifier(n_estimators=60, random_state=42),
    "Gradient boosting": GradientBoostingClassifier(random_state=42),
    "Hist gradient boosting": HistGradientBoostingClassifier(random_state=42),
}
for name, model in models.items():
    print(f"{name:<23} {cross_val_score(model, X, y, cv=5).mean():.3f}")

Decision tree           0.865
Random forest           0.967
Gradient boosting       0.939
Hist gradient boosting  0.955