Decision Trees

A decision tree makes predictions the way a person follows a flowchart: "Is the petal shorter than 2.5 cm? Then it is a setosa. If not, is the petal narrower than 1.75 cm?" Each question splits the data, and the answers lead to a final decision.

Trees are easy to read, need no scaling and handle mixed kinds of features, which is why they are popular in banks, hospitals and anywhere decisions must be explained. They are also the building block of random forests and gradient boosting. In this lesson you will train, read, control and plot decision trees.

Your first tree

A tree has a root (the first question), internal nodes (further questions) and leaves (final answers). Each question compares one feature with a threshold.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=42).fit(iris.data, iris.target)
print(export_text(tree, feature_names=iris.feature_names))
print("Prediction:", iris.target_names[tree.predict([[6.0, 2.9, 4.5, 1.5]])[0]])

|--- petal length (cm) <= 2.45
|   |--- class: 0
|--- petal length (cm) >  2.45
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

Prediction: versicolor

Two questions are enough to classify most iris flowers. The new flower has a petal longer than 2.45 cm and narrower than 1.75 cm, so it lands in the class 1 (versicolor) leaf. export_text prints the rules, so anyone can check them by hand.

How a tree chooses its questions

At every node the tree tries many features and thresholds and keeps the split that makes the two groups as pure as possible, meaning each group is mostly one class. Purity is measured with an impurity score. scikit-learn's default is Gini impurity:

Gini = 1 - (p1² + p2² + ...)

where p1, p2 are the shares of each class in the node. A pure node has Gini 0; a 50/50 node has Gini 0.5. Suppose a cooperative bank in Rohtak has 12 past loan applications, 6 approved and 6 rejected, and tries the question "Is monthly income above ₹40,000?".

import numpy as np

def gini(counts):
    p = np.array(counts) / sum(counts)
    return 1 - np.sum(p ** 2)

print(f"Pure node [10, 0]: {gini([10, 0]):.3f}")
print(f"Mixed node [5, 5]: {gini([5, 5]):.3f}")

parent = [6, 6]                  # 6 approved, 6 rejected
left, right = [5, 1], [1, 5]     # income above / not above Rs 40,000
after = (sum(left) * gini(left) + sum(right) * gini(right)) / sum(parent)
print(f"Before split {gini(parent):.3f}, after split {after:.3f}, gain {gini(parent) - after:.3f}")

Pure node [10, 0]: 0.000
Mixed node [5, 5]: 0.500
Before split 0.500, after split 0.278, gain 0.222

The split reduces impurity from 0.500 to 0.278 (a weighted average of both sides). The tree compares this gain with every other possible question and picks the largest. It then repeats the process inside each side, until the leaves are pure or a stopping rule is reached. The alternative criterion="entropy" (or "log_loss") measures impurity with information theory instead; results are usually very similar.

Controlling the size of the tree

Left alone, a tree keeps splitting until every leaf is pure. It memorises the training data, including its noise, and overfits. You limit growth with a few parameters:

• max_depth: the maximum number of questions from root to leaf.
• min_samples_leaf: every leaf must contain at least this many training samples.
• max_leaf_nodes: an upper limit on the number of leaves.
• ccp_alpha: cost-complexity pruning; grow the full tree, then cut back branches that add little. Larger values prune more.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

settings = [{}, {"max_depth": 2}, {"max_depth": 4}, {"min_samples_leaf": 10}, {"ccp_alpha": 0.01}]
for params in settings:
    tree = DecisionTreeClassifier(random_state=42, **params).fit(X_train, y_train)
    print(f"{str(params):<23} leaves {tree.get_n_leaves():>2}   "
          f"train {tree.score(X_train, y_train):.3f}   test {tree.score(X_test, y_test):.3f}")

{}                      leaves 18   train 1.000   test 0.923
{'max_depth': 2}        leaves  4   train 0.958   test 0.909
{'max_depth': 4}        leaves 11   train 0.988   test 0.944
{'min_samples_leaf': 10} leaves 10   train 0.962   test 0.951
{'ccp_alpha': 0.01}     leaves  5   train 0.972   test 0.944

The unlimited tree is perfect on training data, yet on test data it is beaten by every limited tree except the tiny depth-2 one. The limited trees are smaller, easier to read and trust, and here also more accurate. Choose these settings with cross-validation, as you will learn in Cross-Validation & Hyperparameter Tuning.

Plotting a tree and feature importance

plot_tree draws the tree with matplotlib. With filled=True, each node is coloured by its majority class. The fitted tree also has feature_importances_: how much each feature reduced impurity across all its splits, scaled to add up to 1.

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier, plot_tree

data = load_breast_cancer()
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(data.data, data.target)

top = np.argsort(tree.feature_importances_)[::-1][:4]
for i in top:
    print(f"{data.feature_names[i]:<22} {tree.feature_importances_[i]:.3f}")

plt.figure(figsize=(12, 6))
plot_tree(tree, feature_names=list(data.feature_names),
          class_names=list(data.target_names), filled=True, fontsize=7)
plt.show()

worst radius           0.757
worst concave points   0.117
worst texture          0.042
mean concave points    0.035

Most of the work is done by worst radius, the largest radius measured in the tumour. Be careful with these numbers: impurity-based importance can favour features with many distinct values, and when two features are correlated, the tree may use one and ignore the other. Random Forests & Gradient Boosting shows a more reliable method called permutation importance.

Regression trees

DecisionTreeRegressor uses the same idea for numbers. Each leaf predicts the average target value of its training samples, and splits are chosen to reduce the squared error (criterion="squared_error", the default). The prediction is therefore a staircase rather than a smooth line.

import numpy as np
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(42)
speed = rng.uniform(20, 100, 60).reshape(-1, 1)                      # km/h
mileage = 22 - 0.004 * (speed.ravel() - 55) ** 2 + rng.normal(0, 1, 60)

for depth in [1, 3]:
    reg = DecisionTreeRegressor(max_depth=depth, random_state=42).fit(speed, mileage)
    preds = reg.predict([[30], [55], [90]])
    print(f"depth {depth}: {reg.get_n_leaves()} leaves, mileage at 30/55/90 km/h = {preds.round(1)}")

depth 1: 2 leaves, mileage at 30/55/90 km/h = [20.3 20.3 16.3]
depth 3: 8 leaves, mileage at 30/55/90 km/h = [18.9 21.3 15.8]

A depth-1 tree has only two possible answers, so it misses the peak at 55 km/h. A depth-3 tree, with 8 steps, follows the curve much better. Like KNN, a tree cannot extrapolate: beyond the fastest training speed, it simply repeats its last step.

Strengths and weaknesses

Trees are easy to explain, work with features on any scale, capture non-linear patterns and interactions, and are fast to predict. Their big weaknesses are that they overfit easily and are unstable: a small change in the training data can produce a completely different tree. Combining many trees fixes both problems, which is exactly what random forests and gradient boosting do.

Common mistakes

• Leaving the tree unlimited. A perfect training score with a weaker test score is classic overfitting. Set max_depth, min_samples_leaf or ccp_alpha.
• Scaling features for a tree. It does no harm, but it does nothing either; trees only compare values with thresholds.
• Trusting impurity importance blindly. Check with permutation importance before drawing conclusions.
• Forgetting random_state. When several splits are equally good, the tree picks among them at random, so results can change between runs.

Key points

• A decision tree asks a sequence of threshold questions and predicts at the leaves.
• Splits are chosen to reduce impurity, measured by Gini (default) or entropy for classification and squared error for regression.
• Unlimited trees overfit; control them with max_depth, min_samples_leaf, max_leaf_nodes or ccp_alpha.
• export_text and plot_tree show the learned rules; feature_importances_ summarises which features were used.
• Trees need no scaling, cannot extrapolate, and are unstable on their own.

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

wine = load_wine()
results = {}
for depth in range(1, 7):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    results[depth] = cross_val_score(tree, wine.data, wine.target, cv=5).mean()
    print(f"max_depth={depth}: {results[depth]:.3f}")

best = max(results, key=results.get)
tree = DecisionTreeClassifier(max_depth=best, random_state=42).fit(wine.data, wine.target)
top = np.argsort(tree.feature_importances_)[::-1][:3]
print("Best depth:", best)
print("Top features:", [wine.feature_names[i] for i in top])

max_depth=1: 0.646
max_depth=2: 0.826
max_depth=3: 0.871
max_depth=4: 0.894
max_depth=5: 0.865
max_depth=6: 0.865
Best depth: 4
Top features: ['proline', 'od280/od315_of_diluted_wines', 'flavanoids']