Gradient Boosting in Practice (Machine Learning Tutorial)

For tables of business data (customers, loans, transactions, listings) gradient-boosted trees are usually the strongest model you can train in an afternoon. Random Forests & Gradient Boosting explained how boosting works. This lesson is about using it well at work: feeding it categories directly, stopping at the right number of trees, forcing sensible behaviour with constraints, giving a range instead of a single number, and knowing how the popular XGBoost and LightGBM libraries map onto what you already know.

Every example uses the same small generator of used-car listings, make_cars(), so each block runs on its own. Prices are in ₹ lakh (1 lakh = 100,000 rupees). The data is synthetic, but the patterns are realistic: older cars with more kilometres sell for less.

Categories without one-hot encoding

HistGradientBoostingRegressor and HistGradientBoostingClassifier can split directly on categories. With the default categorical_features="from_dtype", any pandas column with the category dtype is treated as categorical. A split can then send any group of categories left and the rest right (for example, {Toyota, Mahindra} versus everything else), which one-hot columns can only mimic with several splits.

PythonRun it yourself

import numpy as np
import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder

def make_cars(n=4000, seed=42):
    """Synthetic used-car listings; price in ₹ lakh."""
    rng = np.random.default_rng(seed)
    base = {"Maruti": 6, "Hyundai": 7.5, "Tata": 7, "Mahindra": 10, "Honda": 9, "Toyota": 14}
    df = pd.DataFrame({"brand": rng.choice(list(base), n),
                       "fuel": rng.choice(["Petrol", "Diesel", "CNG"], n, p=[0.55, 0.3, 0.15]),
                       "age_years": rng.integers(0, 15, n),
                       "owners": rng.choice([1, 2, 3], n, p=[0.6, 0.3, 0.1])})
    df["km_thousands"] = (df["age_years"] * rng.uniform(6, 15, n) + rng.uniform(0, 10, n)).round(1)
    df["price_lakh"] = (df["brand"].map(base) * np.exp(-0.1 * df["age_years"])
                        * (1 - 0.002 * df["km_thousands"]) * df["fuel"].map({"Petrol": 1, "Diesel": 1.1, "CNG": 0.9})
                        * (1 - 0.06 * (df["owners"] - 1)) * rng.lognormal(0, 0.08, n)).round(2)
    return df.astype({"brand": "category", "fuel": "category"})

cars = make_cars()
print(cars.head(3))
X, y = cars.drop(columns="price_lakh"), cars["price_lakh"]

native = HistGradientBoostingRegressor(random_state=42)
one_hot = make_pipeline(
    make_column_transformer((OneHotEncoder(), ["brand", "fuel"]), remainder="passthrough"),
    HistGradientBoostingRegressor(random_state=42))
for name, model in [("native categories", native), ("one-hot encoded", one_hot)]:
    mae = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"{name:<18} MAE ₹{mae:.3f} lakh")
print("Treated as categorical:", native.fit(X, y).is_categorical_)

Output

      brand    fuel  age_years  owners  km_thousands  price_lakh
0    Maruti  Diesel          7       1          81.2        2.37
1     Honda  Petrol          6       2          58.8        3.97
2  Mahindra     CNG          1       1           8.9        6.95
native categories  MAE ₹0.293 lakh
one-hot encoded    MAE ₹0.294 lakh
Treated as categorical: [ True  True False False False]

Both versions reach the same error, about ₹0.29 lakh (₹29,000) on average. With only six brands and three fuels, one-hot encoding costs nothing. is_categorical_ confirms that the model treated brand and fuel as categories without any encoder in front of it.

Native categories are most useful when a column has many values (dozens of car models, hundreds of pin codes), where one-hot encoding creates hundreds of sparse columns. Two practical rules: the column must have the category dtype (plain text columns raise an error), and categories not seen during training are treated like missing values.

Early stopping: how many trees?

More boosting rounds always fit the training data better, but at some point they start fitting noise. Early stopping holds back part of the training data (validation_fraction, 10% by default) and stops when the validation score has not improved for n_iter_no_change rounds. By default (early_stopping="auto") it is switched on only for datasets above 10,000 rows, so turn it on yourself for smaller data. Then set max_iter high and let the model choose.

PythonRun it yourself

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

def make_cars(n=4000, seed=42):
    rng = np.random.default_rng(seed)
    base = {"Maruti": 6, "Hyundai": 7.5, "Tata": 7, "Mahindra": 10, "Honda": 9, "Toyota": 14}
    df = pd.DataFrame({"brand": rng.choice(list(base), n),
                       "fuel": rng.choice(["Petrol", "Diesel", "CNG"], n, p=[0.55, 0.3, 0.15]),
                       "age_years": rng.integers(0, 15, n),
                       "owners": rng.choice([1, 2, 3], n, p=[0.6, 0.3, 0.1])})
    df["km_thousands"] = (df["age_years"] * rng.uniform(6, 15, n) + rng.uniform(0, 10, n)).round(1)
    df["price_lakh"] = (df["brand"].map(base) * np.exp(-0.1 * df["age_years"])
                        * (1 - 0.002 * df["km_thousands"]) * df["fuel"].map({"Petrol": 1, "Diesel": 1.1, "CNG": 0.9})
                        * (1 - 0.06 * (df["owners"] - 1)) * rng.lognormal(0, 0.08, n)).round(2)
    return df.astype({"brand": "category", "fuel": "category"})

cars = make_cars()
X, y = cars.drop(columns="price_lakh"), cars["price_lakh"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

for rate in [0.3, 0.1, 0.03]:
    model = HistGradientBoostingRegressor(learning_rate=rate, max_iter=3000, early_stopping=True,
                                          n_iter_no_change=20, random_state=42).fit(X_train, y_train)
    mae = mean_absolute_error(y_test, model.predict(X_test))
    print(f"learning_rate {rate:<5} stopped after {model.n_iter_:>4} trees  test MAE ₹{mae:.3f} lakh")

Output

learning_rate 0.3   stopped after   37 trees  test MAE ₹0.296 lakh
learning_rate 0.1   stopped after   76 trees  test MAE ₹0.294 lakh
learning_rate 0.03  stopped after  231 trees  test MAE ₹0.290 lakh

A large learning rate stops after 37 trees; a small one needs 231 trees and gives the lowest test error (₹0.290 lakh). Early stopping chooses the number of trees for you, so the learning rate is the main knob left. The differences are small on this clean data; on larger, noisier data they are often bigger.

Monotonic constraints: make the model sensible

Everyone in the used-car business knows that, other things equal, more kilometres never makes a car more valuable. A boosted model does not know this. Where data is thin, it can learn small bumps where the price rises with kilometres, which looks absurd to a customer comparing two listings. A monotonic constraint forbids that: 1 means "prediction may only go up as this feature increases", -1 means "only down", 0 means no constraint. You can pass a dictionary keyed by column name.

PythonRun it yourself

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split

def make_cars(n=4000, seed=42):
    rng = np.random.default_rng(seed)
    base = {"Maruti": 6, "Hyundai": 7.5, "Tata": 7, "Mahindra": 10, "Honda": 9, "Toyota": 14}
    df = pd.DataFrame({"brand": rng.choice(list(base), n),
                       "fuel": rng.choice(["Petrol", "Diesel", "CNG"], n, p=[0.55, 0.3, 0.15]),
                       "age_years": rng.integers(0, 15, n),
                       "owners": rng.choice([1, 2, 3], n, p=[0.6, 0.3, 0.1])})
    df["km_thousands"] = (df["age_years"] * rng.uniform(6, 15, n) + rng.uniform(0, 10, n)).round(1)
    df["price_lakh"] = (df["brand"].map(base) * np.exp(-0.1 * df["age_years"])
                        * (1 - 0.002 * df["km_thousands"]) * df["fuel"].map({"Petrol": 1, "Diesel": 1.1, "CNG": 0.9})
                        * (1 - 0.06 * (df["owners"] - 1)) * rng.lognormal(0, 0.08, n)).round(2)
    return df.astype({"brand": "category", "fuel": "category"})

cars = make_cars()
X, y = cars.drop(columns="price_lakh"), cars["price_lakh"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

free = HistGradientBoostingRegressor(random_state=42).fit(X_train, y_train)
constrained = HistGradientBoostingRegressor(
    monotonic_cst={"km_thousands": -1, "age_years": -1, "owners": -1},
    random_state=42).fit(X_train, y_train)

# One 5-year-old, first-owner Hyundai petrol car at different kilometres
km = np.arange(30, 91, 10)
car = pd.DataFrame({"brand": "Hyundai", "fuel": "Petrol", "age_years": 5, "owners": 1,
                    "km_thousands": km}).astype({"brand": X["brand"].dtype, "fuel": X["fuel"].dtype})
print("km (thousands):", km)
print("free model:    ", free.predict(car).round(2))
print("constrained:   ", constrained.predict(car).round(2))
for name, model in [("free", free), ("constrained", constrained)]:
    print(f"{name:<12} test R² {model.score(X_test, y_test):.4f}")

Output

km (thousands): [30 40 50 60 70 80 90]
free model:     [4.28 4.08 4.12 4.04 4.02 3.88 3.89]
constrained:    [4.27 4.2  4.2  3.97 3.95 3.79 3.73]
free         test R² 0.9736
constrained  test R² 0.9721

The free model says the car is worth ₹4.08 lakh at 40,000 km but ₹4.12 lakh at 50,000 km, and the price creeps up again between 80,000 and 90,000 km. The bumps are small, but impossible to explain to a seller. The constrained model never rises as kilometres increase (equal neighbouring values are allowed), and its test R² is almost the same (0.972 against 0.974). A tiny loss of fit buys a model that respects domain knowledge.

Constraints are popular in credit scoring and pricing, where regulators and customers expect "more income never lowers your score". interaction_cst is a related option that limits which features may be combined inside one tree.

A range, not just a number: quantile loss

A seller in Pune asks "what will my car fetch?" A single number hides how uncertain it is. With loss="quantile", the model predicts a chosen quantile instead of the average: quantile=0.1 gives a price that 10% of similar cars sell below, quantile=0.9 one that 90% sell below. Together they form an 80% prediction interval. Check on the test set that about 80% of real prices fall inside it.

PythonRun it yourself

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split

def make_cars(n=4000, seed=42):
    rng = np.random.default_rng(seed)
    base = {"Maruti": 6, "Hyundai": 7.5, "Tata": 7, "Mahindra": 10, "Honda": 9, "Toyota": 14}
    df = pd.DataFrame({"brand": rng.choice(list(base), n),
                       "fuel": rng.choice(["Petrol", "Diesel", "CNG"], n, p=[0.55, 0.3, 0.15]),
                       "age_years": rng.integers(0, 15, n),
                       "owners": rng.choice([1, 2, 3], n, p=[0.6, 0.3, 0.1])})
    df["km_thousands"] = (df["age_years"] * rng.uniform(6, 15, n) + rng.uniform(0, 10, n)).round(1)
    df["price_lakh"] = (df["brand"].map(base) * np.exp(-0.1 * df["age_years"])
                        * (1 - 0.002 * df["km_thousands"]) * df["fuel"].map({"Petrol": 1, "Diesel": 1.1, "CNG": 0.9})
                        * (1 - 0.06 * (df["owners"] - 1)) * rng.lognormal(0, 0.08, n)).round(2)
    return df.astype({"brand": "category", "fuel": "category"})

cars = make_cars()
X, y = cars.drop(columns="price_lakh"), cars["price_lakh"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

low = HistGradientBoostingRegressor(loss="quantile", quantile=0.1, random_state=42).fit(X_train, y_train)
mid = HistGradientBoostingRegressor(loss="quantile", quantile=0.5, random_state=42).fit(X_train, y_train)
high = HistGradientBoostingRegressor(loss="quantile", quantile=0.9, random_state=42).fit(X_train, y_train)

lo, md, hi = low.predict(X_test), mid.predict(X_test), high.predict(X_test)
inside = (y_test >= lo) & (y_test <= hi)
print(f"Share of test prices inside the 80% interval: {inside.mean():.1%}")
for i in range(3):
    print(f"{X_test.iloc[i]['brand']:<9} {X_test.iloc[i]['age_years']:>2} yrs: "
          f"₹{lo[i]:.2f} to ₹{hi[i]:.2f} lakh (median ₹{md[i]:.2f}, actual ₹{y_test.iloc[i]:.2f})")

Output

Share of test prices inside the 80% interval: 69.0%
Mahindra   9 yrs: ₹3.33 to ₹4.10 lakh (median ₹3.56, actual ₹3.31)
Mahindra   8 yrs: ₹3.73 to ₹4.57 lakh (median ₹3.98, actual ₹5.26)
Maruti    10 yrs: ₹1.53 to ₹2.01 lakh (median ₹1.87, actual ₹1.60)

Only 69% of test prices fall inside the interval, well short of the 80% it promises. The quantile models fit their training rows a little too closely, so the interval is too narrow on new data, and this is exactly why you check coverage. Common fixes: widen the interval using a held-out calibration set (the idea behind conformal prediction), regularise more (for example a larger min_samples_leaf), or use quantiles further apart such as 0.05 and 0.95, then check again.

What to tune

HistGradientBoosting works well with defaults, so tune only after you have a baseline. The parameters that matter most:

learning_rate and max_iter: tune together, or fix a small rate and use early stopping.
max_leaf_nodes (default 31) or max_depth: how complex each tree is.
min_samples_leaf (default 20): raise it for noisy data.
l2_regularization (default 0): shrinks leaf values; try values from 0.01 to 10.
max_features (default 1.0): the share of features each split may consider; below 1 adds randomness, like a random forest.

Use RandomizedSearchCV from Cross-Validation & Hyperparameter Tuning with log-uniform ranges, as in the exercise below.

XGBoost and LightGBM

Two outside libraries dominate boosting at work: XGBoost and LightGBM (CatBoost is a third, strong with categories). Both offer scikit-learn style classes, so they drop into pipelines, cross_val_score and searches. Their ideas are the same; the parameter names differ:

Idea	scikit-learn HGB	XGBoost	LightGBM
Number of trees	`max_iter`	`n_estimators`	`n_estimators`
Step size	`learning_rate`	`learning_rate`	`learning_rate`
Tree size	`max_leaf_nodes`, `max_depth`	`max_depth`, `max_leaves`	`num_leaves`, `max_depth`
Smallest leaf	`min_samples_leaf` (rows)	`min_child_weight` (a weight, not a row count)	`min_child_samples` (rows)
L2 penalty	`l2_regularization`	`reg_lambda`	`reg_lambda`
Column sampling	`max_features` (per split)	`colsample_bytree`	`colsample_bytree`

They are installed separately and do not run in the browser editor. A typical LightGBM run with early stopping on a validation set looks like this:

Python

# python -m pip install lightgbm
import lightgbm as lgb

model = lgb.LGBMRegressor(n_estimators=2000, learning_rate=0.05, num_leaves=31, random_state=42)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)],
          callbacks=[lgb.early_stopping(stopping_rounds=50)])
print(model.best_iteration_)

Start with scikit-learn's version: it is already fast, needs no extra install and is maintained with the rest of your pipeline. Move to XGBoost or LightGBM when you need GPU training, very large data, or a feature they have and scikit-learn lacks. Check each library's official documentation for current parameter names, since they change between major versions.

Common mistakes

Passing text columns and expecting native categories. Convert them with astype("category") first.
Running thousands of rounds without early stopping on small data, which overfits.
Tuning before you have a baseline, or tuning on the test set.
Trusting a single price when the decision needs a range; use quantile models and check their coverage.
Ignoring business logic. If a relationship must only go one way, enforce it with monotonic_cst.

Key points

HistGradientBoosting models handle category columns natively with categorical_features="from_dtype".
Turn on early_stopping=True for smaller data, set max_iter high, and read n_iter_.
monotonic_cst forces predictions to only rise or only fall with a feature, matching domain knowledge.
loss="quantile" gives prediction intervals; always check their coverage on test data.
Tune learning_rate, max_leaf_nodes, min_samples_leaf and l2_regularization first.
XGBoost and LightGBM follow the same ideas with different parameter names and scikit-learn style APIs.

Exercise

Tune a HistGradientBoostingRegressor(random_state=42) on the car data with RandomizedSearchCV (n_iter=8, cv=3, scoring="neg_mean_absolute_error", random_state=42). Search learning_rate with loguniform(0.02, 0.3), max_leaf_nodes in [7, 15, 31, 63], min_samples_leaf in [5, 20, 50] and l2_regularization with loguniform(0.01, 10). Print the best parameters, the best cross-validated MAE and the test MAE, using train_test_split(X, y, test_size=0.25, random_state=42).

Show solution

Python

import numpy as np
import pandas as pd
from scipy.stats import loguniform
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import RandomizedSearchCV, train_test_split

def make_cars(n=4000, seed=42):
    rng = np.random.default_rng(seed)
    base = {"Maruti": 6, "Hyundai": 7.5, "Tata": 7, "Mahindra": 10, "Honda": 9, "Toyota": 14}
    df = pd.DataFrame({"brand": rng.choice(list(base), n),
                       "fuel": rng.choice(["Petrol", "Diesel", "CNG"], n, p=[0.55, 0.3, 0.15]),
                       "age_years": rng.integers(0, 15, n),
                       "owners": rng.choice([1, 2, 3], n, p=[0.6, 0.3, 0.1])})
    df["km_thousands"] = (df["age_years"] * rng.uniform(6, 15, n) + rng.uniform(0, 10, n)).round(1)
    df["price_lakh"] = (df["brand"].map(base) * np.exp(-0.1 * df["age_years"])
                        * (1 - 0.002 * df["km_thousands"]) * df["fuel"].map({"Petrol": 1, "Diesel": 1.1, "CNG": 0.9})
                        * (1 - 0.06 * (df["owners"] - 1)) * rng.lognormal(0, 0.08, n)).round(2)
    return df.astype({"brand": "category", "fuel": "category"})

cars = make_cars()
X, y = cars.drop(columns="price_lakh"), cars["price_lakh"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

space = {"learning_rate": loguniform(0.02, 0.3), "max_leaf_nodes": [7, 15, 31, 63],
         "min_samples_leaf": [5, 20, 50], "l2_regularization": loguniform(0.01, 10)}
search = RandomizedSearchCV(HistGradientBoostingRegressor(random_state=42), space, n_iter=8, cv=3,
                            scoring="neg_mean_absolute_error", random_state=42).fit(X_train, y_train)
print({k: round(float(v), 3) if isinstance(v, float) else v for k, v in search.best_params_.items()})
print(f"Best CV MAE: ₹{-search.best_score_:.3f} lakh")
print(f"Test MAE:    ₹{mean_absolute_error(y_test, search.predict(X_test)):.3f} lakh")

Output

{'l2_regularization': 0.075, 'learning_rate': 0.105, 'max_leaf_nodes': 15, 'min_samples_leaf': 50}
Best CV MAE: ₹0.294 lakh
Test MAE:    ₹0.289 lakh

The search settles on a moderate learning rate (about 0.1), smaller trees (15 leaves) and large leaves (50 rows), and the test MAE of ₹0.289 lakh is only slightly better than the defaults. That is typical of gradient boosting: the defaults are good and tuning gives small gains. Spend your effort on features and data quality first.

Quick Quiz

1. How do you tell HistGradientBoostingRegressor that a column is categorical, with the default settings?

With categorical_features="from_dtype", columns with the category dtype are treated as categories.

2. Your dataset has 3,000 rows. Is early stopping on by default?

On smaller data you switch it on yourself and read the chosen number of trees from n_iter_.

3. What does monotonic_cst={"km_thousands": -1} guarantee?

4. Two quantile models predict the 0.1 and 0.9 quantiles. About what share of real values should fall between them?

Categories without one-hot encoding

Early stopping: how many trees?

Monotonic constraints: make the model sensible

A range, not just a number: quantile loss

What to tune

XGBoost and LightGBM

Common mistakes

Key points

Exercise

Quick Quiz

Learn Machine Learning with a mentor