Cross-Validation & Hyperparameter Tuning

A single train/test split gives a single score, and that score depends heavily on which rows happened to land in the test set. Change the random seed and the score can jump by several points. When you compare models or tune settings, you need something steadier.

Cross-validation evaluates a model on several different splits and averages the results. Hyperparameter tuning uses cross-validation to try many settings and keep the best. In this lesson you will use both, with cross_val_score, cross_validate, GridSearchCV and RandomizedSearchCV.

One split is not enough

Here the same decision tree is trained and tested on ten different random splits of the wine data.

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_wine(return_X_y=True)
scores = []
for seed in range(10):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=seed, stratify=y)
    tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
    scores.append(tree.score(X_test, y_test))

print("Accuracy on 10 different splits:", np.round(scores, 2))
print(f"Lowest {min(scores):.2f}, highest {max(scores):.2f}")

Accuracy on 10 different splits: [0.89 0.89 0.96 0.93 0.96 0.87 0.89 0.91 0.78 0.93]
Lowest 0.78, highest 0.96

Same model, same data, and the "accuracy" ranges from 0.78 to 0.96. If you compared two models on one lucky or unlucky split, you could easily pick the wrong one.

k-fold cross-validation

k-fold cross-validation splits the data into k equal parts called folds. It trains on k - 1 folds and tests on the remaining one, then repeats until every fold has been the test fold once. With k = 5, every sample is used for testing exactly once and for training four times. You report the mean of the k scores, and their spread tells you how stable the estimate is.

For classifiers, cv=5 automatically uses stratified folds (StratifiedKFold), which keep the class proportions in every fold. Pass a splitter object to control shuffling. cross_validate goes further: several metrics at once, training scores and timings.

from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=cv)
print("Fold scores:", scores.round(3))
print(f"Mean {scores.mean():.3f}, standard deviation {scores.std():.3f}")

results = cross_validate(model, X, y, cv=cv, scoring=["accuracy", "f1_macro"],
                         return_train_score=True)
for key in ["train_accuracy", "test_accuracy", "test_f1_macro"]:
    print(f"{key:<15} {results[key].mean():.3f}")

Fold scores: [0.972 0.972 0.972 1.    1.   ]
Mean 0.983, standard deviation 0.014
train_accuracy  1.000
test_accuracy   0.983
test_f1_macro   0.983

Always put preprocessing inside a pipeline before cross-validating, so the scaler is refitted on each training fold and nothing leaks from the test fold.

Grid search

Hyperparameters are the settings you choose before training (C, max_depth, n_neighbors), as opposed to the parameters a model learns. GridSearchCV tries every combination in a grid, scores each with cross-validation, and then refits the best combination on all the training data. With a pipeline, parameter names use the step__parameter pattern.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

pipe = Pipeline([("scale", StandardScaler()), ("svm", SVC())])
grid = {"svm__C": [0.1, 1, 10, 100], "svm__gamma": ["scale", 0.001, 0.01, 0.1]}
search = GridSearchCV(pipe, grid, cv=5, scoring="accuracy")
search.fit(X_train, y_train)

print("Best parameters:", search.best_params_)
print(f"Best CV accuracy: {search.best_score_:.3f}")
print(f"Test accuracy:    {search.score(X_test, y_test):.3f}")
table = pd.DataFrame(search.cv_results_)
cols = ["param_svm__C", "param_svm__gamma", "mean_test_score", "rank_test_score"]
print(table[cols].sort_values("rank_test_score").head(4).round(3).to_string(index=False))

Best parameters: {'svm__C': 10, 'svm__gamma': 0.001}
Best CV accuracy: 0.977
Test accuracy:    0.979
 param_svm__C param_svm__gamma  mean_test_score  rank_test_score
         10.0            0.001            0.977                1
          1.0             0.01            0.972                2
        100.0            0.001            0.969                3
          1.0            scale            0.965                4

The grid has 4 × 4 = 16 combinations, and with 5 folds that is 80 fits plus one final refit. cv_results_ holds every score, which is useful for seeing how sensitive the model is. After fitting, the search object itself behaves like the best model: search.predict() and search.score() use it. Note that the test set was used only once, at the very end.

Randomized search

Grids grow fast: five parameters with five values each is 3,125 combinations. RandomizedSearchCV instead tries a fixed number of random combinations (n_iter), drawn from lists or from probability distributions. For parameters like C and gamma, whose useful values span several powers of ten, a log-uniform distribution from SciPy samples small and large values fairly.

from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

pipe = Pipeline([("scale", StandardScaler()), ("svm", SVC())])
distributions = {"svm__C": loguniform(0.01, 1000), "svm__gamma": loguniform(1e-4, 1)}
search = RandomizedSearchCV(pipe, distributions, n_iter=20, cv=5, random_state=42)
search.fit(X_train, y_train)

best = search.best_params_
print(f"Best C = {best['svm__C']:.3f}, gamma = {best['svm__gamma']:.5f}")
print(f"Best CV accuracy: {search.best_score_:.3f}")
print(f"Test accuracy:    {search.score(X_test, y_test):.3f}")

Best C = 10.907, gamma = 0.00048
Best CV accuracy: 0.979
Test accuracy:    0.951

Its best cross-validated score is a little higher than the grid's, yet its test score is lower. Differences this small are mostly noise (the test set has only 143 tumours), so both searches found equally good settings. With the same budget, randomized search explores many more distinct values of each parameter than a grid does, so it often finds good settings faster. scikit-learn also has HalvingGridSearchCV and HalvingRandomSearchCV, which give many candidates a small budget and keep only the best for the next round. They are still experimental: you must first run from sklearn.experimental import enable_halving_search_cv.

Honest scores: nested cross-validation

The best_score_ of a search is slightly optimistic, because you picked the best of many tries using those same folds. To estimate how well the whole tuning procedure works, wrap the search in another cross-validation. This is nested cross-validation: the inner loop tunes, the outer loop tests.

from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
inner = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression()),
                     {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=5)
outer_scores = cross_val_score(inner, X, y, cv=5)
print("Outer fold scores:", outer_scores.round(3))
print(f"Nested CV estimate: {outer_scores.mean():.3f}")

Outer fold scores: [0.972 0.944 1.    1.    1.   ]
Nested CV estimate: 0.983

Other splitters

Choose the splitter that matches how your data was collected:

• TimeSeriesSplit: for time-ordered data; every test fold comes after its training data.
• GroupKFold / StratifiedGroupKFold: when several rows belong to one group (one patient, one customer), keep each group entirely in either training or test.
• RepeatedStratifiedKFold: repeats k-fold with different shuffles for an even steadier estimate on small data.

Common mistakes

• Preprocessing outside the cross-validation. Put scalers, encoders and selectors in the pipeline being cross-validated.
• Reporting best_score_ as the final result. It is optimistic; report the test-set score or a nested CV estimate.
• Huge grids. Start coarse (powers of ten), then refine around the best values, or use randomized search.
• Shuffling time series. Use TimeSeriesSplit so the model never trains on the future.

Key points

• One train/test split gives a noisy score; k-fold cross-validation averages over k splits.
• cross_val_score returns one score per fold; cross_validate adds several metrics, training scores and timings.
• GridSearchCV tries every combination; RandomizedSearchCV samples a fixed number, often from log-uniform distributions.
• Tune pipelines with step__parameter names, and keep the test set for one final check.
• Nested cross-validation gives an honest estimate of a tuned model.

from sklearn.datasets import load_wine
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier

X, y = load_wine(return_X_y=True)
grid = {"max_depth": [2, 3, 4, 6, None], "min_samples_leaf": [1, 3, 5, 10]}
search = GridSearchCV(DecisionTreeClassifier(random_state=42), grid, cv=5).fit(X, y)

print("Best parameters:", search.best_params_)
print(f"Best CV accuracy: {search.best_score_:.3f}")
print("Combinations tried:", len(search.cv_results_["params"]))

Best parameters: {'max_depth': 4, 'min_samples_leaf': 1}
Best CV accuracy: 0.894
Combinations tried: 20