Calibration & Cost-Based Thresholds

A classifier's predict_proba returns numbers between 0 and 1, but are they really probabilities? If a model says "0.8 chance of churn" for 100 customers, roughly 80 of them should actually leave. When that holds, the model is calibrated. Many models are not: some are overconfident, others too timid.

Calibration matters whenever a probability feeds a decision with money attached: how much to reserve for bad loans, which customers to call, whether to send a delivery agent. In this lesson you will measure calibration, fix it with CalibratedClassifierCV, and then turn calibrated probabilities into decisions using rupee costs.

Measuring calibration

A calibration curve (also called a reliability diagram) groups predictions into bins, such as all predictions between 0.6 and 0.8, and compares the average predicted probability in each bin with the share of positives that actually occurred. A perfectly calibrated model lies on the diagonal. Two single-number summaries are common:

• Brier score: the mean squared difference between the predicted probability and the actual outcome (0 or 1). Lower is better.
• Log loss: punishes confident wrong predictions very heavily. Lower is better.

import matplotlib.pyplot as plt
from sklearn.calibration import CalibrationDisplay, calibration_curve
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=8000, n_features=10, n_informative=4,
                           n_redundant=4, weights=[0.85, 0.15], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y)

models = {"Logistic": make_pipeline(StandardScaler(), LogisticRegression()),
          "Balanced": make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced")),
          "Naive Bayes": GaussianNB()}
fig, ax = plt.subplots(figsize=(5, 5))
for name, model in models.items():
    model.fit(X_train, y_train)
    proba = model.predict_proba(X_test)[:, 1]
    observed, predicted = calibration_curve(y_test, proba, n_bins=5, strategy="quantile")
    print(f"{name:<12} Brier {brier_score_loss(y_test, proba):.4f}  log loss {log_loss(y_test, proba):.3f}")
    print("   predicted:", predicted.round(2), "\n   observed: ", observed.round(2))
    CalibrationDisplay.from_predictions(y_test, proba, n_bins=5, strategy="quantile", name=name, ax=ax)
ax.set_title("Calibration curves")
plt.show()

Logistic     Brier 0.0970  log loss 0.337
   predicted: [0.02 0.06 0.1  0.15 0.4 ] 
   observed:  [0.05 0.07 0.09 0.12 0.43]
Balanced     Brier 0.1738  log loss 0.528
   predicted: [0.1  0.24 0.37 0.51 0.77] 
   observed:  [0.02 0.06 0.06 0.18 0.44]
Naive Bayes  Brier 0.0935  log loss 0.334
   predicted: [0.   0.   0.05 0.23 0.67] 
   observed:  [0.01 0.03 0.1  0.16 0.46]

Plain logistic regression is close to the diagonal: in its top bin it predicts 0.40 on average and 43% of those customers actually churn. The balanced version, trained with class_weight="balanced" as in Imbalanced Data, is badly overconfident: it predicts 0.77 in its top bin where only 44% churn, and 0.10 in its lowest bin where just 2% churn. Its Brier score (0.174) is almost double the plain model's. Naive Bayes is overconfident at the top too (0.67 predicted, 0.46 observed). Class weights and resampling help a model notice rare cases, but they distort its probabilities, which is the most common way calibration breaks in practice.

Fixing calibration with CalibratedClassifierCV

CalibratedClassifierCV learns a small mapping from the model's raw scores to better probabilities. It offers three methods:

The mapping must be learned on data the model was not trained on, or it learns nothing useful. There are two ways to do that:

• Cross-validation (cv=5): the model is trained on four folds and calibrated on the fifth, five times; predictions average the five pairs.
• A separate calibration set: train the model yourself, wrap it in FrozenEstimator so it is not retrained, and fit the calibrator on held-out rows.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification
from sklearn.frozen import FrozenEstimator
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=8000, n_features=10, n_informative=4,
                           n_redundant=4, weights=[0.85, 0.15], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y)

def report(name, model):
    proba = model.predict_proba(X_test)[:, 1]
    print(f"{name:<22} Brier {brier_score_loss(y_test, proba):.4f}  ROC-AUC {roc_auc_score(y_test, proba):.3f}")

def balanced_logistic():
    return make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced"))

report("Balanced, raw", balanced_logistic().fit(X_train, y_train))
for method in ["sigmoid", "isotonic"]:
    calibrated = CalibratedClassifierCV(balanced_logistic(), method=method, cv=5)
    report(f"CV + {method}", calibrated.fit(X_train, y_train))

# Option 2: a model that is already trained, plus a separate calibration set
X_fit, X_cal, y_fit, y_cal = train_test_split(
    X_train, y_train, test_size=0.3, random_state=42, stratify=y_train)
trained = balanced_logistic().fit(X_fit, y_fit)
frozen = CalibratedClassifierCV(FrozenEstimator(trained), method="sigmoid").fit(X_cal, y_cal)
report("Frozen + sigmoid", frozen)

Balanced, raw          Brier 0.1738  ROC-AUC 0.816
CV + sigmoid           Brier 0.1048  ROC-AUC 0.816
CV + isotonic          Brier 0.1023  ROC-AUC 0.816
Frozen + sigmoid       Brier 0.1053  ROC-AUC 0.815

Both calibration methods cut the Brier score from 0.174 to about 0.10, close to the plain logistic model, while ROC-AUC stays at 0.816. That is typical: calibration fixes what the probabilities mean without changing the order of the customers, so ranking metrics barely move. The frozen model with its own calibration set does about as well, even though it trained on fewer rows.

From probability to a decision

Now the business part. A Jaipur broadband company offers a ₹500 discount to customers it thinks will leave. Losing a customer costs about ₹4,000 in future profit. Assume, to keep the maths simple, that the offer always keeps a customer who was about to leave.

For one customer with churn probability p:

• Call and offer: you always pay ₹500.
• Do nothing: you lose ₹4,000 with probability p, so the expected loss is p × 4,000.

Offer the discount when p × 4,000 > 500, that is when p > 500 / 4,000 = 0.125. In general, with a cost C_FP for acting unnecessarily and C_FN for a missed case, the best threshold is C_FP / (C_FP + C_FN) when correct actions cost nothing. Here the offer is paid on correct calls too, so the threshold becomes 500 / 4,000. The default 0.5 is right only when both mistakes cost the same.

This formula assumes the probabilities are calibrated. Feed it an overconfident model and you act on the wrong customers:

import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=8000, n_features=10, n_informative=4,
                           n_redundant=4, weights=[0.85, 0.15], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y)

OFFER, LOST = 500, 4000
def total_cost(y_true, act):
    missed = (y_true == 1) & ~act
    return act.sum() * OFFER + missed.sum() * LOST

balanced = make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced"))
raw = balanced.fit(X_train, y_train)
cal = CalibratedClassifierCV(balanced, method="isotonic", cv=5).fit(X_train, y_train)
threshold = OFFER / LOST
print(f"Do nothing for anyone:  ₹{total_cost(y_test, np.zeros(len(y_test), bool)):,}")
for name, model in [("raw balanced", raw), ("calibrated", cal)]:
    p = model.predict_proba(X_test)[:, 1]
    for t in [0.5, threshold]:
        act = p >= t
        print(f"{name:<16} threshold {t:<5} calls {act.sum():>4}  cost ₹{total_cost(y_test, act):,}")

Do nothing for anyone:  ₹1,952,000
raw balanced     threshold 0.5   calls  995  cost ₹989,500
raw balanced     threshold 0.125 calls 2780  cost ₹1,410,000
calibrated       threshold 0.5   calls  195  cost ₹1,637,500
calibrated       threshold 0.125 calls 1017  cost ₹988,500

With the raw balanced model, the formula threshold of 0.125 flags 2,780 of the 3,200 test customers and costs ₹14.1 lakh, more than its own 0.5 threshold. Its inflated probabilities push almost everyone over the line. After calibration, the same formula picks 1,017 customers and costs ₹9.89 lakh, the cheapest option in the table. The raw model at 0.5 comes close only by luck: the balanced weights happen to shift its scores so that 0.5 behaves like a lower threshold. A decision rule should not depend on that kind of luck.

Letting cross-validation choose the threshold

The formula needs calibrated probabilities and simple costs. When costs are more complicated, or you would rather not trust the calibration, search for the threshold directly. TunedThresholdClassifierCV (introduced in the Imbalanced Data lesson) accepts any scorer, so you can write your own business metric with make_scorer. It receives true labels and the predicted labels at a candidate threshold and returns the money saved:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import make_scorer
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split

X, y = make_classification(n_samples=8000, n_features=10, n_informative=4,
                           n_redundant=4, weights=[0.85, 0.15], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y)

OFFER, LOST = 500, 4000
def rupees_saved(y_true, y_pred):
    """Money saved compared with doing nothing for anyone."""
    caught = ((y_true == 1) & (y_pred == 1)).sum()
    calls = (y_pred == 1).sum()
    return caught * LOST - calls * OFFER

savings = make_scorer(rupees_saved)
base = HistGradientBoostingClassifier(random_state=42)
tuned = TunedThresholdClassifierCV(base, scoring=savings, cv=5).fit(X_train, y_train)
base.fit(X_train, y_train)

print(f"Threshold chosen by CV: {tuned.best_threshold_:.3f}")
for name, model in [("default 0.5", base), ("tuned", tuned)]:
    saved = rupees_saved(y_test, model.predict(X_test))
    print(f"{name:<12} saves ₹{saved:,} on the test customers")

Threshold chosen by CV: 0.051
default 0.5  saves ₹1,410,000 on the test customers
tuned        saves ₹1,451,000 on the test customers

Cross-validation picks a threshold of 0.051, far below 0.5, because a missed churner costs eight times more than an offer. On the test customers it saves ₹14.51 lakh against ₹14.10 lakh at the default threshold: an extra ₹41,000 from changing one number. The tuned threshold is also below the formula's 0.125. Boosted trees are not perfectly calibrated, and the search adapts to the probabilities this model really produces.

Common mistakes

• Treating predict_proba as true probabilities without checking. Plot a calibration curve on held-out data first.
• Calibrating on the training data. Use cv=5 or a separate calibration set with FrozenEstimator.
• Using isotonic calibration on a few hundred rows. It overfits; prefer "sigmoid".
• Keeping the default 0.5 threshold when a missed case costs eight times more than a false alarm.
• Choosing the threshold on the test set. Use cross-validation or a validation set, then test once.

Key points

• A calibrated model's probability of 0.8 means the event happens about 80% of the time.
• Check calibration with calibration_curve or CalibrationDisplay, and summarise it with the Brier score or log loss.
• CalibratedClassifierCV fixes calibration with "sigmoid", "isotonic" or "temperature"; it rarely changes ROC-AUC.
• With calibrated probabilities, act when p is above C_FP / (C_FP + C_FN) (adjusted for what correct actions cost).
• TunedThresholdClassifierCV with a make_scorer business metric picks the threshold that saves the most money.

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import make_scorer
from sklearn.model_selection import (FixedThresholdClassifier, TunedThresholdClassifierCV,
                                     train_test_split)

X, y = make_classification(n_samples=8000, n_features=10, n_informative=4,
                           n_redundant=4, weights=[0.85, 0.15], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y)

OFFER, LOST = 500, 10_000
def rupees_saved(y_true, y_pred):
    caught = ((y_true == 1) & (y_pred == 1)).sum()
    return caught * LOST - (y_pred == 1).sum() * OFFER

base = HistGradientBoostingClassifier(random_state=42)
formula = FixedThresholdClassifier(base, threshold=OFFER / LOST).fit(X_train, y_train)
tuned = TunedThresholdClassifierCV(base, scoring=make_scorer(rupees_saved), cv=5).fit(X_train, y_train)

print(f"Formula threshold: {OFFER / LOST:.3f}, tuned threshold: {tuned.best_threshold_:.3f}")
for name, model in [("formula", formula), ("tuned", tuned)]:
    print(f"{name:<8} saves ₹{rupees_saved(y_test, model.predict(X_test)):,}")

Formula threshold: 0.050, tuned threshold: 0.010
formula  saves ₹4,119,500
tuned    saves ₹4,091,500