Classification Metrics

"My model is 99% accurate" can be wonderful news or completely meaningless. If only 1 in 100 UPI payments is fraud, a model that never flags anything is 99% accurate and catches no fraud at all. To judge a classifier you must look at which mistakes it makes.

In this lesson you will build a confusion matrix, compute precision, recall and F1 by hand and with scikit-learn, read a classification report, and compare models with the ROC curve and ROC-AUC.

The accuracy trap

Accuracy is the share of predictions that are correct. It is easy to understand, but it hides everything that matters when one class is rare.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score

y = np.array([0] * 990 + [1] * 10)   # 10 fraud payments in 1,000
X = np.zeros((1000, 1))              # features do not matter for this model

lazy = DummyClassifier(strategy="most_frequent").fit(X, y)
pred = lazy.predict(X)
print(f"Accuracy: {accuracy_score(y, pred):.3f}")
print(f"Fraud caught: {recall_score(y, pred):.3f}")

Accuracy: 0.990
Fraud caught: 0.000

A model that always answers "genuine" scores 99% and is useless. You need metrics that look at each class separately. (The full story of rare classes is in Imbalanced Data.)

The confusion matrix

A confusion matrix counts the four possible outcomes of a yes/no prediction. We call the class we care about the positive class (spam, fraud, malignant), even though it is bad news.

From these four counts come the main metrics:

• Precision = TP / (TP + FP): of everything we flagged, how much was really positive? High precision means few false alarms.
• Recall = TP / (TP + FN): of all real positives, how many did we catch? High recall means few missed cases. Recall is also called sensitivity or the true positive rate.
• F1 score = 2 × precision × recall / (precision + recall): one number that is high only when both are high (their harmonic mean).

Here is a spam filter tested on ten SMS messages.

import numpy as np
from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score,
                             precision_score, recall_score)

# 1 = spam, 0 = normal message
actual    = np.array([1, 0, 1, 1, 0, 0, 0, 1, 0, 0])
predicted = np.array([1, 0, 0, 1, 0, 1, 1, 1, 0, 0])

cm = confusion_matrix(actual, predicted)
print(cm)
tn, fp, fn, tp = cm.ravel()
print(f"TN={tn} FP={fp} FN={fn} TP={tp}")

print(f"Precision by hand {tp / (tp + fp):.3f}  sklearn {precision_score(actual, predicted):.3f}")
print(f"Recall    by hand {tp / (tp + fn):.3f}  sklearn {recall_score(actual, predicted):.3f}")
print(f"F1 {f1_score(actual, predicted):.3f}   accuracy {accuracy_score(actual, predicted):.3f}")

[[4 2]
 [1 3]]
TN=4 FP=2 FN=1 TP=3
Precision by hand 0.600  sklearn 0.600
Recall    by hand 0.750  sklearn 0.750
F1 0.667   accuracy 0.700

In scikit-learn's matrix, rows are the actual classes and columns the predicted classes, both in sorted order (0 then 1). cm.ravel() flattens it into tn, fp, fn, tp for a binary problem. The filter caught 3 of 4 spam messages (recall 0.75), but 2 of its 5 spam flags were wrong (precision 0.60).

Which metric matters?

Precision and recall usually pull against each other: flag more and you catch more (recall up) but raise more false alarms (precision down). Pick the metric from the cost of each mistake.

The classification report

classification_report prints precision, recall and F1 for every class at once, plus averages. Support is the number of real samples of each class in the test set.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y  # 1 = malignant
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test),
                            target_names=["benign", "malignant"], digits=3))

              precision    recall  f1-score   support

      benign      0.957     0.989     0.973        90
   malignant      0.980     0.925     0.951        53

    accuracy                          0.965       143
   macro avg      0.968     0.957     0.962       143
weighted avg      0.966     0.965     0.965       143

The macro average treats every class equally; the weighted average weights each class by its support. For problems with many classes, the same averages are available in the metric functions with average="macro" or average="weighted".

ROC curve and ROC-AUC

Precision and recall depend on the threshold you choose (see Logistic Regression). The ROC curve (receiver operating characteristic) shows all thresholds at once: for each one it plots the true positive rate (recall) against the false positive rate, FP / (FP + TN). A perfect model hugs the top-left corner; random guessing follows the diagonal.

ROC-AUC is the area under that curve, from 0.5 (random) to 1.0 (perfect). It has a friendly meaning: the probability that the model gives a randomly chosen positive a higher score than a randomly chosen negative. It needs scores, not hard labels, so pass predict_proba(...)[:, 1] or decision_function(...).

import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import RocCurveDisplay, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

models = {"Logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
          "Tree of depth 1": DecisionTreeClassifier(max_depth=1, random_state=42)}
fig, ax = plt.subplots()
for name, model in models.items():
    scores = model.fit(X_train, y_train).predict_proba(X_test)[:, 1]
    print(f"{name:<20} ROC-AUC {roc_auc_score(y_test, scores):.3f}")
    RocCurveDisplay.from_predictions(y_test, scores, name=name, ax=ax)
ax.plot([0, 1], [0, 1], "k--", label="Random guess")
ax.legend()
plt.show()

Logistic regression  ROC-AUC 0.996
Tree of depth 1      ROC-AUC 0.857

The single-split tree can only give two different scores, so its "curve" is just a couple of straight lines. The logistic regression curve sits almost in the corner. ROC-AUC is a good way to compare models before you settle on a threshold. When positives are very rare, also look at the precision-recall curve and average_precision_score, which show the false-alarm problem more clearly.

Plotting the confusion matrix

ConfusionMatrixDisplay draws the matrix as a coloured grid, which is easier to read in reports.

import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)

ConfusionMatrixDisplay.from_estimator(model, X_test, y_test,
                                      display_labels=["benign", "malignant"], cmap="Blues")
plt.title("Breast cancer test set")
plt.show()

Common mistakes

• Reporting only accuracy on imbalanced data. Always show precision and recall for the rare class.
• Passing predicted labels to roc_auc_score. It needs scores or probabilities; hard 0/1 labels throw away the ranking.
• Getting the positive class wrong. scikit-learn treats label 1 as positive by default. In the breast cancer data, 1 means benign, which is why we flipped the labels.
• Comparing models at different thresholds. Compare with ROC-AUC or at the same threshold.

Key points

• Accuracy can be high while the model misses every rare case.
• The confusion matrix counts TN, FP, FN and TP; in scikit-learn rows are actual and columns predicted.
• Precision = TP / (TP + FP) measures false alarms; recall = TP / (TP + FN) measures missed cases; F1 combines them.
• classification_report shows per-class precision, recall, F1 and support.
• ROC-AUC measures ranking quality across all thresholds, from 0.5 (random) to 1.0 (perfect).

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)).fit(X_train, y_train)
pred = knn.predict(X_test)
print(confusion_matrix(y_test, pred))
print(f"Precision {precision_score(y_test, pred):.3f}")
print(f"Recall    {recall_score(y_test, pred):.3f}")
print(f"F1        {f1_score(y_test, pred):.3f}")
print(f"ROC-AUC   {roc_auc_score(y_test, knn.predict_proba(X_test)[:, 1]):.3f}")

[[89  1]
 [ 5 48]]
Precision 0.980
Recall    0.906
F1        0.941
ROC-AUC   0.986