EthicsComputerCourses
Join training
  1. Tutorials
  2. Machine Learning Tutorial
  3. Building Better Models

Imbalanced Data

Machine Learning · Lesson 23 of 32 7 min read Updated

In many important problems, the class you care about is rare. Perhaps 2 in 100 UPI payments are fraud, 1 in 50 machines fails this month, or 5 in 100 customers cancel. A model trained on such data learns that "almost always the common class" is a safe bet, and it can look excellent by accuracy while missing nearly every case that matters.

In this lesson you will see the problem in action, measure it properly, and fix it with three tools: class weights, threshold tuning and resampling.

The accuracy paradox in action

make_classification creates synthetic data with a chosen class balance. Here, 3% of 5,000 payments are fraud.

PythonRun it yourself
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
                           weights=[0.97, 0.03], class_sep=1.0,
                           n_clusters_per_class=1, flip_y=0, random_state=42)
print(f"Fraud share: {y.mean():.3f}")
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, pred):.3f}")
print(classification_report(y_test, pred, target_names=["genuine", "fraud"], digits=3))
Output
Fraud share: 0.030
Accuracy: 0.984
              precision    recall  f1-score   support

     genuine      0.984     1.000     0.992      1455
       fraud      1.000     0.467     0.636        45

    accuracy                          0.984      1500
   macro avg      0.992     0.733     0.814      1500
weighted avg      0.984     0.984     0.981      1500

Accuracy looks wonderful, but the fraud row tells the real story: the model catches only about half of the fraud cases. This is the accuracy paradox: with imbalanced classes, accuracy mostly measures how common the majority class is. Note stratify=y in the split; without it, a test set could easily end up with very few fraud cases.

Better metrics for rare classes

Judge imbalanced problems with metrics that focus on the rare class (see Classification Metrics):

  • Recall and precision of the rare class, and their F1 score.
  • Balanced accuracy: the average of the recall of each class, so the rare class counts as much as the common one. A model that always predicts "genuine" gets 0.5.
  • Average precision (average_precision_score, the area under the precision-recall curve): a threshold-free score that, unlike ROC-AUC, is sensitive to false alarms when positives are rare. A random model scores about the positive share (0.03 here).

Fix 1: class weights

Most classifiers accept class_weight. With class_weight="balanced", each class gets a weight inversely proportional to its frequency, so every fraud case counts about 32 times as much as a genuine one during training. The model can no longer ignore the rare class cheaply.

PythonRun it yourself
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (average_precision_score, balanced_accuracy_score,
                             f1_score, precision_score, recall_score)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
                           weights=[0.97, 0.03], class_sep=1.0,
                           n_clusters_per_class=1, flip_y=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y)

for name, weight in [("plain", None), ("balanced", "balanced")]:
    model = make_pipeline(StandardScaler(), LogisticRegression(class_weight=weight))
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    proba = model.predict_proba(X_test)[:, 1]
    print(f"{name:<9} recall {recall_score(y_test, pred):.3f}  precision {precision_score(y_test, pred):.3f}  "
          f"F1 {f1_score(y_test, pred):.3f}  balanced acc {balanced_accuracy_score(y_test, pred):.3f}  "
          f"avg precision {average_precision_score(y_test, proba):.3f}")
Output
plain     recall 0.467  precision 1.000  F1 0.636  balanced acc 0.733  avg precision 0.710
balanced  recall 0.800  precision 0.113  F1 0.198  balanced acc 0.803  avg precision 0.432

With balanced weights, recall jumps: the model now catches most fraud cases. But precision collapses, so most alarms are now false, and average precision, which judges the ranking of payments at every threshold, actually gets worse. Class weights are a blunt tool that changes the whole fit. Often a gentler fix works better: keep the model and move its decision threshold.

Fix 2: tune the decision threshold

A classifier's predict uses a probability threshold of 0.5. For a rare class, a lower threshold is usually better. You could pick one by hand, but choosing it on the test set would be cheating. TunedThresholdClassifierCV (added in scikit-learn 1.5) finds the threshold that maximises a metric of your choice using cross-validation on the training data. FixedThresholdClassifier simply applies a threshold you already decided.

PythonRun it yourself
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_score, recall_score
from sklearn.model_selection import (FixedThresholdClassifier, TunedThresholdClassifierCV,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
                           weights=[0.97, 0.03], class_sep=1.0,
                           n_clusters_per_class=1, flip_y=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y)
base = make_pipeline(StandardScaler(), LogisticRegression())

tuned = TunedThresholdClassifierCV(base, scoring="f1", cv=5).fit(X_train, y_train)
fixed = FixedThresholdClassifier(base, threshold=0.1).fit(X_train, y_train)
print(f"Threshold chosen by cross-validation: {tuned.best_threshold_:.3f}")

for name, model in [("default 0.5", base.fit(X_train, y_train)), ("tuned for F1", tuned), ("fixed 0.1", fixed)]:
    pred = model.predict(X_test)
    print(f"{name:<13} recall {recall_score(y_test, pred):.3f}  "
          f"precision {precision_score(y_test, pred):.3f}  F1 {f1_score(y_test, pred):.3f}")
Output
Threshold chosen by cross-validation: 0.172
default 0.5   recall 0.467  precision 1.000  F1 0.636
tuned for F1  recall 0.600  precision 0.900  F1 0.720
fixed 0.1     recall 0.689  precision 0.585  F1 0.633

Threshold tuning lets you pick the balance between catching fraud and raising alarms on purpose. Maximising F1 is one choice; in a real bank, you might instead require at least 80% recall, or minimise a money cost, as in the exercise below.

Fix 3: resampling

Resampling changes the training data itself. Oversampling repeats rare-class rows (drawn with replacement) until the classes are balanced; undersampling throws away majority rows. Resample the training set only. The test set must keep the real-world balance, or your scores will be fiction.

PythonRun it yourself
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.utils import resample

X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
                           weights=[0.97, 0.03], class_sep=1.0,
                           n_clusters_per_class=1, flip_y=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y)

X_fraud, X_ok = X_train[y_train == 1], X_train[y_train == 0]
X_fraud_up = resample(X_fraud, replace=True, n_samples=len(X_ok), random_state=42)
X_bal = np.vstack([X_ok, X_fraud_up])
y_bal = np.array([0] * len(X_ok) + [1] * len(X_fraud_up))
print("Training rows before:", np.bincount(y_train), " after:", np.bincount(y_bal))

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_bal, y_bal)
pred = model.predict(X_test)   # the test set keeps its real 3% fraud
print(f"recall {recall_score(y_test, pred):.3f}  precision {precision_score(y_test, pred):.3f}")
Output
Training rows before: [3395  105]  after: [3395 3395]
recall 0.800  precision 0.114

Random oversampling has an effect similar to class weights. The popular imbalanced-learn library adds smarter methods such as SMOTE, which creates new synthetic minority samples between existing ones, and its own pipeline that resamples only during fit. It is not available in the browser, but on your computer it looks like this:

Python
# python -m pip install imbalanced-learn
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as make_imb_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

model = make_imb_pipeline(StandardScaler(), SMOTE(random_state=42), LogisticRegression())
# model.fit(X_train, y_train) resamples the training folds only

Which fix should you use?

Start with the simplest: stratified splits, the right metrics, then class_weight="balanced" or threshold tuning. They need no extra data handling and work inside pipelines and cross-validation. Try resampling when those are not enough, and always compare options with cross-validated scores on data that keeps the real class balance. Above all, choose the threshold from the real costs: a missed fraud may cost ₹5,000, while a false alarm costs a ₹100 phone call.

Common mistakes

  • Reporting accuracy. Report recall, precision, F1, balanced accuracy or average precision for the rare class.
  • Resampling before splitting. Copies of the same fraud row then appear in both training and test sets, which is leakage. Resample only the training data, inside cross-validation.
  • Choosing the threshold on the test set. Use TunedThresholdClassifierCV or a validation set.
  • Splitting without stratify. A small or unlucky test set may contain almost no positive cases.

Key points

  • With imbalanced classes, a model can have high accuracy and still miss most rare cases.
  • Use stratified splits and metrics focused on the rare class, such as recall, F1, balanced accuracy and average precision.
  • class_weight="balanced" makes rare-class mistakes more costly during training.
  • TunedThresholdClassifierCV picks the decision threshold with cross-validation; FixedThresholdClassifier applies a chosen one.
  • Resample only the training data; imbalanced-learn offers SMOTE and leak-free resampling pipelines.

Exercise

A bank says a missed fraud costs ₹5,000 and a false alarm costs ₹100. Using the plain logistic regression pipeline on the same data, compute predict_proba on the test set and, for thresholds 0.02, 0.05, 0.1, 0.2, 0.3 and 0.5, print the number of missed frauds, the number of false alarms and the total cost. Which threshold is cheapest? (In a real project you would choose it on validation data, not the test set.)

Show solution
Python
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
                           weights=[0.97, 0.03], class_sep=1.0,
                           n_clusters_per_class=1, flip_y=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]

for threshold in [0.02, 0.05, 0.1, 0.2, 0.3, 0.5]:
    flagged = proba >= threshold
    missed = int(((y_test == 1) & ~flagged).sum())
    alarms = int(((y_test == 0) & flagged).sum())
    print(f"threshold {threshold:<4}  missed {missed:>2}  false alarms {alarms:>3}  "
          f"cost ₹{missed * 5000 + alarms * 100:,}")
Output
threshold 0.02  missed  8  false alarms 349  cost ₹74,900
threshold 0.05  missed 11  false alarms  97  cost ₹64,700
threshold 0.1   missed 14  false alarms  22  cost ₹72,200
threshold 0.2   missed 18  false alarms   2  cost ₹90,200
threshold 0.3   missed 21  false alarms   1  cost ₹105,100
threshold 0.5   missed 24  false alarms   0  cost ₹120,000

A threshold of 0.05 is cheapest here: it misses 11 frauds and raises 97 false alarms, for ₹64,700. Because a missed fraud costs 50 times more than a false alarm, the best threshold sits far below the default 0.5, which would cost ₹1.2 lakh by missing 24 frauds.

Quick Quiz

1. A fraud model has 97% accuracy on data with 3% fraud. What can you conclude?

2. What does class_weight="balanced" do?

3. What does TunedThresholdClassifierCV choose?

4. When should you resample the data?

Learn Machine Learning with a mentor

Free lessons get you started. Our Data Science & Machine Learning training adds real projects and code reviews from working developers.

  • Project-based learning
  • Mentor code reviews
  • Free counselling call
Get course details +91 90532 10052