Framing ML Problems & Baselines

Most machine learning projects that fail at work do not fail because of the algorithm. They fail because the team answered the wrong question, measured the wrong thing, or never checked whether a simple rule would have done just as well. Experienced ML engineers spend their first days on framing, not on models.

In this lesson you will learn to turn a vague business request into a precise ML task, pick a metric that reflects real costs, and build baselines: the simple models every real model must beat.

From a business question to an ML task

A manager in a Jaipur broadband company says: "Use AI to reduce customer churn." That is a goal, not a task. Before writing code, answer these questions in writing:

• What decision will the prediction change? The retention team will call some customers and offer a discount.
• What exactly is the target? "Churned" must be defined: for example, "cancelled or stopped paying within the next 90 days".
• What is one row? One customer on one date (the prediction point).
• When is the prediction made, and what is known then? Every Monday, using only data available on that Monday. Anything recorded later must not be a feature.
• How will success be measured? Fewer customers lost per rupee spent on calls, not "high accuracy".

The prediction point matters more than beginners expect. If you build features from the whole history of each customer, including the weeks after the date you pretend to predict, the model learns from the future. That is data leakage, and it is the most common reason a model that looked brilliant fails in production.

Choose the metric from the costs

A metric is a promise about what "better" means. Pick it from the decision the model supports:

Write the metric down before you start modelling, and agree on it with the person who will use the model. The details of each metric are in Regression Metrics and Classification Metrics.

Baselines: the score to beat

A baseline is the score of the simplest reasonable approach. Without one, a number such as "92% accuracy" means nothing: if 92% of customers never churn, a model that always says "stays" also scores 92%. scikit-learn's DummyClassifier and DummyRegressor give you these trivial models with the usual fit/predict API.

DummyClassifier strategies: "most_frequent" always predicts the commonest class, "prior" (the default) does the same but returns the class shares as probabilities, "stratified" guesses randomly in proportion to the class shares, and "uniform" guesses each class equally often.

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# 3,000 broadband customers, about 10% churn
X, y = make_classification(n_samples=3000, n_features=8, n_informative=4,
                           weights=[0.9, 0.1], random_state=42)
models = {
    "always 'stays'": DummyClassifier(strategy="most_frequent"),
    "random guess": DummyClassifier(strategy="stratified", random_state=42),
    "logistic": make_pipeline(StandardScaler(), LogisticRegression()),
}
for name, model in models.items():
    s = cross_validate(model, X, y, cv=5, scoring=["accuracy", "balanced_accuracy", "recall"])
    print(f"{name:<15} accuracy {s['test_accuracy'].mean():.3f}  "
          f"balanced {s['test_balanced_accuracy'].mean():.3f}  recall {s['test_recall'].mean():.3f}")

always 'stays'  accuracy 0.898  balanced 0.500  recall 0.000
random guess    accuracy 0.799  balanced 0.480  recall 0.079
logistic        accuracy 0.910  balanced 0.570  recall 0.144

The model that always says "stays" already scores 0.898 accuracy while catching no churner at all. Logistic regression beats it by only about one point of accuracy, and its recall shows the real picture: at the default threshold it finds just 14% of churners. Balanced accuracy (0.570 against 0.500) tells the story far better than accuracy. Without the dummy line next to it, "91% accuracy" would have sounded like success. (Moving the threshold to catch more churners is covered in Calibration & Cost-Based Thresholds.)

A ladder of baselines

Do not stop at the dummy model. Climb a short ladder, and keep each step only if it clearly beats the one below:

1. A dummy model (mean, median or most frequent class).
2. A rule a domain expert would write, or a model using one strong feature.
3. A simple linear model on all features.
4. A stronger model such as gradient boosting.

Here is the ladder for the diabetes dataset, where the target is a measure of disease progression one year later:

import pandas as pd  # as_frame=True needs pandas
from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import KFold, cross_val_score

X, y = load_diabetes(return_X_y=True, as_frame=True)
cv = KFold(n_splits=5, shuffle=True, random_state=42)
ladder = [
    ("predict the mean", DummyRegressor(strategy="mean"), X),
    ("predict the median", DummyRegressor(strategy="median"), X),
    ("bmi only (one feature)", LinearRegression(), X[["bmi"]]),
    ("linear, all features", LinearRegression(), X),
    ("gradient boosting", HistGradientBoostingRegressor(random_state=42), X),
]
for name, model, features in ladder:
    mae = -cross_val_score(model, features, y, cv=cv, scoring="neg_mean_absolute_error").mean()
    print(f"{name:<24} MAE {mae:5.1f}")

predict the mean         MAE  66.0
predict the median       MAE  65.3
bmi only (one feature)   MAE  52.1
linear, all features     MAE  44.3
gradient boosting        MAE  46.7

Predicting the median beats the mean slightly on MAE, as expected, because the median is the constant that minimises absolute error. One feature, bmi, already removes a large part of the error (66.0 down to 52.1). The linear model on all features is the best rung at 44.3, and gradient boosting, the "fancier" model, is slightly worse at 46.7. On 442 patients the simple model wins. That is a common and useful result: you would ship the linear model, which is also easier to explain and cheaper to run.

When you can only act on the top few

Real teams have limited capacity. If the retention team can call only 150 customers a week, what matters is how many true churners are in the model's top 150, not the score over everyone. This is precision at k. Compare it with the churn rate you would get by calling 150 customers at random: the ratio is called lift.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=3000, n_features=8, n_informative=4,
                           weights=[0.9, 0.1], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)

risk = model.predict_proba(X_test)[:, 1]
k = 150
top_k = np.argsort(risk)[::-1][:k]          # the 150 highest-risk customers
precision_at_k = y_test[top_k].mean()
base_rate = y_test.mean()
print(f"Customers in test set: {len(y_test)}, churners: {y_test.sum()}")
print(f"Churn rate if we call {k} at random: {base_rate:.1%}")
print(f"Churn rate among the model's top {k}: {precision_at_k:.1%}")
print(f"Lift: {precision_at_k / base_rate:.1f}x")

Customers in test set: 1500, churners: 152
Churn rate if we call 150 at random: 10.1%
Churn rate among the model's top 150: 39.3%
Lift: 3.9x

Calling 150 customers at random would reach about 15 churners (10.1%). The model's top 150 contain about 59 (39.3%), a lift of 3.9x. A retention manager understands that number immediately, even though the same model's recall at the default threshold looked weak earlier. A model's ranking can be valuable long before its yes/no predictions are.

Write a one-page problem statement

Before modelling, write a short statement that anyone in the company can check. It keeps the project honest when results come in.

Problem:      Reduce broadband churn in Jaipur and Ajmer.
Decision:     Each Monday, the retention team calls the 150 highest-risk customers.
Target:       Customer cancels or stops paying within 90 days of the Monday.
Features:     Only data known on that Monday (tenure, plan, bills, complaints, outages).
Excluded:     Name, phone number, anything recorded after the Monday.
Metric:       Precision at 150 on the most recent 3 months (held out by time).
Baseline:     Rule "2+ complaints in 30 days" and a DummyClassifier.
Success:      Lift of 3x or more over random calls, checked again after one month live.
Risks:        Seasonal churn around exams and moves; fairness across cities.

Common mistakes

• Starting with the model. Define the target, prediction point and metric first.
• Reporting accuracy on imbalanced data without the "always predict the majority class" baseline next to it.
• Using features that are not known at prediction time, such as a field filled in after the customer cancelled.
• Comparing models on different splits. Use the same cross-validation splitter for every rung of the ladder.
• Optimising a metric nobody cares about. If the team can act on 150 customers, measure precision at 150.

Key points

• Turn every request into a precise task: decision, target, one row, prediction point and metric.
• Only use information that is available at the moment the prediction is made.
• Choose the metric from the real costs and capacity, and agree on it before modelling.
• DummyClassifier and DummyRegressor give the trivial score every model must beat.
• Climb a ladder of baselines (dummy, expert rule, linear, boosting) and keep complexity only when it pays.
• Precision at k and lift measure models used by teams with limited capacity.

import pandas as pd  # as_frame=True needs pandas
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
ladder = [
    ("most frequent", DummyClassifier(strategy="most_frequent"), X),
    ("mean radius only", LogisticRegression(), X[["mean radius"]]),
    ("all 30 features", make_pipeline(StandardScaler(), LogisticRegression()), X),
]
for name, model, features in ladder:
    s = cross_validate(model, features, y, cv=cv, scoring=["accuracy", "recall"])
    print(f"{name:<17} accuracy {s['test_accuracy'].mean():.3f}  recall {s['test_recall'].mean():.3f}")

most frequent     accuracy 0.627  recall 1.000
mean radius only  accuracy 0.877  recall 0.927
all 30 features   accuracy 0.974  recall 0.992