Project: Customer Churn Prediction
In this project you will work through a churn problem the way a data scientist in an Indian telecom, bank or subscription company would: from a vague business request to a saved model with a decision rule, an explanation and a written model card. It pulls together the lessons on framing and baselines, data leakage, pipelines, gradient boosting, calibration and thresholds and interpretation.
The brief. A fibre broadband provider in Rajasthan is losing customers. The head of customer experience says: "Tell us who is about to leave, so our retention team can call them with an offer." The data team has a snapshot of 5,000 customers taken on one date, with what was known about each customer on that date, and a label saying whether the customer left within the next 90 days.
The dataset is synthetic and built by the make_customers() function, which every step repeats so that each example runs on its own in the browser. Its patterns are realistic, and it hides one trap for you to find.
Step 1: Frame the problem
Write the decisions down before touching code:
- Decision: each month, the retention team calls customers and offers ₹400 off the next bill.
- Target:
churned= 1 if the customer cancelled or stopped paying within 90 days of the snapshot. - One row: one customer on the snapshot date. Features must be known on that date.
- Money: a customer who leaves costs about ₹6,000 of future profit. Past campaigns kept about half of the churners who received the offer. So calling a real churner is worth about 0.5 × ₹6,000 − ₹400 = ₹2,600, and calling a customer who would have stayed costs ₹400.
- Metric: rupees saved compared with calling nobody; for model comparison before choosing a threshold, average precision, which suits a rare positive class.
- Capacity: the team can make at most 400 calls a month.
Step 2: Load and explore
import numpy as np
import pandas as pd
def make_customers(n=5000, seed=42):
"""Synthetic broadband customers; churned = left within 90 days of the snapshot."""
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
df = make_customers()
print("Shape:", df.shape)
print(f"Churn rate: {df['churned'].mean():.1%}")
print("Missing values:", df.isna().sum()[df.isna().sum() > 0].to_dict())
print(df.groupby("contract", observed=True)["churned"].mean().round(3).to_dict())
print(df.groupby("payment", observed=True)["churned"].mean().round(3).to_dict())
print(df.groupby("churned")[["tenure_months", "complaints_90d", "outage_hours_90d"]].mean().round(1))Shape: (5000, 12)
Churn rate: 18.5%
Missing values: {'avg_daily_gb': 146}
{'Annual': 0.099, 'Monthly': 0.232}
{'Card': 0.15, 'Cash': 0.235, 'UPI autopay': 0.167}
tenure_months complaints_90d outage_hours_90d
churned
0 40.1 0.5 5.7
1 23.6 0.9 7.6About 18.5% of customers churned within 90 days. Monthly-contract customers churn more than twice as often as annual ones (23.2% against 9.9%), cash payers more often than UPI autopay users, and churners have shorter tenure (23.6 against 40.1 months) and more complaints. Only avg_daily_gb has gaps (146 rows). Nothing here is surprising, and that is good: exploration should first confirm what the business already knows before you trust anything subtler.
Step 3: Hunt for leaks
Before modelling, score every numeric column on its own, as in Data Leakage in Depth, and ask how each column is recorded.
import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score
def make_customers(n=5000, seed=42):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
df = make_customers()
numeric = df.select_dtypes("number").columns.drop("churned")
for col in numeric:
values = df[col].fillna(df[col].median())
auc = roc_auc_score(df["churned"], values)
flag = " <-- check how this is recorded" if max(auc, 1 - auc) > 0.85 else ""
print(f"{col:<19} {max(auc, 1 - auc):.3f}{flag}")tenure_months 0.726 complaints_90d 0.602 outage_hours_90d 0.599 late_payments_12m 0.576 avg_daily_gb 0.560 calls_after_notice 0.957 <-- check how this is recorded
calls_after_notice scores 0.957 on its own, far above everything else. A conversation with the call-centre team explains why: these are calls logged after a customer has given notice to cancel. The column exists only because the customer is already leaving, which is target leakage, so it must go. With it, any model would look brilliant in testing and be useless on the snapshot date, when nobody has given notice yet.
The model will use the remaining columns except customer_id, an identifier that carries no information about behaviour (a model could only memorise it). From here on, X = df.drop(columns=["customer_id", "churned", "calls_after_notice"]).
Step 4: Baselines and candidate models
Split once into training and test sets (stratified, 25% test), and compare candidates with 5-fold cross-validation on the training set only. The ladder: a dummy model, a business rule ("monthly contract and at least two complaints") turned into a score, a logistic regression pipeline, and HistGradientBoostingClassifier, which handles the category columns and the missing avg_daily_gb values natively.
import numpy as np
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
def make_customers(n=5000, seed=42):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
df = make_customers()
X = df.drop(columns=["customer_id", "churned", "calls_after_notice"])
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
rule = (X_train["contract"] == "Monthly").astype(int) + (X_train["complaints_90d"] >= 2).astype(int)
print(f"{'business rule':<14} ROC-AUC {roc_auc_score(y_train, rule):.3f} "
f"avg precision {average_precision_score(y_train, rule):.3f}")
prep = make_column_transformer(
(OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include="category")),
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
make_column_selector(dtype_include="number")))
models = {
"dummy": DummyClassifier(),
"logistic": make_pipeline(prep, LogisticRegression(max_iter=1000)),
"boosting": HistGradientBoostingClassifier(learning_rate=0.05, max_iter=300, max_leaf_nodes=15,
early_stopping=True, random_state=42),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for name, model in models.items():
s = cross_validate(model, X_train, y_train, cv=cv, scoring=["roc_auc", "average_precision"])
print(f"{name:<14} ROC-AUC {s['test_roc_auc'].mean():.3f} "
f"avg precision {s['test_average_precision'].mean():.3f}")business rule ROC-AUC 0.641 avg precision 0.260 dummy ROC-AUC 0.500 avg precision 0.185 logistic ROC-AUC 0.823 avg precision 0.541 boosting ROC-AUC 0.807 avg precision 0.510
The dummy's average precision equals the churn rate (0.185), and the business rule reaches only 0.260. Both learned models are far better, and logistic regression is slightly ahead of boosting (0.541 against 0.510). Boosting is not automatically the winner: here the relationships are mostly smooth and additive, which a linear model captures with fewer moving parts. We choose the logistic pipeline, which is also easier to explain to the retention team. From now on, each step builds it with a small make_model() function.
Step 5: From scores to a rupee decision
Ranking is not a decision. Turn the probabilities into "call" or "do not call" with the business numbers from Step 1. The scorer below computes rupees saved compared with calling nobody, and TunedThresholdClassifierCV finds the threshold that maximises it by cross-validation on the training data. For comparison, the formula from Calibration & Cost-Based Thresholds suggests calling when p × 2,600 is greater than (1 − p) × 400, that is when p > 400 / 3,000 ≈ 0.133, provided the probabilities are calibrated. A quick calibration check comes first.
import numpy as np
import pandas as pd
from sklearn.calibration import calibration_curve
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
def make_customers(n=5000, seed=42):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
def make_model():
prep = make_column_transformer(
(OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include="category")),
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
make_column_selector(dtype_include="number")))
return make_pipeline(prep, LogisticRegression(max_iter=1000))
df = make_customers()
X = df.drop(columns=["customer_id", "churned", "calls_after_notice"])
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
VALUE_OF_SAVE, OFFER = 2600, 400 # net value of calling a churner; cost of calling a stayer
def rupees_saved(y_true, y_pred):
called = y_pred == 1
return int(((y_true == 1) & called).sum() * VALUE_OF_SAVE - ((y_true == 0) & called).sum() * OFFER)
tuned = TunedThresholdClassifierCV(make_model(), scoring=make_scorer(rupees_saved), cv=5).fit(X_train, y_train)
model = make_model().fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
observed, predicted = calibration_curve(y_test, proba, n_bins=5, strategy="quantile")
print("Calibration predicted:", predicted.round(2), " observed:", observed.round(2))
print(f"Tuned threshold: {tuned.best_threshold_:.3f}")
for name, threshold in [("default 0.5", 0.5), ("formula 0.133", 0.133), ("tuned", tuned.best_threshold_)]:
call = (proba >= threshold).astype(int)
print(f"{name:<14} calls {call.sum():>4} rupees saved ₹{rupees_saved(y_test, call):,}")Calibration predicted: [0.02 0.06 0.12 0.23 0.51] observed: [0.02 0.05 0.16 0.22 0.48] Tuned threshold: 0.135 default 0.5 calls 107 rupees saved ₹164,200 formula 0.133 calls 571 rupees saved ₹335,600 tuned calls 569 rupees saved ₹336,400
The calibration check is reassuring: in every risk band, predicted and observed churn rates are close (0.51 against 0.48 in the top band), so the formula can be trusted, and indeed cross-validation picks 0.135, almost exactly the formula's 0.133. The default 0.5 calls only 107 customers and saves ₹1.64 lakh. The rupee-based threshold calls about 570 and saves about ₹3.36 lakh on the 1,250 test customers: double the money from the same model, just by choosing the threshold from the business numbers.
Step 6: Explain and check segments
Before anyone acts on the model, check what it relies on and whether it works equally well for different groups of customers. Permutation importance (scored with average precision on the test set) shows the drivers. Then compare recall and precision for each city at the chosen threshold: a model that misses most churners in Kota while doing well in Jaipur would make the retention budget unfair and less useful.
import numpy as np
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.inspection import permutation_importance
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
def make_customers(n=5000, seed=42):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
def make_model():
prep = make_column_transformer(
(OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include="category")),
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
make_column_selector(dtype_include="number")))
return make_pipeline(prep, LogisticRegression(max_iter=1000))
df = make_customers()
X = df.drop(columns=["customer_id", "churned", "calls_after_notice"])
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
model = make_model().fit(X_train, y_train)
result = permutation_importance(model, X_test, y_test, scoring="average_precision",
n_repeats=10, random_state=42)
importance = pd.Series(result.importances_mean, index=X.columns).sort_values(ascending=False)
print("Drop in average precision when shuffled:")
print(importance.round(3).to_string())
THRESHOLD = 0.133
call = model.predict_proba(X_test)[:, 1] >= THRESHOLD
report = pd.DataFrame({"city": X_test["city"], "churned": y_test, "call": call})
rows = []
for city, part in report.groupby("city", observed=True):
rows.append({"city": city, "customers": len(part), "churn rate": round(part["churned"].mean(), 3),
"recall": round(recall_score(part["churned"], part["call"]), 3),
"precision": round(precision_score(part["churned"], part["call"]), 3)})
print(pd.DataFrame(rows).to_string(index=False))Drop in average precision when shuffled: tenure_months 0.218 contract 0.090 complaints_90d 0.073 late_payments_12m 0.058 outage_hours_90d 0.046 payment 0.030 avg_daily_gb 0.020 plan -0.000 city -0.002 city customers churn rate recall precision Ajmer 132 0.167 0.909 0.323 Jaipur 496 0.185 0.793 0.319 Jodhpur 253 0.202 0.745 0.380 Kota 186 0.156 0.862 0.312 Udaipur 183 0.202 0.865 0.320
Tenure is by far the strongest driver, followed by contract type and recent complaints, which matches what the business believes: a good sign. plan and city add nothing. Across cities, recall stays between 0.75 and 0.91 and precision between 0.31 and 0.38. With only 132 test customers in Ajmer, differences of a few points are noise, so the model looks consistent across cities. Record these numbers in the model card and check them again after deployment.
Step 7: Respect capacity, save and use the model
The team can make at most 400 calls a month. If more customers pass the threshold, call the highest-risk ones first. Train the final model on all the training data, check it once on the test set, save it with joblib (see Saving & Loading Models), and use it the way the monthly job will: load, score the new snapshot, sort, cut at the threshold and the capacity.
import joblib
import numpy as np
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
def make_customers(n=5000, seed=42):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
def make_model():
prep = make_column_transformer(
(OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include="category")),
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
make_column_selector(dtype_include="number")))
return make_pipeline(prep, LogisticRegression(max_iter=1000))
df = make_customers()
FEATURES = ["city", "plan", "contract", "payment", "tenure_months", "complaints_90d",
"outage_hours_90d", "late_payments_12m", "avg_daily_gb"]
X_train, X_test, y_train, y_test = train_test_split(
df[FEATURES], df["churned"], test_size=0.25, random_state=42, stratify=df["churned"])
model = make_model().fit(X_train, y_train)
print(f"Final test average precision: {average_precision_score(y_test, model.predict_proba(X_test)[:, 1]):.3f}")
joblib.dump({"model": model, "features": FEATURES, "threshold": 0.133, "capacity": 400,
"version": "churn-2026-11"}, "churn_model.joblib")
# The monthly job: load, score next month's snapshot, choose whom to call
bundle = joblib.load("churn_model.joblib")
snapshot = make_customers(n=3000, seed=7) # stands in for next month's customers
risk = bundle["model"].predict_proba(snapshot[bundle["features"]])[:, 1]
scored = snapshot.assign(risk=risk).sort_values("risk", ascending=False)
call_list = scored[scored["risk"] >= bundle["threshold"]].head(bundle["capacity"])
print(f"Customers above threshold: {(scored['risk'] >= bundle['threshold']).sum()}, calling {len(call_list)}")
print(f"Churn rate in call list: {call_list['churned'].mean():.1%} (all customers: {snapshot['churned'].mean():.1%})")
print(call_list[["customer_id", "city", "contract", "complaints_90d", "risk"]].head(4).round(3).to_string(index=False))Final test average precision: 0.538 Customers above threshold: 1360, calling 400 Churn rate in call list: 54.8% (all customers: 18.5%) customer_id city contract complaints_90d risk RJ100338 Jodhpur Monthly 6 0.990 RJ100497 Jaipur Monthly 4 0.972 RJ100333 Jaipur Monthly 1 0.939 RJ101535 Ajmer Monthly 1 0.911
The test score (0.538 average precision) matches the cross-validated 0.541, so our choices did not overfit. On the new snapshot, 1,360 customers pass the threshold, far more than the team can call, so the capacity rule keeps the 400 riskiest. Of those, 54.8% really churn, about three times the 18.5% base rate. (We can check this only because the data is synthetic; in real life you learn the outcome 90 days later.) The top of the list looks as expected: monthly contracts and several complaints.
Step 8: Write the model card
A model nobody can audit becomes a liability. Store a short model card next to the saved file:
Model: churn-2026-11, logistic regression pipeline (scikit-learn 1.8):
one-hot categories, median-imputed and scaled numbers
Purpose: Rank broadband customers by 90-day churn risk for retention calls.
Not for pricing, credit or any decision that penalises a customer.
Data: 5,000-customer snapshot; features known on the snapshot date only.
Removed: customer_id (identifier), calls_after_notice (recorded after the
cancellation notice: target leakage).
Performance: Test average precision and rupees saved recorded in Step 7;
cross-validated comparison with a business rule and logistic regression in Step 4.
Decision: Call if risk >= 0.133, highest risk first, at most 400 calls a month.
Segments: Recall and precision by city checked in Step 6; re-check every quarter.
Monitoring: Monthly drift report on inputs; actual churn of called and not-called
customers; retrain when rupees saved falls for two months in a row.
Owner: Data science team; questions to the customer experience head.Where to go next
- Add time-based validation: train on older snapshots and test on the latest, as in Time Series Forecasting with ML.
- Put the monthly scoring job behind the checks and drift monitoring from MLOps Basics.
- Model uplift (the change in churn caused by a call) once you have results from the control group.
Common mistakes
- Starting with the model before agreeing on the decision, costs and capacity.
- Keeping a column because it is "highly predictive" without asking when it is recorded.
- Reporting accuracy for an 18% positive class; use average precision and rupees saved.
- Using the default 0.5 threshold when the costs of the two mistakes are very different.
- Skipping segment checks, which can hide a model that fails for one city or plan.
- Claiming the model saved money without a control group.
Key points
- Frame the decision, target, prediction point, money and capacity before writing code.
- Screen every column for target leakage; a column recorded after the outcome must go.
- Compare a dummy, a business rule, a linear model and gradient boosting with the same cross-validation.
- Turn probabilities into actions with a threshold chosen from rupee costs, then apply the capacity limit.
- Explain the model and check its performance per segment before anyone acts on it.
- Save the model with its features, threshold and version, and document it in a model card.
Exercise
The finance team revises the numbers: the offer now costs ₹250, and the value of keeping a churner rises to ₹8,000 (still with a 50% success rate, so calling a churner is worth ₹3,750 net). Recompute the formula threshold, retune the threshold with TunedThresholdClassifierCV and the updated scorer, and print the number of calls and the rupees saved on the test set for both thresholds.
Show solution
import numpy as np
import pandas as pd
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
def make_customers(n=5000, seed=42):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"customer_id": [f"RJ{100000 + i}" for i in range(n)],
"city": rng.choice(["Jaipur", "Jodhpur", "Udaipur", "Kota", "Ajmer"], n, p=[0.4, 0.2, 0.15, 0.15, 0.1]),
"plan": rng.choice(["Basic", "Standard", "Premium"], n, p=[0.45, 0.4, 0.15]),
"contract": rng.choice(["Monthly", "Annual"], n, p=[0.65, 0.35]),
"payment": rng.choice(["UPI autopay", "Card", "Cash"], n, p=[0.5, 0.2, 0.3]),
"tenure_months": rng.integers(1, 73, n),
"complaints_90d": rng.poisson(0.6, n),
"outage_hours_90d": rng.gamma(1.5, 4, n).round(1),
"late_payments_12m": rng.poisson(0.5, n),
"avg_daily_gb": rng.gamma(2, 2.5, n).round(1)})
logit = (-1.9 + 1.3 * (df["contract"] == "Monthly") - 0.05 * df["tenure_months"]
+ 0.7 * df["complaints_90d"] + 0.07 * df["outage_hours_90d"] + 0.7 * (df["payment"] == "Cash")
+ 0.5 * df["late_payments_12m"] - 0.1 * df["avg_daily_gb"]
+ 0.8 * ((df["plan"] == "Basic") & (df["outage_hours_90d"] > 10)))
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
df["calls_after_notice"] = np.where(df["churned"] == 1, rng.poisson(2.5, n), 0)
df.loc[rng.random(n) < 0.03, "avg_daily_gb"] = np.nan
return df.astype({c: "category" for c in ["city", "plan", "contract", "payment"]})
def make_model():
prep = make_column_transformer(
(OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include="category")),
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
make_column_selector(dtype_include="number")))
return make_pipeline(prep, LogisticRegression(max_iter=1000))
df = make_customers()
X = df.drop(columns=["customer_id", "churned", "calls_after_notice"])
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
VALUE_OF_SAVE, OFFER = 0.5 * 8000 - 250, 250
def rupees_saved(y_true, y_pred):
called = y_pred == 1
return int(((y_true == 1) & called).sum() * VALUE_OF_SAVE - ((y_true == 0) & called).sum() * OFFER)
formula = OFFER / (OFFER + VALUE_OF_SAVE)
tuned = TunedThresholdClassifierCV(make_model(), scoring=make_scorer(rupees_saved), cv=5).fit(X_train, y_train)
proba = make_model().fit(X_train, y_train).predict_proba(X_test)[:, 1]
for name, t in [(f"formula {formula:.3f}", formula), (f"tuned {tuned.best_threshold_:.3f}", tuned.best_threshold_)]:
call = (proba >= t).astype(int)
print(f"{name:<14} calls {call.sum():>4} rupees saved ₹{rupees_saved(y_test, call):,}")formula 0.062 calls 843 rupees saved ₹673,250 tuned 0.077 calls 761 rupees saved ₹665,750
The cheaper offer and higher value push the formula threshold down to 0.062, so many more customers are worth calling (843 in the test set). Cross-validation picks a similar 0.077, and both save about ₹6.7 lakh. With only 400 calls a month, capacity now limits the savings even more, and this analysis gives the manager a rupee figure to justify more callers.
Quick Quiz
1. Which column in this project had to be removed as target leakage?
It is recorded only after a customer gives notice, so it exists because of the outcome.
2. Why is average precision used to compare models here instead of accuracy?
With about 18% churners, accuracy is dominated by the customers who stay.
3. Calling a churner is worth ₹2,600 net and calling a stayer costs ₹400. With calibrated probabilities, above which probability should you call?
Call when p × 2,600 > (1 − p) × 400, which gives p > 400 / 3,000.
4. How can the company prove that the retention calls actually reduce churn?
Only a randomised comparison separates the effect of the call from customers who would have stayed anyway.
Learn Machine Learning with a mentor
Free lessons get you started. Our Data Science & Machine Learning training adds real projects and code reviews from working developers.
- Project-based learning
- Mentor code reviews
- Free counselling call