MLOps Basics

A model in a notebook helps nobody. To create value it must run reliably in a real system: answering requests from an app, scoring yesterday's transactions every night, and staying accurate as the world changes. MLOps (machine learning operations) is the set of practices that gets models into production and keeps them healthy there, much like DevOps does for ordinary software.

In this lesson you will learn the main ideas of MLOps and try small, runnable versions of three everyday tasks: tracking experiments, validating incoming data and detecting data drift.

The machine learning lifecycle

A production model goes round a loop, not a straight line:

1. Data: collect, clean and version the training data.
2. Train: run experiments, record settings and scores, pick the best model.
3. Evaluate: test on held-out data, per group as well as overall (see Responsible AI).
4. Deploy: package the model with its preprocessing and serve it.
5. Monitor: watch input data, predictions, errors and business results.
6. Retrain: when performance drops, train on fresher data and go round again.

Models decay because the world changes. Data drift means the inputs change: a festival season brings different shoppers, or a new UPI app changes payment patterns. Concept drift means the relationship itself changes: fraudsters invent new tricks, so old patterns no longer signal fraud. Monitoring exists to catch both.

Reproducibility and experiment tracking

Six months later, someone will ask: "Which data, code and settings produced the model now in production?" You can only answer if you recorded them. Good habits:

• Keep code in Git, and record the commit used for each training run.
• Version the data: store snapshots, or at least a fingerprint (hash) of the exact data used. Tools such as DVC version large data files alongside Git.
• Pin the environment: record package versions (for example in requirements.txt or a lock file) and set random_state everywhere.
• Track every experiment: parameters, metrics and the resulting model file.

Here is a tiny experiment tracker that writes one JSON line per run, including a fingerprint of the data:

import hashlib
import json
import sklearn
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
data_hash = hashlib.sha256(X.tobytes() + y.tobytes()).hexdigest()[:12]

with open("experiments.jsonl", "w", encoding="utf-8") as log:
    for run, C in enumerate([0.01, 0.1, 1.0], start=1):
        model = make_pipeline(StandardScaler(), LogisticRegression(C=C))
        score = cross_val_score(model, X, y, cv=5).mean()
        record = {"run": run, "data": data_hash, "params": {"C": C},
                  "cv_accuracy": round(float(score), 3), "sklearn": sklearn.__version__}
        log.write(json.dumps(record) + "\n")

with open("experiments.jsonl", encoding="utf-8") as log:
    runs = [json.loads(line) for line in log]
for r in runs:
    print(r)
print("Best run:", max(runs, key=lambda r: r["cv_accuracy"])["run"])

{'run': 1, 'data': '932e4c898919', 'params': {'C': 0.01}, 'cv_accuracy': 0.972, 'sklearn': '1.8.0'}
{'run': 2, 'data': '932e4c898919', 'params': {'C': 0.1}, 'cv_accuracy': 0.983, 'sklearn': '1.8.0'}
{'run': 3, 'data': '932e4c898919', 'params': {'C': 1.0}, 'cv_accuracy': 0.983, 'sklearn': '1.8.0'}
Best run: 2

If the data changes by even one value, the fingerprint changes, so you can always tell which data a score came from. In real projects, a tool such as MLflow does this tracking for you, with a web dashboard to compare runs and a model registry that records which model version is in production:

import mlflow

with mlflow.start_run():
    mlflow.log_param("C", 0.1)
    mlflow.log_metric("cv_accuracy", 0.983)

Validating incoming data

Many production failures are caused not by the model but by the data fed to it: a renamed column, a field sent in the wrong unit, an app bug that sends age 230. Check every batch before predicting, and log or reject bad rows.

import pandas as pd

RULES = {"age": (18, 80), "income": (5_000, 1_000_000), "city": None}

def validate(batch):
    problems = []
    for col, limits in RULES.items():
        if col not in batch.columns:
            problems.append(f"missing column: {col}")
            continue
        if batch[col].isna().any():
            problems.append(f"{col}: {batch[col].isna().sum()} missing value(s)")
        if limits:
            low, high = limits
            outside = batch[(batch[col] < low) | (batch[col] > high)]
            if len(outside):
                problems.append(f"{col}: {len(outside)} value(s) outside {low} to {high}")
    return problems

new_batch = pd.DataFrame({"age": [25, 41, 230, None],
                          "income": [32_000, 54_000, 41_000, 2_500_000],
                          "city": ["Pune", "Jaipur", None, "Kochi"]})
issues = validate(new_batch)
print("Batch OK" if not issues else "Problems found:")
for issue in issues:
    print(" -", issue)

Problems found:
 - age: 1 missing value(s)
 - age: 1 value(s) outside 18 to 80
 - income: 1 value(s) outside 5000 to 1000000
 - city: 1 missing value(s)

Checks like these also belong in automated tests that run whenever code or data changes, a practice called continuous integration (CI). Libraries such as Great Expectations and Pandera offer richer, reusable versions.

Serving a model

There are two common ways to use a model in production:

• Batch prediction: score many rows on a schedule, for example every night, and save the results to a database. Simple and cheap.
• Online prediction: answer one request at a time within milliseconds, usually through a web API.

A popular way to build such an API in Python is FastAPI. The service loads the saved pipeline once at start-up (see Saving & Loading Models) and returns predictions as JSON. This runs on a server, not in the browser:

# app.py, start with: uvicorn app:app
import joblib
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
model = joblib.load("loan_model.joblib")      # the whole fitted pipeline

class Applicant(BaseModel):
    income: float
    credit_score: float
    city: str

@app.post("/predict")
def predict(applicant: Applicant):
    row = pd.DataFrame([applicant.model_dump()])
    probability = float(model.predict_proba(row)[0, 1])
    return {"approve_probability": round(probability, 3), "model_version": "2026-10-30"}

The service is usually packaged in a Docker container with pinned package versions, so it runs the same everywhere. Returning the model version with each answer makes problems traceable later.

Monitoring for data drift

Once deployed, compare live inputs with the training data. Two common measures for a numeric feature:

• The Kolmogorov-Smirnov (KS) test from SciPy compares two distributions; a large statistic and tiny p-value mean they differ.
• The population stability index (PSI) compares how the data falls into bins. A common rule of thumb: below 0.1 is stable, 0.1 to 0.2 is worth watching, above 0.2 is a significant shift.

Here, monthly incomes of new loan applicants are compared with the training data.

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(42)
train_income = rng.lognormal(np.log(40_000), 0.4, 5000)
months = {"January": rng.lognormal(np.log(40_000), 0.4, 1000),
          "April": rng.lognormal(np.log(43_000), 0.4, 1000),
          "July": rng.lognormal(np.log(55_000), 0.5, 1000)}

def psi(expected, actual, bins=10):
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

for month, values in months.items():
    ks = ks_2samp(train_income, values)
    score = psi(train_income, values)
    status = "DRIFT" if score > 0.2 else "watch" if score > 0.1 else "stable"
    print(f"{month:<8} KS {ks.statistic:.3f}  p-value {ks.pvalue:.2g}  PSI {score:.3f}  -> {status}")

January  KS 0.045  p-value 0.065  PSI 0.009  -> stable
April    KS 0.092  p-value 1.5e-06  PSI 0.086  -> stable
July     KS 0.292  p-value 2.6e-63  PSI 0.480  -> DRIFT

Notice April: the KS test calls the change highly significant, yet PSI rates it stable. With a thousand rows, a statistical test flags even small shifts, while PSI measures how big the shift is, which is usually more useful for deciding when to alert. July's shift is large by both measures. Drift in inputs is a warning, not proof that the model got worse. The real test is performance on new labelled data, which often arrives later (you learn whether a loan defaulted months afterwards). Monitor both, and alert a person when either crosses a threshold.

Retraining and releasing safely

When it is time to update a model:

• Retrain on a schedule (for example monthly) or when monitoring triggers it, using the same versioned pipeline.
• Compare the new model with the current one on the same recent test data, overall and per group.
• Release gradually: run the new model in shadow mode (it predicts, but its answers are not used), then send it a small share of traffic (a canary), and keep the old version ready to roll back.
• Keep humans in the loop for high-impact decisions, and log every prediction with the model version that made it.

Workflow tools such as Apache Airflow and Kubeflow, and cloud platforms such as Amazon SageMaker, Google Vertex AI and Azure Machine Learning, automate these steps at scale. The ideas are the same at any size.

Common mistakes

• Treating deployment as the finish line. Models decay; plan monitoring and retraining from day one.
• Training and serving with different preprocessing. Deploy the whole fitted pipeline so both use identical steps.
• Not recording data and package versions. Without them you cannot reproduce or debug a model.
• Replacing a model without comparison or rollback. Always compare, release gradually and keep the previous version.

Key points

• MLOps applies engineering discipline to the whole lifecycle: data, training, deployment, monitoring and retraining.
• Version code, data and environments, and track every experiment's parameters and metrics.
• Validate incoming data before predicting; many failures are data problems.
• Serve models in batch or online (for example with FastAPI), packaged with pinned versions.
• Watch for data drift (KS test, PSI) and concept drift, measure live performance, and release new models gradually.

import numpy as np

def psi(expected, actual, bins=10):
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

def drift_report(train, live):
    score = psi(train, live)
    status = "DRIFT" if score > 0.2 else "watch" if score > 0.1 else "stable"
    return round(score, 3), status

rng = np.random.default_rng(42)
train = rng.normal(700, 50, 5000)
for label, live in [("same", rng.normal(700, 50, 1000)),
                    ("slight", rng.normal(690, 55, 1000)),
                    ("strong", rng.normal(660, 60, 1000))]:
    print(label, drift_report(train, live))

same (0.009, 'stable')
slight (0.095, 'stable')
strong (0.568, 'DRIFT')