Notebooks, Scripts & Reproducibility

An analysis is reproducible when someone else, or you in six months, can run it again on the same data and get the same numbers. It sounds obvious, yet the most common answer to "how did you get this figure?" in many offices is "I'm not sure, it was in a notebook somewhere". Reproducibility is what turns a one-off exploration into work a team can rely on, audit and rerun every month.

In this lesson you will see where notebooks go wrong, how to lay out a project, how to write an analysis as a chain of small functions, and how to control randomness, log what happened and lock package versions.

Notebooks and scripts

You met Jupyter and Colab in Tools & Setup. Notebooks are excellent for exploring: you see tables and charts next to the code. Their weakness is hidden state. Cells can be run in any order, run twice, edited after running or deleted, and the variables stay in memory. A notebook can show a correct-looking result that the code on the page can no longer produce.

A practical routine used by many teams: explore in a notebook, then move the code that works into functions in a .py file, and keep the notebook as a thin report that calls them. Before sharing any notebook, use Restart kernel and run all cells. If it fails or gives different numbers, it was never reproducible.

Two command-line tools help: jupyter nbconvert --to script analysis.ipynb turns a notebook into a .py file, and jupyter nbconvert --to html --execute report.ipynb runs a notebook from top to bottom and saves the result as a web page, which proves it runs cleanly.

A project layout

Put every analysis in its own folder with the same structure, so anyone can find things:

sales-review/
  data/
    raw/            original files, never edited
    processed/      cleaned outputs written by code
  notebooks/
    01-explore.ipynb
  src/
    pipeline.py     load, clean, transform, summarise functions
  reports/
    q3-summary.html
  pyproject.toml    project and package versions
  uv.lock           exact versions of every package
  README.md         how to run it, data sources, owner

The most important rule is that data/raw is read-only. Never fix a value by hand in Excel and save over the original: if the fix is in code, it is documented and can be rerun when next month's file arrives. Use relative paths such as Path("data") / "raw" / "sales.csv" built with pathlib, never C:\Users\priya\Desktop\final_final2.csv, which works on one laptop only.

Pipelines of small functions

Write each step as a function that takes a DataFrame and returns a new one, then chain the steps with .pipe(). Each function is short, has a name that explains it, and can be tested on its own. Because nothing outside the function is changed, running the chain twice always gives the same answer.

import pandas as pd

CITY_FIXES = {"Bangalore": "Bengaluru", "Cochin": "Kochi"}

def load(path):
    return pd.read_csv(path, parse_dates=["order_date"])

def clean(df):
    out = df.drop_duplicates()
    out = out[out["quantity"].between(1, 20)]
    return out.assign(city=out["city"].str.strip().str.title().replace(CITY_FIXES))

def add_revenue(df):
    return df.assign(revenue=df["quantity"] * df["unit_price"] * (1 - df["discount"] / 100))

def quarterly_by_city(df, cities):
    return (
        df[df["city"].isin(cities)]
        .groupby([df["order_date"].dt.to_period("Q"), "city"])["revenue"].sum()
        .unstack()
        .round(0)
    )

report = (
    load("sales.csv")
    .pipe(clean)
    .pipe(add_revenue)
    .pipe(quarterly_by_city, cities=["Pune", "Kochi", "Jaipur"])
)
print(report)

city         Jaipur    Kochi     Pune
order_date
2025Q1      20517.0  26991.0  39562.0
2025Q2      22075.0  11332.0  25383.0
2025Q3      24948.0  21903.0  46455.0
2025Q4      46029.0  34058.0  54944.0

df.pipe(f, x=1) is the same as f(df, x=1), but it reads from top to bottom in the order the steps happen. Settings such as CITY_FIXES live in one place at the top instead of being repeated. If the finance team asks "do you remove orders with 60 units?", you can point to one line in clean().

Seeds: making randomness repeatable

Sampling, bootstrapping, train/test splits and simulations use random numbers. Without a seed (a starting value for the random generator), every run gives slightly different numbers, and nobody can check your result. Pass random_state= to pandas and create NumPy generators with np.random.default_rng(seed), as in Random Numbers & Simulation.

import numpy as np
import pandas as pd

students = pd.read_csv("students.csv").drop_duplicates()

a = students.sample(5, random_state=2025)["student_id"].tolist()
b = students.sample(5, random_state=2025)["student_id"].tolist()
c = students.sample(5)["student_id"].tolist()
print(a == b, "| without a seed the sample changes each run")

rng = np.random.default_rng(2025)
boot = [students["study_hours"].sample(frac=1, replace=True, random_state=rng).mean() for _ in range(1000)]
print("Bootstrap 95% interval:", np.percentile(boot, [2.5, 97.5]).round(3))

True | without a seed the sample changes each run
Bootstrap 95% interval: [2.465 2.646]

The two seeded samples are identical; sample c changes every time you press Run. Passing one generator rng to all 1,000 bootstrap samples makes the whole simulation repeatable while each sample is still different from the others. Run it twice and the interval is the same to three decimals.

Logging and fingerprints

A script that runs every month should say what it did: how many rows it loaded, what it removed, what looked odd. Python's logging module prints messages with a level (INFO, WARNING, ERROR) and can also write them to a file. A fingerprint, a short number computed from the data's contents, tells you whether the input file changed since the last run.

import logging
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s", force=True)
log = logging.getLogger("sales_report")

def fingerprint(df):
    return format(pd.util.hash_pandas_object(df, index=False).sum() % 10**8, "08d")

raw = pd.read_csv("sales.csv")
log.info("loaded %d rows, fingerprint %s", len(raw), fingerprint(raw))

clean = raw.drop_duplicates()
removed = len(raw) - len(clean)
log.info("removed %d duplicate rows", removed)
if clean["payment_mode"].isna().any():
    log.warning("%d orders have no payment mode", clean["payment_mode"].isna().sum())

assert clean["order_id"].is_unique, "order_id must be unique after cleaning"
print("Fingerprint again:", fingerprint(raw), "| changed copy:", fingerprint(raw.head(1216)))

INFO loaded 1217 rows, fingerprint 26570322
INFO removed 8 duplicate rows
WARNING 11 orders have no payment mode
Fingerprint again: 26570322 | changed copy: 53546279

pd.util.hash_pandas_object() gives one hash number per row; adding them up and keeping 8 digits makes a compact fingerprint. The same data always gives the same fingerprint, and removing one row changes it completely. Write the fingerprint into your report's footer: if someone reruns the report and gets different numbers, comparing fingerprints shows at once whether the data or the code changed. (Fingerprints can differ between pandas versions, so compare them within one environment.) The assert stops the script if a basic rule is broken, as in Data Validation & Quality Checks.

Locking the environment

The same code can give different results, or fail, with different package versions; What's New in pandas 3.0 showed how much can change. Record the versions with the project. With uv:

uv init sales-review
cd sales-review
uv add pandas matplotlib pyarrow
uv run python src/pipeline.py

uv add records the packages in pyproject.toml and the exact versions of everything in uv.lock. A colleague who clones the project runs uv sync to get the identical environment, and uv run runs a command inside it, so the script uses the locked versions without any manual activation step. In older projects you will see pip freeze > requirements.txt and pip install -r requirements.txt, which does a similar job with less precision. Keep the code, pyproject.toml and uv.lock in Git; keep large or personal data out of Git, and note in the README where the data comes from.

Finally, give the script one entry point, so the whole analysis runs with one command:

# src/pipeline.py
import argparse

def main():
    parser = argparse.ArgumentParser(description="Quarterly sales report")
    parser.add_argument("csv_path")
    parser.add_argument("--cities", nargs="+", default=["Pune", "Kochi", "Jaipur"])
    args = parser.parse_args()
    report = load(args.csv_path).pipe(clean).pipe(add_revenue).pipe(quarterly_by_city, cities=args.cities)
    report.to_csv("reports/quarterly_by_city.csv")
    print("Saved", report.shape)

if __name__ == "__main__":
    main()

Saved (4, 3)

Running uv run python src/pipeline.py data/raw/sales.csv --cities Pune Kochi Jaipur performs every step from raw file to saved report, with the functions from the pipeline section defined in the same file.

Common mistakes

• Sharing a notebook without "Restart and run all". Hidden state hides broken code.
• Editing raw files by hand. Fix data in code so the fix is recorded and repeatable.
• Absolute paths. C:\Users\... paths break on every other computer.
• No seed. Random results that change every run cannot be checked.
• No version record. "It worked on my laptop" usually means different package versions.

Key points

• Reproducible means the same data and code give the same numbers for anyone, any time.
• Explore in notebooks; move working code into functions in .py files; restart and run all before sharing.
• Keep raw data read-only, use relative pathlib paths and a standard folder layout.
• Chain small DataFrame-in, DataFrame-out functions with .pipe().
• Seed all randomness with random_state= or np.random.default_rng(seed).
• Log what the script did, fingerprint the input, assert key rules, and lock versions with uv.

import pandas as pd

SUBJECTS = ["maths", "science", "english", "hindi", "social_science"]

def load(path):
    return pd.read_csv(path, na_values=["AB"])

def clean(df):
    out = df.drop_duplicates()
    return out.assign(gender=out["gender"].str.upper())

def add_average(df):
    return df.assign(average=df[SUBJECTS].mean(axis=1).round(1))

def top_per_city(df):
    return (
        df.sort_values(["city", "average"], ascending=[True, False])
        .groupby("city").head(1)[["city", "name", "average"]]
    )

result = load("students.csv").pipe(clean).pipe(add_average).pipe(top_per_city)
print(result.to_string(index=False))

   city          name  average
 Jaipur Ishita Ansari     88.0
  Kochi Fatima George     88.0
Lucknow  Kabir Sharma     99.2
   Pune  Gauri Ansari     95.4
 Rohtak  Sahil Ansari     91.6