Profiling & Performance

"My script is slow" is one of the most common complaints you will hear at work. The beginner's reaction is to start rewriting lines that look slow. Professionals do the opposite: they measure first, find the one or two places where the time really goes, fix those, and measure again. Most programs spend nearly all their time in a small part of the code, so guessing usually means optimising the wrong thing.

In this lesson you will time small pieces of code with timeit, find the slow functions in a whole program with a profiler, learn the fixes that matter most in Python (choosing the right data structure, doing work once instead of many times, and letting built-ins do the looping), and check memory use.

Rule one: measure, do not guess

Timing with time.perf_counter() around a block, as in Threads, Processes & Free-threading, is fine for a whole program. For small pieces of code, use the timeit module. It runs the code many times and gives a reliable total, because a single run of fast code is too short to measure.

Here is the classic first finding: checking whether a value is in a list means looking at the items one by one, but a set jumps straight to the answer using a hash, like the index at the back of a book.

import timeit

roll_list = list(range(100_000))
roll_set = set(roll_list)

t_list = timeit.timeit(lambda: 99_999 in roll_list, number=200)
t_set = timeit.timeit(lambda: 99_999 in roll_set, number=200)

print("Same answer:", (99_999 in roll_list) == (99_999 in roll_set))
print("Set lookup at least 100 times faster:", t_list > 100 * t_set)

Same answer: True
Set lookup at least 100 times faster: True

Timings change from run to run and from computer to computer, so this example prints a comparison instead of raw numbers. From the terminal, python -m timeit compares two ways of doing the same job and picks a sensible number of loops for you:

python -m timeit -s "names = [str(n) for n in range(1000)]" "s = ''" "for n in names: s += n"
python -m timeit -s "names = [str(n) for n in range(1000)]" "''.join(names)"

500 loops, best of 5: 544 usec per loop
10000 loops, best of 5: 40.5 usec per loop

-s is setup code that is not timed. On this laptop, joining 1,000 strings with str.join() was about 13 times faster than adding them one by one. Your numbers will differ; the ratio is what matters.

Finding hot spots with cProfile

timeit answers "which of these two is faster?". A profiler answers the bigger question: "where does my whole program spend its time?". Here is a report script for a shop with 5,000 orders that feels slow:

def load_orders():
    """Make 5,000 sample orders: (customer, amount in rupees)."""
    return [(f"C{n % 200:03}", (n * 37) % 2000 + 100) for n in range(5_000)]


def unique_customers(orders):
    seen = []
    for customer, _ in orders:
        if customer not in seen:
            seen.append(customer)
    return seen


def total_for(customer, orders):
    return sum(amount for c, amount in orders if c == customer)


def build_report(orders):
    return {c: total_for(c, orders) for c in unique_customers(orders)}


if __name__ == "__main__":
    report = build_report(load_orders())
    print(len(report), "customers, C007 spent", report["C007"])

Run it under the standard cProfile profiler, sorted by cumulative time (the time spent in a function including everything it calls):

python -m cProfile -s cumulative report.py

200 customers, C007 spent 26975
         5808 function calls in 0.491 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.491    0.491 {built-in method builtins.exec}
        1    0.001    0.001    0.491    0.491 report.py:1(<module>)
        1    0.001    0.001    0.423    0.423 report.py:18(build_report)
      200    0.002    0.000    0.286    0.001 report.py:14(total_for)
      200    0.005    0.000    0.284    0.001 {built-in method builtins.sum}
     5200    0.279    0.000    0.279    0.000 report.py:15(<genexpr>)
        1    0.136    0.136    0.136    0.136 report.py:6(unique_customers)
        1    0.067    0.067    0.067    0.067 report.py:1(load_orders)
      200    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

How to read it: ncalls is how many times a function ran, tottime is time spent in the function's own code, and cumtime includes the functions it called. Two lines stand out. total_for() ran 200 times, and each call scanned all 5,000 orders: a million checks to add up 5,000 numbers. And unique_customers() uses not in seen on a list, which you now know is slow. Profilers slow the program down a little themselves, so look at the proportions, not the exact seconds.

You can also profile one part of a program from code, with cProfile.Profile() and the pstats module to sort and print the results. VS Code and PyCharm can show the same data as charts.

Fixing the hot spot

Both problems have the same cure: go through the data once and keep running totals in a dictionary. The fixed version also checks that it gives exactly the same report, which is the rule for every optimisation: same results, less time.

import timeit
from collections import defaultdict

orders = [(f"C{n % 200:03}", (n * 37) % 2000 + 100) for n in range(5_000)]


def build_report_slow(orders):
    seen = []
    for customer, _ in orders:
        if customer not in seen:
            seen.append(customer)
    return {c: sum(a for x, a in orders if x == c) for c in seen}


def build_report_fast(orders):
    totals = defaultdict(int)
    for customer, amount in orders:          # one pass over the data
        totals[customer] += amount
    return dict(totals)


slow = build_report_slow(orders)
fast = build_report_fast(orders)
print("Same report:", slow == fast)
print("C007 spent", fast["C007"])

t_slow = timeit.timeit(lambda: build_report_slow(orders), number=3)
t_fast = timeit.timeit(lambda: build_report_fast(orders), number=3)
print("Fast version at least 20 times faster:", t_slow > 20 * t_fast)

Same report: True
C007 spent 26975
Fast version at least 20 times faster: True

The slow version does work proportional to customers × orders; the fast one does work proportional to orders. With 200 customers that is roughly 200 times less work, and the gap grows as the shop grows. This way of describing how work grows with input size is called big-O notation: the slow report is O(n × m), the fast one O(n).

The fixes that matter most

Change one thing at a time and measure after each change. Readable code that is fast enough beats clever code that is slightly faster.

Memory

Speed is not the only cost. A list holds all its items at once; a generator produces them one by one, so its own size stays tiny however much data flows through it:

import sys

amounts_list = [n * 2 for n in range(1_000_000)]
amounts_gen = (n * 2 for n in range(1_000_000))

print("List object over 1 MB:", sys.getsizeof(amounts_list) > 1_000_000)
print("Generator object under 1 KB:", sys.getsizeof(amounts_gen) < 1_000)
print(sum(amounts_list) == sum(amounts_gen))

List object over 1 MB: True
Generator object under 1 KB: True
True

sys.getsizeof() measures one object only, not the items inside it. To find where a whole program allocates memory, use the standard tracemalloc module: call tracemalloc.start(), run your code, then tracemalloc.take_snapshot().statistics("lineno") lists the lines that allocated the most.

Common mistakes

• Optimising without measuring. You will speed up code that was never the problem.
• Measuring once. A single timing is noisy; use timeit, or repeat and take the best.
• Breaking correctness for speed. Always check the fast version gives the same result, ideally with a test.
• Forgetting the size of real data. Code that is fine for 100 rows can take hours for 1,000,000 if it is O(n²). Test with realistic sizes.
• Reaching for threads first. Threads do not speed up pure-Python calculations on the normal build; fix the algorithm first.

Key points

• Measure first: timeit for small pieces, a profiler for whole programs.
• python -m cProfile -s cumulative script.py shows call counts and where time goes; read proportions.
• Most big speed-ups come from better data structures and doing work once, not from micro-tweaks.
• Use sets and dictionaries for lookups, "".join() for building strings, built-ins for loops.
• Check memory with generators, sys.getsizeof() and tracemalloc.
• Python 3.15's profiling.sampling gives low-overhead profiling and flame graphs, even for running programs.

def scholarship_students(jaipur, scholarship):
    found = []
    for name in jaipur:
        if name in scholarship:
            found.append(name)
    return found

import timeit

def scholarship_students(jaipur, scholarship):
    found = []
    for name in jaipur:
        if name in scholarship:
            found.append(name)
    return found

def scholarship_students_fast(jaipur, scholarship):
    allowed = set(scholarship)                  # built once
    return [name for name in jaipur if name in allowed]

jaipur = [f"student{n}" for n in range(3_000)]
scholarship = [f"student{n}" for n in range(0, 6_000, 3)]

print(scholarship_students(jaipur, scholarship) == scholarship_students_fast(jaipur, scholarship))
slow = timeit.timeit(lambda: scholarship_students(jaipur, scholarship), number=2)
fast = timeit.timeit(lambda: scholarship_students_fast(jaipur, scholarship), number=2)
print("Fast version at least 10 times faster:", slow > 10 * fast)

True
Fast version at least 10 times faster: True