itertools & functools

Two standard modules give you small, well-tested building blocks that replace whole loops. itertools works on iterables (anything you can loop over) and returns lazy iterators, which produce one value at a time instead of building big lists. functools works on functions: it fixes some of their arguments, combines results, caches answers and helps you write decorators.

You met count(), islice(), batched() and chain() in Iterators & Generators, and @cache and @wraps in Decorators. This lesson covers the rest of the most useful tools and shows where each one saves real work.

itertools: joining and slicing

chain() joins several iterables end to end. When the iterables are already inside a list, as in a list of lists, use chain.from_iterable() to flatten it into one stream. islice(iterable, start, stop, step) works like list slicing, but on any iterator, without building a list first.

import itertools

sections = [["Aarav", "Isha"], ["Rohan"], ["Meera", "Kabir", "Ananya"]]
everyone = list(itertools.chain.from_iterable(sections))
print(everyone)

results = (f"Roll {n}" for n in range(101, 200))     # a lazy generator
page_2 = itertools.islice(results, 5, 10)            # items 5 to 9
print(list(page_2))

['Aarav', 'Isha', 'Rohan', 'Meera', 'Kabir', 'Ananya']
['Roll 106', 'Roll 107', 'Roll 108', 'Roll 109', 'Roll 110']

The generator could hold millions of rows; islice() reads only as far as it needs. This is how you show "page 2" of results without loading everything into memory.

groupby: grouping neighbours

groupby(iterable, key) walks through the data and starts a new group every time the key changes. It returns pairs of (key, group), where each group is an iterator over the matching items. Because it only looks at neighbours, you almost always sort by the same key first.

import itertools

bookings = [("Pune", "Rohan"), ("Jaipur", "Priya"), ("Pune", "Isha"),
            ("Kochi", "Farhan"), ("Jaipur", "Kabir")]

def city(booking):
    return booking[0]

print("Unsorted:", [k for k, _ in itertools.groupby(bookings, key=city)])

for place, group in itertools.groupby(sorted(bookings, key=city), key=city):
    names = [name for _, name in group]
    print(place, names)

Unsorted: ['Pune', 'Jaipur', 'Pune', 'Kochi', 'Jaipur']
Jaipur ['Priya', 'Kabir']
Kochi ['Farhan']
Pune ['Rohan', 'Isha']

Without sorting, Pune and Jaipur each appear twice, because their bookings were not next to each other. Use groupby() on sorted data or data that is already in order, such as log lines by date. If the data is unordered and you need groups, a defaultdict(list) from The collections Module is often simpler.

Combinations: product, permutations, combinations

These tools generate every arrangement of some items, which saves you from writing nested loops.

import itertools

sizes = ["M", "L"]
colours = ["white", "navy", "maroon"]
variants = [f"{c}-{s}" for c, s in itertools.product(colours, sizes)]
print(len(variants), variants[:4])

teams = ["Rohtak", "Jaipur", "Pune", "Kochi"]
matches = list(itertools.combinations(teams, 2))
print(len(matches), "league matches:", matches[:3])

podiums = list(itertools.permutations(["Aarav", "Meera", "Kabir"], 2))
print(len(podiums), "gold-silver results:", podiums[:2])

6 ['white-M', 'white-L', 'navy-M', 'navy-L']
6 league matches: [('Rohtak', 'Jaipur'), ('Rohtak', 'Pune'), ('Rohtak', 'Kochi')]
6 gold-silver results: [('Aarav', 'Meera'), ('Aarav', 'Kabir')]

• product(a, b) pairs every item of a with every item of b, like nested for loops: every colour in every size.
• combinations(items, r) picks r items where order does not matter: Rohtak against Jaipur is the same match as Jaipur against Rohtak.
• permutations(items, r) picks r items where order matters: Aarav gold and Meera silver is different from the reverse.

The counts grow very fast with more items, so keep them lazy and avoid list() on large inputs.

accumulate, batched and pairwise

accumulate() gives running totals: each output is the result so far. Pass a different function, such as max, to get a running maximum. pairwise() (Python 3.10 and later) gives each item with its neighbour, ideal for differences between readings. batched() (Python 3.12 and later) cuts data into fixed-size chunks; since 3.13, strict=True raises ValueError if the last chunk is short.

import itertools

daily_sales = [1200, 950, 1800, 700, 2100]
print(list(itertools.accumulate(daily_sales)))
print(list(itertools.accumulate(daily_sales, max)))

readings = [31, 33, 36, 34]                  # temperature in Jaipur
print([b - a for a, b in itertools.pairwise(readings)])

for batch in itertools.batched(range(1, 8), 3):
    print("SMS batch:", batch)

[1200, 2150, 3950, 4650, 6750]
[1200, 1200, 1800, 1800, 2100]
[2, 3, -2]
SMS batch: (1, 2, 3)
SMS batch: (4, 5, 6)
SMS batch: (7,)

functools: partial and reduce

partial(func, *args, **kwargs) creates a new function with some arguments already filled in. It is cleaner than writing a tiny wrapper function or lambda, and the result can be passed anywhere a function is expected.

reduce(func, iterable) combines all items into one value by applying func to the result so far and the next item. Python has sum(), max(), min() and math.prod() for common cases, so use reduce() only for other combining jobs, such as intersecting several sets.

from functools import partial, reduce

def price_with_tax(amount, rate):
    return round(amount * (1 + rate / 100), 2)

with_gst_18 = partial(price_with_tax, rate=18)
with_gst_5 = partial(price_with_tax, rate=5)
print(with_gst_18(1000), with_gst_5(1000))

from_binary = partial(int, base=2)
print(list(map(from_binary, ["101", "1100", "11111111"])))

present = [{"Aarav", "Isha", "Kabir"}, {"Isha", "Kabir"}, {"Kabir", "Isha", "Meera"}]
print(sorted(reduce(set.intersection, present)))

1180.0 1050.0
[5, 12, 255]
['Isha', 'Kabir']

reduce(set.intersection, present) works out present[0] & present[1], then intersects that with present[2], giving the students who attended on every day.

functools: caching and wraps

@cache remembers every result forever. @lru_cache(maxsize=n) keeps only the n most recently used results and drops the least recently used one when full, which keeps memory under control in long-running programs. Both add cache_info() and cache_clear() methods.

from functools import lru_cache

@lru_cache(maxsize=2)
def fare(source, destination):
    print(f"  looking up {source}-{destination}...")
    return len(source + destination) * 45

print(fare("Pune", "Kochi"))
print(fare("Pune", "Kochi"))          # cached
print(fare("Jaipur", "Rohtak"))
print(fare("Lucknow", "Pune"))        # cache full: oldest entry dropped
print(fare("Pune", "Kochi"))          # looked up again
print(fare.cache_info())

  looking up Pune-Kochi...
405
405
  looking up Jaipur-Rohtak...
540
  looking up Lucknow-Pune...
495
  looking up Pune-Kochi...
405
CacheInfo(hits=1, misses=4, maxsize=2, currsize=2)

@wraps(func) belongs inside every decorator you write. It copies the wrapped function's name, docstring and other details onto the wrapper, so tools and error messages show the real name. The full story is in Decorators. Other functools tools worth knowing are @total_ordering (fills in <=, > and >= from __eq__ and __lt__) and @singledispatch (picks a function version based on the argument's type).

Common mistakes

• Using groupby() on unsorted data. You get the same key several times. Sort by the same key first.
• Reusing a used-up iterator. islice(), chain() and friends return iterators that work only once; make a list if you need the values twice.
• Calling list() on huge combinations. permutations() of 12 items has 479,001,600 results. Loop lazily or reduce the input.
• Caching functions with side effects. A cached function that prints, writes files or reads the clock will skip that work on repeated calls.

Key points

• itertools returns lazy iterators: chain.from_iterable() flattens and islice() slices any iterator.
• groupby() groups neighbouring items, so sort by the key first.
• product(), combinations() and permutations() replace nested loops for arrangements.
• accumulate() gives running totals, pairwise() gives neighbours and batched() gives chunks.
• partial() pre-fills arguments (with Placeholder gaps since 3.14), and reduce() combines items into one value.
• @lru_cache(maxsize=n) caches with a size limit; @cache has none; @wraps keeps decorated functions' details.

from itertools import accumulate, pairwise, combinations
from functools import partial

scores = [34, 12, 78, 0, 45, 101, 23]

print(list(accumulate(scores)))
print([b - a for a, b in pairwise(scores)])
print(sum(1 for _ in combinations(scores, 2)), "pairs")

def strike_rate(runs, balls):
    return round(runs / balls * 100, 1)

strike_rate_for_balls_50 = partial(strike_rate, balls=50)
print(strike_rate_for_balls_50(scores[0]))

[34, 46, 124, 124, 169, 270, 293]
[-22, 66, -78, 45, 56, -78]
21 pairs
68.0