apply, map & Vectorisation

Sooner or later you need a calculation that is not built into pandas: a grade from marks, a delivery charge that depends on the city and order value, a tax band from a salary. There are several ways to write it, and the fastest is usually not the one beginners reach for first.

In this lesson you will compare vectorised code, map(), apply(), np.where(), np.select() and pd.cut(), and learn a simple rule for choosing between them.

The rule: vectorise first

A vectorised operation works on a whole column at once, using NumPy's fast compiled loops: arithmetic, comparisons, .str and .dt methods, np.where() and so on. apply() with a Python function, by contrast, calls your function once per value or row, from Python. On large data that can be tens or hundreds of times slower.

map(): replace values using a dictionary or function

Series.map() transforms each value of one column. With a dictionary it acts as a lookup table; with a function it calls the function on each value.

import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()

zones = {"Maharashtra": "West", "Gujarat": "West", "Delhi": "North", "Rajasthan": "North",
         "Uttar Pradesh": "North", "Karnataka": "South", "Tamil Nadu": "South",
         "Telangana": "South", "Kerala": "South", "West Bengal": "East"}
sales["zone"] = sales["state"].map(zones)
print(sales["zone"].value_counts())

sales["size"] = sales["unit_price"].map(lambda p: "premium" if p >= 1500 else "regular")
print(sales[["product", "unit_price", "size"]].drop_duplicates("product").head(4))

zone
South    475
North    349
West     289
East      96
Name: count, dtype: int64
             product  unit_price     size
0           Yoga Mat         699  regular
1        Cricket Bat        1599  premium
3  Bluetooth Earbuds        1499  regular
4            T-Shirt         399  regular

The dictionary version is fast and readable. The lambda version works, but it is a disguised loop; np.where(sales["unit_price"] >= 1500, "premium", "regular") gives the same result faster. For a whole DataFrame, DataFrame.map() applies a function to every single cell.

np.where and np.select: vectorised if-else

For a choice between two values, np.where() is the vectorised if ... else. For several conditions, np.select() takes a list of conditions and a matching list of results, checks them in order, and uses default when none match.

import numpy as np
import pandas as pd

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()
m = students["maths"]

students["passed"] = np.where(m >= 33, "Pass", "Fail")

conditions = [m >= 91, m >= 75, m >= 60, m >= 33]
grades = ["A1", "A2", "B", "C"]
students["grade"] = np.select(conditions, grades, default="Needs support")
students.loc[m.isna(), "grade"] = "Absent"

print(students[["name", "maths", "passed", "grade"]].head(6))
print(students["grade"].value_counts())

           name  maths passed grade
0    Shreya Rao   50.0   Pass     C
1  Kabir Pillai   52.0   Pass     C
2   Tushar Iyer   64.0   Pass     B
3   Kunal Joshi   63.0   Pass     B
4    Tarun Khan   55.0   Pass     C
5  Bhavna Singh   33.0   Pass     C
grade
C                214
B                144
A2                65
Needs support     38
A1                17
Absent             2
Name: count, dtype: int64

The conditions are checked from first to last, so a mark of 95 gets A1 even though it is also above 75. Missing marks fail every comparison, so np.select() would give them the default; the extra .loc line labels absent students honestly. (The grade bands here are made up for the example.)

pd.cut: bins from number ranges

When grades or bands are simply ranges of numbers, pd.cut() is even clearer. Give it the edges of the bins and a label for each.

import pandas as pd

students = pd.read_csv("students.csv", na_values=["AB"]).drop_duplicates()

students["band"] = pd.cut(
    students["maths"],
    bins=[0, 32, 59, 74, 90, 100],
    labels=["Below 33", "33-59", "60-74", "75-90", "91-100"],
)
print(students["band"].value_counts(sort=False))

students["attendance_q"] = pd.qcut(students["attendance"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])
print(students.groupby("attendance_q", observed=True)["maths"].mean().round(1))

band
Below 33     38
33-59       214
60-74       144
75-90        65
91-100       17
Name: count, dtype: int64
attendance_q
Q1    50.3
Q2    53.9
Q3    61.3
Q4    67.2
Name: maths, dtype: float64

Bins include their right edge, so (32, 59] holds marks from 33 to 59. The result is a category column whose order follows the bins, which is why sort=False lists the bands low to high. pd.qcut() instead makes bins with roughly equal numbers of rows, here attendance quartiles. Average maths marks rise steadily from the lowest attendance quarter to the highest.

apply(): when you really need it

apply() calls a function for each column (default) or for each row with axis=1. Use row-wise apply() only when the logic is truly complicated and the data is small. Here is a delivery charge that depends on three columns, written both ways.

import time
import numpy as np
import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()
sales["value"] = sales["quantity"].fillna(1) * sales["unit_price"]

def charge(row):
    if row["value"] >= 999:
        return 0
    if row["state"] in ("Kerala", "West Bengal") or row["category"] == "Groceries":
        return 79
    return 49

start = time.perf_counter()
slow = sales.apply(charge, axis=1)
t_apply = time.perf_counter() - start

start = time.perf_counter()
far = sales["state"].isin(["Kerala", "West Bengal"]) | (sales["category"] == "Groceries")
fast = np.select([sales["value"] >= 999, far], [0, 79], default=49)
t_vector = time.perf_counter() - start

print("Same answers:", (slow.to_numpy() == fast).all())
print("Total delivery charges: ₹", fast.sum())
print("Vectorised was faster:", t_vector < t_apply)

Same answers: True
Total delivery charges: ₹ 29829
Vectorised was faster: True

Both versions give identical charges, but apply(axis=1) builds a Series object for every row and calls Python code 1,209 times, while the vectorised version runs a handful of whole-column operations. On a million rows the difference is minutes versus a second. The function version is easier to read for some people, so it is a fine first draft; once it works, rewrite it with np.select().

assign and pd.col: readable pipelines

assign() adds columns in a method chain. In older code you see a lambda df: ... inside it so the new column can use columns created earlier in the chain. pandas 3.0 adds pd.col("name"), a shorter way to refer to a column inside assign() and loc[].

import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()

summary = (
    sales
    .assign(gross=pd.col("quantity") * pd.col("unit_price"))
    .assign(net=pd.col("gross") * (1 - pd.col("discount") / 100))
    .loc[pd.col("category") == "Electronics", ["product", "gross", "net"]]
    .groupby("product")
    .sum()
    .round(0)
)
print(summary)

old_style = sales.assign(gross=lambda d: d["quantity"] * d["unit_price"])
print(old_style["gross"].sum() == sales.assign(gross=pd.col("quantity") * pd.col("unit_price"))["gross"].sum())

                      gross       net
product                              
Bluetooth Earbuds  106429.0  100433.0
Laptop Stand        79134.0   73559.0
Power Bank          83916.0   79620.0
Smartwatch         200933.0  188937.0
USB-C Cable         15847.0   14786.0
Wireless Mouse      47377.0   45527.0
True

pd.col("gross") refers to the column created in the previous step, which a plain sales["gross"] could not do, because sales itself never gets that column. The lambda style still works and is what you will see in most existing code.

Common mistakes

• Reaching for apply() first. Check whether arithmetic, np.where(), np.select(), pd.cut(), .map() or a .str method already does the job.
• Forgetting axis=1. df.apply(func) passes columns to your function, not rows.
• Overlapping conditions in the wrong order. np.select() uses the first true condition, so list the strictest one first.
• Using applymap(). It was removed in pandas 3.0; use DataFrame.map().

Key points

• Prefer vectorised code: whole-column arithmetic, comparisons and .str/.dt methods.
• Series.map() with a dictionary is a fast lookup; DataFrame.map() replaced applymap().
• np.where() handles two-way choices and np.select() handles many conditions in order.
• pd.cut() turns number ranges into labelled bins; pd.qcut() makes equal-sized bins.
• Row-wise apply(axis=1) is flexible but slow; use it for small data or as a first draft.
• pandas 3.0's pd.col() refers to columns inside assign() and loc[] chains.

import numpy as np
import pandas as pd

emp = pd.read_csv("employees.csv").drop_duplicates()
emp["band"] = pd.cut(emp["experience_years"], bins=[-1, 1, 4, 9, 50],
                     labels=["Fresher", "Junior", "Mid", "Senior"])
r = emp["rating"]
emp["bonus_pct"] = np.select([r == 5, r == 4, r == 3], [15, 10, 5], default=0)
print(emp.groupby("band", observed=True)[["salary", "bonus_pct"]].mean().round(1))

            salary  bonus_pct
band                         
Fresher   612400.0        8.2
Junior    769446.3        7.0
Mid       958194.7        6.3
Senior   1339129.0        6.1