Finding & Handling Outliers

An outlier is a value far away from the rest of the data: a temperature of 99.9 °C, an order of 120 power banks, a salary ten times the average. Some outliers are mistakes that must be fixed. Others are the most interesting rows in the whole dataset, like a genuine bulk order before Diwali.

In this lesson you will find outliers in four ways, look at each one before deciding, and then handle it in the way that fits: fix it, remove it, cap it, flag it, or keep it.

Why outliers matter

A single extreme value can drag the mean far from what is typical. The median, which is just the middle value, barely moves.

import pandas as pd

weather = pd.read_csv("weather.csv")
jaipur_aug = weather[(weather["city"] == "Jaipur") & (weather["date"].str.startswith("2025-08"))]

print(jaipur_aug["temp_max"].describe().round(2))
print("Mean:  ", round(jaipur_aug["temp_max"].mean(), 2))
print("Median:", jaipur_aug["temp_max"].median())
print(jaipur_aug.nlargest(2, "temp_max")[["date", "temp_max"]])

count    31.00
mean     34.74
std      12.36
min      28.70
25%      30.40
50%      32.20
75%      34.70
max      99.90
Name: temp_max, dtype: float64
Mean:   34.74
Median: 32.2
            date  temp_max
1318  2025-08-12      99.9
1312  2025-08-06      37.4

One faulty sensor reading of 99.9 °C raises Jaipur's August mean by about two degrees, while the median stays put. If you reported the mean, readers would think August was much hotter than it was. This is why you check for outliers before summarising.

Method 1: domain rules

The most reliable check uses what you know about the subject. Some values are simply impossible: a negative quantity, marks above 100, attendance above 100%, a temperature above about 55 °C anywhere in India.

import pandas as pd

sales = pd.read_csv("sales.csv")
students = pd.read_csv("students.csv", na_values=["AB"])
weather = pd.read_csv("weather.csv")

print(sales.loc[sales["quantity"] < 1, ["order_id", "product", "quantity"]])

subjects = ["maths", "science", "english", "hindi", "social_science"]
bad_marks = (students[subjects] > 100).any(axis=1)
print(students.loc[bad_marks | (students["attendance"] > 100), ["student_id", "attendance", "science"]])

print(weather.loc[weather["temp_max"] > 55, ["date", "city", "temp_max"]])

     order_id           product  quantity
274  ORD10274  Badminton Racket      -1.0
    student_id  attendance  science
24       S1025        86.0    110.0
426      S1425       104.0     59.0
            date    city  temp_max
1318  2025-08-12  Jaipur      99.9
1523  2025-03-05   Kochi      99.9

Each of these rows breaks a rule of the real world, so they are errors, not interesting extremes. Write rules like these down and run them every time new data arrives.

Method 2: the IQR rule

When there is no hard limit, a common statistical rule uses the interquartile range (IQR): the distance between the 25th percentile (Q1) and the 75th percentile (Q3), which holds the middle half of the data. Values more than 1.5 × IQR below Q1 or above Q3 are flagged as possible outliers. This is the same rule box plots use to draw separate dots.

import pandas as pd

def iqr_bounds(s, k=1.5):
    q1, q3 = s.quantile([0.25, 0.75])
    iqr = q3 - q1
    return q1 - k * iqr, q3 + k * iqr

emp = pd.read_csv("employees.csv").drop_duplicates()
low, high = iqr_bounds(emp["salary"])
print(f"Salary fences: ₹{low:,.0f} to ₹{high:,.0f}")
flagged = emp[(emp["salary"] < low) | (emp["salary"] > high)]
print(flagged[["department", "experience_years", "salary"]].sort_values("salary"))

sales = pd.read_csv("sales.csv")
low, high = iqr_bounds(sales["quantity"])
print("Quantity fences:", low, high, "| flagged:", (sales["quantity"] > high).sum())

Salary fences: ₹56,000 to ₹1,640,000
       department  experience_years     salary
89   Data Science                 8  1676000.0
69   Data Science                 7  1722000.0
223  Data Science                 9  1775000.0
158  Data Science                11  1788000.0
2     Engineering                12  1791000.0
248   Engineering                18  1849000.0
240   Engineering                15  1867000.0
255  Data Science                14  1908000.0
25    Engineering                16  1979000.0
Quantity fences: -0.5 3.5 | flagged: 45

The salary rule flags a handful of high earners, and every one of them has many years of experience. They are not errors; they are senior staff. For quantity, the rule is even less helpful: most orders are for 1 or 2 items, so the upper fence is 3.5 and every grocery order of 4 or 5 items gets flagged. The IQR rule works best for roughly symmetric, continuous data. Treat its output as "rows to look at", never as "rows to delete".

Method 3: z-scores within groups

A z-score says how many standard deviations a value is from the mean: (value - mean) / std. Values with a z-score beyond about ±3 are rare in bell-shaped data. Comparing within the right group matters: 42 °C is normal for Delhi in May and impossible for Kochi.

import pandas as pd

weather = pd.read_csv("weather.csv", parse_dates=["date"])
weather["month"] = weather["date"].dt.month

grp = weather.groupby(["city", "month"])["temp_max"]
weather["z"] = ((weather["temp_max"] - grp.transform("mean")) / grp.transform("std")).round(2)

suspects = weather[weather["z"].abs() > 3]
print(suspects[["date", "city", "temp_max", "z"]])

           date    city  temp_max     z
1318 2025-08-12  Jaipur      99.9  5.27
1523 2025-03-05   Kochi      99.9  5.36

Each temperature is compared with the same city in the same month, and only the two sensor errors stand out. Note one weakness: an extreme value raises the standard deviation of its own group, which shrinks its z-score. With small groups, a robust version based on the median works better, but for a quick check this is fine.

Method 4: look at it

A box plot draws the median, the box from Q1 to Q3, whiskers out to the IQR fences, and every value beyond them as a separate dot. It shows outliers for many groups at once.

import pandas as pd
import matplotlib.pyplot as plt

weather = pd.read_csv("weather.csv")
cities = ["Delhi", "Mumbai", "Chennai", "Jaipur", "Kochi"]
data = [weather.loc[weather["city"] == c, "temp_max"].dropna() for c in cities]

plt.boxplot(data, tick_labels=cities)
plt.title("Daily maximum temperature, 2025")
plt.ylabel("°C")
plt.show()

The chart shows five boxes between roughly 25 °C and 40 °C, and two lonely dots near 100 °C above the Jaipur and Kochi boxes. Those are impossible to miss. Every box also has a few small dots close to the whiskers; those are ordinary hot or cool days, which is a reminder that a dot on a box plot is not automatically an error.

Handling outliers

Once you have looked at each suspect, choose one of these actions:

import pandas as pd

sales = pd.read_csv("sales.csv").drop_duplicates()
print(sales.nlargest(3, "quantity")[["order_id", "product", "quantity", "unit_price"]])

sales.loc[sales["quantity"] < 1, "quantity"] = None                     # impossible: unknown
sales.loc[sales["order_id"] == "ORD11014", "quantity"] = 1              # checked: really 1 copy
sales["is_bulk"] = sales["quantity"] >= 20                              # genuine: keep and flag

print(sales["quantity"].describe().round(2))
print(sales.loc[sales["is_bulk"], ["order_id", "product", "quantity"]])

capped = sales["quantity"].clip(upper=sales["quantity"].quantile(0.99))
print("Max after capping at the 99th percentile:", capped.max())

      order_id           product  quantity  unit_price
1020  ORD11014        Exam Guide     120.0         450
99    ORD10100  Masala Chai 250g      60.0         180
87    ORD10088         Atta 10kg       5.0         480
count    1202.00
mean        1.55
std         1.87
min         1.00
25%         1.00
50%         1.00
75%         2.00
max        60.00
Name: quantity, dtype: float64
    order_id           product  quantity
99  ORD10100  Masala Chai 250g      60.0
Max after capping at the 99th percentile: 4.0

The three biggest orders tell different stories. 120 copies of one exam guide, paid by cash on delivery, is suspicious, so in this example you check the warehouse record, find that one copy was shipped, and fix the value. Atta at 5 bags is an ordinary grocery order. 60 packs of masala chai fits a small office or canteen buying in bulk, so it is kept and flagged. Capping with clip() is shown at the end; it is common before machine learning, but in a sales report you would rather keep the true number.

Common mistakes

• Deleting every flagged value. Statistical rules find unusual values, not wrong ones. Look first.
• Using one rule for different groups. 42 °C is normal in Delhi in May; compare within city and month.
• Removing outliers to get a nicer result. That is cherry-picking. Decide your rules before you see the answer.
• Forgetting that outliers distort the mean and standard deviation. Use the median and IQR when outliers are present.

Key points

• An outlier is a value far from the rest; it may be an error or a real, important event.
• Domain rules (impossible values) are the most reliable check.
• The IQR rule flags values beyond 1.5 × IQR from the quartiles; z-scores beyond ±3 are rare in bell-shaped data.
• Compare within meaningful groups, and use box plots to see outliers in many groups at once.
• Handle each outlier deliberately: fix, set to missing, remove, cap with clip(), flag, or keep.

import pandas as pd

emp = pd.read_csv("employees.csv").drop_duplicates()
emp["department"] = emp["department"].str.strip().str.title().replace({"Engg": "Engineering", "Hr": "HR"})
grp = emp.groupby("department")["salary"]
emp["z"] = (emp["salary"] - grp.transform("mean")) / grp.transform("std")
print(emp.loc[emp["z"] > 2.5, ["department", "experience_years", "rating", "salary", "z"]].round(2))
# Genuine: every high earner has far more experience than the typical employee.

      department  experience_years  rating     salary     z
25   Engineering                16       3  1979000.0  2.96
175      Finance                11       3  1364000.0  3.02
201        Sales                21       3  1520000.0  3.53
240  Engineering                15       3  1867000.0  2.59
248  Engineering                18       1  1849000.0  2.53