Selecting & Filtering Data

Most questions about data start with "show me only...": only the orders from Pune, only students with attendance below 75%, only the rainy days in July. In pandas, answering them means selecting the right rows and the right columns.

In this lesson you will use .loc and .iloc on DataFrames, filter with conditions, use the readable query() method, and change selected values the safe way.

loc and iloc on a DataFrame

You met .loc (labels) and .iloc (positions) with Series. On a DataFrame they take two parts: [rows, columns].

import pandas as pd

sales = pd.read_csv("sales.csv", index_col="order_id")
cols = ["city", "product", "quantity", "unit_price"]

print(sales.loc["ORD10010", "product"])                       # one value
print(sales.loc["ORD10010":"ORD10013", cols])                  # label range, end included
print(sales.iloc[0:3, 1:4])                                    # positions, end excluded
print(sales.iloc[-2:][["city", "payment_mode"]])               # last two rows

Bedsheet Set
               city       product  quantity  unit_price
order_id                                               
ORD10010       Pune  Bedsheet Set       1.0        1099
ORD10011  Bengaluru   Denim Jeans       1.0        1299
ORD10012  Bengaluru    Power Bank       1.0         999
ORD10013    Chennai    Power Bank       3.0         999
             city        state category
order_id                               
ORD10001   Mumbai  Maharashtra   Sports
ORD10002  Kolkata  West Bengal   Sports
ORD10003   Mumbai  Maharashtra   Sports
               city payment_mode
order_id                        
ORD11208    Lucknow          UPI
ORD11209  Bengaluru       Wallet

With order_id as the index, .loc can look orders up by their ID. .iloc[0:3, 1:4] takes the first three rows and the columns at positions 1, 2 and 3, regardless of names. Use .loc most of the time, because names make code readable and still work if the column order changes.

Filtering with conditions

A condition on a column, such as sales["city"] == "Pune", gives a boolean Series. Putting it inside .loc[...] (or plain square brackets) keeps the rows where it is True. Combine conditions with & (and), | (or) and ~ (not), and always wrap each condition in brackets.

import pandas as pd

sales = pd.read_csv("sales.csv")
cols = ["order_date", "city", "product", "quantity", "unit_price"]

delhi = sales.loc[sales["city"] == "Delhi", cols]
print(len(delhi), "orders from Delhi")

big_electronics = sales.loc[(sales["category"] == "Electronics") & (sales["unit_price"] > 2000), cols]
print(big_electronics.head(3))

no_upi = sales.loc[~(sales["payment_mode"] == "UPI")]
print(len(no_upi), "orders not paid by UPI")

187 orders from Delhi
    order_date     city     product  quantity  unit_price
24  2025-01-08   Mumbai  Smartwatch       1.0        2999
27  2025-01-08  Lucknow  Smartwatch       1.0        2999
40  2025-01-13    Kochi  Smartwatch       1.0        2999
609 orders not paid by UPI

Writing .loc[condition, columns] filters rows and picks columns in one step. Remember that the city column is still messy, so == "Delhi" misses rows written as DELHI. Also, the "not UPI" count includes the lowercase upi rows and the missing payment modes. Filters are exact, which is one more reason to clean text first.

isin, between and string matching

Long chains of | get hard to read. These helpers keep filters short:

import pandas as pd

sales = pd.read_csv("sales.csv")

south = sales[sales["state"].isin(["Kerala", "Tamil Nadu", "Karnataka", "Telangana"])]
print("South India orders:", len(south))

festive = sales[sales["order_date"].between("2025-10-01", "2025-11-15")]
print("Festive season orders:", len(festive))

delhi_any_spelling = sales[sales["city"].str.strip().str.lower() == "delhi"]
print("Delhi, any spelling:", len(delhi_any_spelling))

books = sales[sales["product"].str.contains("book", case=False)]
print(books["product"].value_counts())

South India orders: 478
Festive season orders: 236
Delhi, any spelling: 190
product
Story Book         34
Python Handbook    23
Name: count, dtype: int64

between() works on the date strings here because ISO dates (2025-10-01) sort correctly as text; with a real date column it works the same way. Cleaning the city text inside the filter (.str.strip().str.lower()) finds the Delhi orders that the exact match missed. case=False makes contains() ignore upper and lower case.

query(): filters as readable text

query() takes the condition as a string, which often reads more like plain English. Inside the string, use and, or, not, and refer to Python variables with @.

import pandas as pd

students = pd.read_csv("students.csv")
cols = ["name", "city", "attendance", "study_hours", "english"]

low = students.query("attendance < 70")
print(low[cols])

min_hours = 4.5
city = "Kochi"
keen = students.query("city == @city and study_hours >= @min_hours and english > 80")
print(keen[cols])

               name    city  attendance  study_hours  english
239  Ishaan Chauhan  Jaipur        66.0          4.0       49
266     Aarav Patel  Jaipur        69.0          2.0       51
397    Saanvi Saini    Pune        66.0          2.5       26
472      Neha Malik    Pune        67.0          1.0       18
              name   city  attendance  study_hours  english
309  Fatima George  Kochi        88.0          5.0       88

query() is handy for exploring, and it saves you from repeating the DataFrame name. Column names with spaces must be wrapped in backticks inside the string, for example "`total marks` > 400".

Changing selected values safely

To change values in a subset of rows, use .loc[row_condition, column] = value on the original DataFrame, in one step.

import pandas as pd

sales = pd.read_csv("sales.csv")

sales.loc[sales["quantity"] < 0, "quantity"] = None          # impossible values become missing
sales.loc[sales["payment_mode"] == "COD", "payment_mode"] = "Cash on Delivery"
print(sales["quantity"].min())
print((sales["payment_mode"] == "COD").sum())

top = sales.nlargest(3, "unit_price")[["product", "unit_price"]]
print(top)

pune = sales[sales["city"] == "Pune"]
pune["discount"] = 0                      # changes only the pune copy
print(sales.loc[sales["city"] == "Pune", "discount"].max())

1.0
0
          product  unit_price
21  Mixer Grinder        3299
32  Mixer Grinder        3299
49  Mixer Grinder        3299
20

The first two assignments changed sales itself. The last part shows an important pandas 3.0 rule called Copy-on-Write: a filtered DataFrame such as pune behaves as a separate copy, so changing it never changes sales. If you want to change the original, select and assign in one .loc step. nlargest() is a quick way to get the top rows by a column.

Common mistakes

• Using and/or between conditions. df[df.a > 1 and df.b < 5] raises a ValueError. Use &, | and brackets, or query().
• Forgetting brackets around conditions. df[df.a > 1 & df.b < 5] is evaluated in the wrong order.
• Exact matches on messy text. == "Delhi" misses DELHI and Delhi . Clean or normalise text first.
• Changing a filtered copy and expecting the original to change. Use df.loc[condition, "col"] = value.

Key points

• .loc[rows, columns] selects by label (slices include the end); .iloc selects by position.
• Boolean conditions filter rows; combine them with &, |, ~ and brackets.
• isin(), between() and .str.contains() keep filters short and readable.
• query() writes the condition as a string; use @name for Python variables.
• Change values with a single df.loc[condition, column] = value; filtered DataFrames behave as copies in pandas 3.0.

import pandas as pd

weather = pd.read_csv("weather.csv")
hot = weather.loc[(weather["city"] == "Delhi") & (weather["temp_max"] >= 40), ["date", "temp_max"]]
print(len(hot))
print(hot.nlargest(5, "temp_max"))

wet = weather.query("city == 'Mumbai' and '2025-07-01' <= date <= '2025-07-31' and rainfall_mm > 50")
print(len(wet))

18
           date  temp_max
131  2025-05-12      42.5
136  2025-05-17      42.2
141  2025-05-22      41.5
152  2025-06-02      41.5
130  2025-05-11      41.3
2