DataFrame Operations

By the end of this lesson you will be able to select and rename columns, filter rows with several conditions, add calculated columns with withColumn and when, sort, and remove duplicates in a PySpark DataFrame.

If you know pandas or SQL, most of this will feel familiar. The main differences are how you refer to columns and how you combine conditions. Get those two right and the rest is easy.

The sample data

This lesson and the next few use a small orders file, orders.csv:

order_id,customer,city,category,amount,status,order_date
1,Aarav,Pune,Books,499,paid,2026-10-01
2,Priya,Kochi,Electronics,12999,paid,2026-10-01
3,Kabir,Jaipur,Fashion,1299,cancelled,2026-10-02
4,Meera,Pune,Electronics,2450,paid,2026-10-02
5,Rohan,Lucknow,Books,350,paid,2026-10-03
6,Ananya,Kochi,Fashion,899,paid,2026-10-03
7,Farhan,Pune,Fashion,1599,paid,2026-10-04
8,Isha,Jaipur,Electronics,7999,returned,2026-10-04

from pyspark.sql import functions as F

orders = spark.read.csv("orders.csv", header=True,
    schema="order_id INT, customer STRING, city STRING, category STRING, "
           "amount INT, status STRING, order_date DATE")

The line from pyspark.sql import functions as F is in almost every PySpark file. F holds hundreds of built-in functions: F.col, F.sum, F.upper, F.when and more.

Step 1: choose columns with select

There are three common ways to name a column. All three work in select.

orders.select("customer", "amount").show(3)

+--------+------+
|customer|amount|
+--------+------+
|   Aarav|   499|
|   Priya| 12999|
|   Kabir|  1299|
+--------+------+
only showing top 3 rows

You can also write F.col("customer"), orders.amount or orders["city"] inside select; all three name a column. Plain strings are shortest. F.col("name") is needed when you build expressions, such as F.col("amount") * 1.18. You can also compute and name new columns inside select with .alias():

orders.select(
    "customer",
    F.upper("city").alias("city_upper"),
    F.round(F.col("amount") * 1.18).cast("int").alias("with_gst"),
).show(3)

+--------+----------+--------+
|customer|city_upper|with_gst|
+--------+----------+--------+
|   Aarav|      PUNE|     589|
|   Priya|     KOCHI|   15339|
|   Kabir|    JAIPUR|    1533|
+--------+----------+--------+
only showing top 3 rows

Step 2: filter rows

filter and where are the same method. Combine conditions with & (and), | (or) and ~ (not), and put every condition in brackets. Python's and and or do not work on columns.

orders.filter(
    (F.col("status") == "paid") & (F.col("amount") >= 1000)
).select("order_id", "customer", "amount").show()

+--------+--------+------+
|order_id|customer|amount|
+--------+--------+------+
|       2|   Priya| 12999|
|       4|   Meera|  2450|
|       7|  Farhan|  1599|
+--------+--------+------+

Other handy conditions: F.col("city").isin("Pune", "Kochi"), F.col("amount").between(500, 2000), F.col("customer").startswith("A"), and F.col("amount").isNull(). You can also write a SQL-style string: orders.where("status = 'paid' AND amount >= 1000").

Here is the same filter in pandas. Notice that pandas also needs & and brackets:

import io
import pandas as pd

csv_text = """order_id,customer,city,category,amount,status,order_date
1,Aarav,Pune,Books,499,paid,2026-10-01
2,Priya,Kochi,Electronics,12999,paid,2026-10-01
3,Kabir,Jaipur,Fashion,1299,cancelled,2026-10-02
4,Meera,Pune,Electronics,2450,paid,2026-10-02
5,Rohan,Lucknow,Books,350,paid,2026-10-03
6,Ananya,Kochi,Fashion,899,paid,2026-10-03
7,Farhan,Pune,Fashion,1599,paid,2026-10-04
8,Isha,Jaipur,Electronics,7999,returned,2026-10-04
"""
orders = pd.read_csv(io.StringIO(csv_text))
big_paid = orders[(orders["status"] == "paid") & (orders["amount"] >= 1000)]
print(big_paid[["order_id", "customer", "amount"]].to_string(index=False))

order_id customer  amount
       2    Priya   12999
       4    Meera    2450
       7   Farhan    1599

Step 3: add, change, rename and drop columns

withColumn(name, expression) adds a column, or replaces it if the name already exists. F.when(...).otherwise(...) works like SQL CASE WHEN or an if-else for every row.

labelled = (
    orders
    .withColumn("gst", F.round(F.col("amount") * 0.18, 2))
    .withColumn(
        "size",
        F.when(F.col("amount") >= 5000, "large")
         .when(F.col("amount") >= 1000, "medium")
         .otherwise("small"),
    )
    .withColumnRenamed("customer", "buyer")
    .drop("order_date", "category")
)
labelled.show()

+--------+------+-------+------+---------+-------+------+
|order_id| buyer|   city|amount|   status|    gst|  size|
+--------+------+-------+------+---------+-------+------+
|       1| Aarav|   Pune|   499|     paid|  89.82| small|
|       2| Priya|  Kochi| 12999|     paid|2339.82| large|
|       3| Kabir| Jaipur|  1299|cancelled| 233.82|medium|
|       4| Meera|   Pune|  2450|     paid|  441.0|medium|
|       5| Rohan|Lucknow|   350|     paid|   63.0| small|
|       6|Ananya|  Kochi|   899|     paid| 161.82| small|
|       7|Farhan|   Pune|  1599|     paid| 287.82|medium|
|       8|  Isha| Jaipur|  7999| returned|1439.82| large|
+--------+------+-------+------+---------+-------+------+

when conditions are checked from top to bottom, and the first true one wins, exactly like if / elif / else. To add many columns at once, withColumns({"a": ..., "b": ...}) takes a dictionary and is tidier than a long chain.

Step 4: sort, deduplicate and limit

(orders
    .select("city", "category")
    .dropDuplicates()
    .orderBy("city", F.desc("category"))
    .show())

+-------+-----------+
|   city|   category|
+-------+-----------+
| Jaipur|    Fashion|
| Jaipur|Electronics|
|  Kochi|    Fashion|
|  Kochi|Electronics|
|Lucknow|      Books|
|   Pune|    Fashion|
|   Pune|Electronics|
|   Pune|      Books|
+-------+-----------+

• dropDuplicates() removes repeated rows; dropDuplicates(["city"]) keeps one row per city (which one is not guaranteed).
• orderBy (or sort) sorts the whole DataFrame. Sorting needs a shuffle, so on big data sort only when you need to.
• limit(n) keeps the first n rows and returns a DataFrame; show(n) only prints.

Here is the when logic in pandas, using numpy.select, which you can run and compare:

import numpy as np
import pandas as pd

orders = pd.DataFrame({
    "buyer": ["Aarav", "Priya", "Kabir", "Meera"],
    "amount": [499, 12999, 1299, 2450],
})
orders["size"] = np.select(
    [orders["amount"] >= 5000, orders["amount"] >= 1000],
    ["large", "medium"],
    default="small",
)
print(orders)

   buyer  amount    size
0  Aarav     499   small
1  Priya   12999   large
2  Kabir    1299  medium
3  Meera    2450  medium

Common mistakes

• Using and / or instead of & / |. Spark 4.3 stops with PySparkValueError: [CANNOT_CONVERT_COLUMN_INTO_BOOL] Cannot convert column into bool ... please use '&' for 'and', '|' for 'or', '~' for 'not'.
• Missing brackets. F.col("a") == 1 & F.col("b") == 2 is evaluated in the wrong order. Write (F.col("a") == 1) & (F.col("b") == 2).
• Forgetting to assign the result. DataFrames are immutable; each method returns a new one.
• Long chains of withColumn in a loop (hundreds of columns). It makes huge plans; use one select or withColumns instead.

Key points

• select picks or computes columns; .alias() names them.
• filter/where keeps rows; combine conditions with &, |, ~ and brackets.
• withColumn adds or replaces a column; F.when().otherwise() handles conditions.
• withColumnRenamed, drop, dropDuplicates, orderBy and limit cover daily cleanup.
• Always assign the result: every operation returns a new DataFrame.

(orders
    .filter(F.col("status") != "paid")
    .withColumn("refund_due",
        F.when(F.col("status") == "returned", F.col("amount")).otherwise(0))
    .select("order_id", "customer", "status", "refund_due")
    .orderBy("order_id")
    .show())

+--------+--------+---------+----------+
|order_id|customer|   status|refund_due|
+--------+--------+---------+----------+
|       3|   Kabir|cancelled|         0|
|       8|    Isha| returned|      7999|
+--------+--------+---------+----------+