RDDs vs DataFrames

By the end of this lesson you will be able to explain what an RDD is, how lineage lets Spark rebuild lost data, write a word count with both the RDD API and the DataFrame API, and choose the right one for a job.

Spark has two main ways to hold distributed data. The older one is the RDD. The newer one is the DataFrame, which is built on top of RDDs. Almost all new code uses DataFrames, but you will still meet RDDs in older projects, interviews and some special cases.

What is an RDD?

RDD stands for Resilient Distributed Dataset. Each word matters:

• Resilient: if a partition is lost, Spark can rebuild it.
• Distributed: the data is split into partitions across executors.
• Dataset: it is a collection of items, such as numbers, strings or tuples.

An RDD has no column names and no types that Spark understands. To Spark, each item is just a Python object. You tell Spark what to do with functions such as map, filter and reduceByKey.

Lineage: the recipe, not the dish

An RDD does not store how its data looks. It stores how to make it: "start from this file, split every line, keep long words, count them". This recipe is called lineage. If an executor dies and a partition is lost, Spark simply cooks that partition again from the recipe.

Think of a halwai who writes down every recipe. If one tray of laddoos falls on the floor, he does not panic. He makes that one tray again, using the same recipe and ingredients.

Step 1: build a mini RDD with lineage

This small class imitates an RDD. Transformations only add steps to the recipe. Nothing runs until collect().

class MiniRDD:
    def __init__(self, partitions, steps=()):
        self.partitions = partitions      # the source data, already split
        self.steps = list(steps)          # the lineage: list of (name, function)

    def map(self, f):
        return MiniRDD(self.partitions, self.steps + [("map", lambda part: [f(x) for x in part])])

    def filter(self, f):
        return MiniRDD(self.partitions, self.steps + [("filter", lambda part: [x for x in part if f(x)])])

    def compute(self, i):
        part = self.partitions[i]
        for name, step in self.steps:
            part = step(part)
        return part

    def collect(self):
        return [x for i in range(len(self.partitions)) for x in self.compute(i)]

marks = MiniRDD([[45, 78, 91], [33, 67, 88], [52, 99, 40]])
passed = marks.filter(lambda m: m >= 50).map(lambda m: m + 5)   # grace marks

print("lineage:", [name for name, _ in passed.steps])
print("result :", passed.collect())
print("rebuild partition 1 only:", passed.compute(1))

lineage: ['filter', 'map']
result : [83, 96, 72, 93, 57, 104]
rebuild partition 1 only: [72, 93]

compute(1) rebuilt one partition from the recipe without touching the others. That is exactly how Spark recovers a lost partition.

Word count with the RDD API

Here is the classic word count in PySpark's RDD API. spark.sparkContext (often called sc) is the entry point for RDDs.

sc = spark.sparkContext
lines = sc.parallelize([
    "rain in pune today",
    "rain again in pune",
    "kochi rain",
])
counts = (
    lines.flatMap(lambda line: line.split())
         .map(lambda word: (word, 1))
         .reduceByKey(lambda a, b: a + b)
)
print(sorted(counts.collect()))

[('again', 1), ('in', 2), ('kochi', 1), ('pune', 2), ('rain', 3), ('today', 1)]

flatMap turns each line into many words. map makes (word, 1) pairs. reduceByKey adds the ones for each word, combining inside each partition first (like a MapReduce combiner) and then across partitions. We wrap the result in sorted() because RDD results come back in no guaranteed order.

Word count with the DataFrame API

A DataFrame has named columns with known types. Because Spark understands the columns, it can optimise your query before running it. Spark's optimiser is called Catalyst. It reorders steps, removes unused columns and pushes filters close to the data source.

from pyspark.sql import functions as F

lines = spark.createDataFrame(
    [("rain in pune today",), ("rain again in pune",), ("kochi rain",)], "line STRING"
)
counts = (
    lines.select(F.explode(F.split("line", " ")).alias("word"))
         .groupBy("word").count()
         .orderBy(F.desc("count"), "word")
)
counts.show()

+-----+-----+
| word|count|
+-----+-----+
| rain|    3|
|   in|    2|
| pune|    2|
|again|    1|
|kochi|    1|
|today|    1|
+-----+-----+

F.split cuts each line into an array of words. F.explode turns one row with an array into one row per word. Then groupBy().count() counts. Every function here is a built-in Spark function, so the whole job runs inside the JVM with no Python code per row.

The pandas version uses the same idea, and you can run it here:

import pandas as pd

lines = pd.DataFrame({"line": ["rain in pune today", "rain again in pune", "kochi rain"]})
words = lines["line"].str.split(" ").explode().rename("word")
counts = words.value_counts().rename_axis("word").reset_index()
counts = counts.sort_values(["count", "word"], ascending=[False, True], ignore_index=True)
print(counts)

    word  count
0   rain      3
1     in      2
2   pune      2
3  again      1
4  kochi      1
5  today      1

Why DataFrames are usually faster in PySpark

With RDDs, every Python lambda runs in a Python worker. Spark must turn each row into bytes, send it from the JVM to Python, run your function, and send the result back. That round trip is called serialisation, and it is slow.

With DataFrames, your Python code only describes the query. The real work runs in the JVM, using Spark's optimised engine (called Tungsten) that stores data in a compact binary form. So PySpark DataFrame code is usually about as fast as the same code in Scala.

Scala and Java also have Datasets, which are DataFrames with compile-time types. Python has no Datasets because Python is not statically typed; in PySpark you use DataFrames.

Moving between them

You can convert in both directions when you need to: df.rdd gives an RDD of Row objects, and rdd.toDF(["word", "n"]) turns an RDD of tuples into a DataFrame. Use this rarely. If you find yourself writing df.rdd.map(...), check first whether a built-in function in pyspark.sql.functions already does the job.

Common mistakes

• Starting new projects with RDDs. DataFrames are faster in PySpark and easier to read.
• Using groupByKey then summing. It ships every value across the network; reduceByKey combines first.
• Expecting order from collect(). Sort explicitly.

Key points

• An RDD is a resilient, distributed collection of Python objects without a schema.
• Lineage is the recipe of transformations; Spark uses it to rebuild lost partitions.
• RDD transformations such as flatMap, map and reduceByKey run Python code per row.
• DataFrames have typed columns, the Catalyst optimiser and JVM execution.
• Prefer DataFrames and built-in functions; use RDDs only for special low-level needs.

bills = MiniRDD([[120, 450], [80, 999, 30], [610]])
with_gst = bills.filter(lambda b: b >= 100).map(lambda b: round(b * 1.18))

print("lineage:", [name for name, _ in with_gst.steps])
print("result :", with_gst.collect())
print("partition 0:", with_gst.compute(0))

lineage: ['filter', 'map']
result : [142, 531, 1179, 720]
partition 0: [142, 531]