Indexing & Subsetting

Subsetting means picking out part of an object: some elements of a vector, some rows and columns of a data frame, one element of a list. It is the skill you use most in R, because almost every analysis starts with "only the rows where..." or "just these columns".

You have already seen pieces of it. This lesson puts the rules together, shows how to filter data frames, and explains the NA trap that catches many learners.

Four ways to index a vector

Inside [ ] you can use four kinds of index:

x <- c(a = 10, b = 20, c = 30, d = 40)
x[c(1, 3)]
x[-c(1, 3)]
x[c(TRUE, FALSE, TRUE, FALSE)]
x[c("d", "a")]
x[x > 15]
x[10]

 a  c 
10 30 
 b  d 
20 40 
 a  c 
10 30 
 d  a 
40 10 
 b  c  d 
20 30 40 
<NA> 
  NA 

Asking for a position that does not exist, like x[10], does not give an error; it gives NA. That can hide mistakes, so check lengths when results look odd. You cannot mix positive and negative positions in one index.

Rows and columns of a data frame

A data frame takes two indexes, [rows, columns], and each one can use any of the four kinds. A logical condition in the row position is how you filter:

mtcars[mtcars$mpg > 30, c("mpg", "hp", "wt")]
mtcars["Fiat 128", "mpg"]
mtcars[1:3, ]
head(mtcars[order(mtcars$hp, decreasing = TRUE), c("hp", "mpg")], 3)

                mpg  hp    wt
Fiat 128       32.4  66 2.200
Honda Civic    30.4  52 1.615
Toyota Corolla 33.9  65 1.835
Lotus Europa   30.4 113 1.513
[1] 32.4
               mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4     21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710    22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
                hp  mpg
Maserati Bora  335 15.0
Ford Pantera L 264 15.8
Duster 360     245 14.3

• mtcars$mpg > 30 is TRUE for four cars, so only their rows are kept, with three chosen columns.
• mtcars has car names as row names, so mtcars["Fiat 128", "mpg"] picks one cell by names.
• An empty column position (mtcars[1:3, ]) keeps every column.
• The last line sorts by horsepower, keeps two columns and shows the top three: the most powerful cars.

Combine conditions with & (and) and | (or): mtcars[mtcars$cyl == 4 & mtcars$am == 1, ] keeps 4-cylinder cars with a manual gearbox.

The NA trap and how to avoid it

airquality has missing ozone readings. Watch what happens when you filter on that column:

nrow(airquality[airquality$Ozone > 100, ])
airquality[airquality$Ozone > 100, ][1:4, ]
airquality[which(airquality$Ozone > 100), ]
subset(airquality, Ozone > 100 & Month == 8, select = c(Ozone, Temp, Day))

[1] 44
     Ozone Solar.R Wind Temp Month Day
NA      NA      NA   NA   NA    NA  NA
NA.1    NA      NA   NA   NA    NA  NA
NA.2    NA      NA   NA   NA    NA  NA
NA.3    NA      NA   NA   NA    NA  NA
    Ozone Solar.R Wind Temp Month Day
30    115     223  5.7   79     5  30
62    135     269  4.1   84     7   1
86    108     223  8.0   85     7  25
99    122     255  4.0   89     8   7
101   110     207  8.0   90     8   9
117   168     238  3.4   81     8  25
121   118     225  2.3   94     8  29
    Ozone Temp Day
99    122   89   7
101   110   90   9
117   168   81  25
121   118   94  29

Only 7 days had ozone above 100, yet the first line says 44 rows. Why? For a missing reading, NA > 100 is NA (unknown), and indexing with NA returns a row full of NA values. The 37 missing days became junk rows named NA, NA.1 and so on.

Two safe ways to filter:

• which() turns the condition into row positions and ignores NA, so you get exactly the 7 real days.
• subset(data, condition, select = ...) drops rows where the condition is NA and lets you write column names without airquality$. It is handy at the console. Inside functions you write, prefer [ ] with which(), because subset() looks up column names in a way that can surprise you.

Changing values by subsetting

Subsetting also works on the left side of <-, which changes only the selected elements. Here impossible marks (below 0 or above 100) are replaced with NA:

scores <- c(45, -1, 78, 102, 66)
scores[scores < 0 | scores > 100] <- NA
scores
cars <- head(mtcars, 3)
cars[, "mpg"]
cars[, "mpg", drop = FALSE]
cars["mpg"]
cars[["mpg"]]

[1] 45 NA 78 NA 66
[1] 21.0 21.0 22.8
               mpg
Mazda RX4     21.0
Mazda RX4 Wag 21.0
Datsun 710    22.8
               mpg
Mazda RX4     21.0
Mazda RX4 Wag 21.0
Datsun 710    22.8
[1] 21.0 21.0 22.8

The second part shows how to choose between a vector and a one-column data frame:

This is the same [ ] versus [[ ]] rule you learned in Lists: a data frame is a list of columns.

Common mistakes

• Filtering with NA in the condition. Use which() or subset(), or handle the missing values first.
• Forgetting the comma in a data frame. df[df$x > 5] tries to select columns. Write df[df$x > 5, ].
• Using = instead of ==. df[df$cyl = 4, ] is an error; comparison needs ==.
• Writing 0.1 style positions. Positions are whole numbers; x[1.9] silently becomes x[1].

Key points

• Index vectors with positive positions, negative positions, logical vectors or names.
• Data frames take [rows, columns]; a logical condition in the row position filters rows.
• Conditions that contain NA produce junk NA rows; filter with which() or subset().
• Subsetting on the left of <- changes only the chosen elements.
• df[, "col"] and df[["col"]] give vectors; df["col"] and drop = FALSE keep a data frame.

aq <- airquality
aq[aq$Month == 6 & aq$Temp >= 90, c("Month", "Day", "Temp")]
sum(is.na(aq$Solar.R))
aq$Wind[aq$Wind > 20] <- 20
max(aq$Wind)

   Month Day Temp
40     6   9   90
42     6  11   93
43     6  12   92
[1] 7
[1] 20