Missing Values

Real data has gaps. A sensor was switched off, a student was absent, a shopkeeper left a cell blank. R marks these gaps with NA, short for not available. NA is not zero and not an empty string: it means "a value exists but we do not know it". Learning how NA behaves saves you from two classic errors: a mean() that mysteriously returns NA, and a report that quietly drops half the rows.

In this lesson you will see how NA spreads, how to find and count it, how to remove or fill it, and how to tell NA, NaN and NULL apart.

How NA behaves

Any calculation that involves an unknown value gives an unknown result. That is logical, but it surprises everyone the first time:

x <- c(12, NA, 30, NA, 18)
is.na(x)
sum(is.na(x))
mean(x)
mean(x, na.rm = TRUE)
NA > 5
NA == NA
x == NA
NA & FALSE
NA | TRUE

[1] FALSE  TRUE FALSE  TRUE FALSE
[1] 2
[1] NA
[1] 20
[1] NA
[1] NA
[1] NA NA NA NA NA
[1] FALSE
[1] TRUE

• is.na() is the only reliable test for missing values. x == NA gives all NA, because "is this unknown value equal to that unknown value?" has no answer.
• sum(is.na(x)) counts the gaps, since TRUE is 1.
• mean(x) is NA. Add na.rm = TRUE ("remove NA") and the mean uses the three known values: 20. sum(), median(), sd(), min() and max() all accept na.rm.
• NA & FALSE is FALSE and NA | TRUE is TRUE, because the answer is the same whatever the unknown value turns out to be.

NA, NaN and NULL

Three different things look similar:

length(c(1, NA, 3))
length(c(1, NULL, 3))
is.na(NaN)
is.nan(NA)
typeof(NA)
typeof(NA_character_)
c(NA, "Pune")

[1] 3
[1] 2
[1] TRUE
[1] FALSE
[1] "logical"
[1] "character"
[1] NA     "Pune"

NA adapts to the vector it sits in: inside a character vector it is a missing string (printed without quotes), inside numbers a missing number. The typed forms NA_character_, NA_integer_ and NA_real_ exist for when you need to be explicit, for example in ifelse().

Finding gaps in a data set

airquality has missing ozone and solar readings. These functions show where:

colSums(is.na(airquality))
sum(complete.cases(airquality))
aq <- na.omit(airquality)
nrow(aq)
mean(airquality$Ozone, na.rm = TRUE)
tapply(is.na(airquality$Ozone), airquality$Month, sum)

  Ozone Solar.R    Wind    Temp   Month     Day 
     37       7       0       0       0       0 
[1] 111
[1] 111
[1] 42.12931
 5  6  7  8  9 
 5 21  5  5  1 

• colSums(is.na(df)) is the one-line missing-value report: 37 ozone and 7 solar readings are missing.
• complete.cases() is TRUE for rows with no gaps at all; only 111 of the 153 days are complete.
• na.omit() drops every row that has any NA. It is convenient but blunt: here it throws away 42 days, including days where only the solar reading was missing.
• The tapply() line shows that June has 21 missing ozone readings. Gaps are often clustered, and knowing where they are matters before you decide what to do.

Removing or filling

There are three honest choices. Drop the missing rows if there are few of them. Use na.rm = TRUE so each statistic uses all the data it has. Or impute: fill the gaps with a reasonable value, such as the median. Imputation keeps rows but invents data, so say so in your report.

oz <- airquality$Ozone
oz[is.na(oz)] <- median(oz, na.rm = TRUE)
summary(airquality$Ozone)
summary(oz)
city <- c("Pune", NA, "Kochi", "Pune")
table(city)
table(city, useNA = "ifany")
cor(airquality$Ozone, airquality$Temp)
cor(airquality$Ozone, airquality$Temp, use = "complete.obs")

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.     NAs 
   1.00   18.00   31.50   42.13   63.25  168.00      37 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00   21.00   31.50   39.56   46.00  168.00 
city
Kochi  Pune 
    1     2 
city
Kochi  Pune  <NA> 
    1     2     1 
[1] NA
[1] 0.6983603

• oz[is.na(oz)] <- median(oz, na.rm = TRUE) replaces each gap with the median, using the subsetting-assignment trick from Indexing & Subsetting. Note how the mean dropped from 42.1 to 39.6: filling 37 days with the median pulled the average down, which is exactly the kind of side effect you must mention.
• summary() shows the NA count on its own; after imputation that column disappears.
• table() ignores NA unless you ask with useNA = "ifany".
• cor() has no na.rm; it uses use = "complete.obs" instead. Different functions spell the option differently, so check the help page when a result is NA.

Common mistakes

• Testing with == NA. Always use is.na().
• Filtering with a condition that contains NA. df[df$x > 5, ] returns junk rows; use which() or subset().
• Reaching for na.omit() first. It drops whole rows for one missing cell; often na.rm = TRUE is enough.
• Treating 0 or "" as missing, or NA as zero. sum(c(10, NA)) is NA, not 10, and a blank cell is not a zero sale.

Key points

• NA means unknown; almost any calculation with it returns NA.
• Test with is.na(), count with sum(is.na(x)) or colSums(is.na(df)).
• na.rm = TRUE makes summary functions skip gaps; cor() uses use = "complete.obs".
• complete.cases() and na.omit() drop rows with gaps; x[is.na(x)] <- value fills them.
• NaN is a failed calculation and counts as NA; NULL is nothing and takes no space.

steps <- c(8200, NA, 6300, 12000, NA, 9100, 4500, NA, 11300, 7600)
sum(is.na(steps))
avg <- mean(steps, na.rm = TRUE)
avg
filled <- steps
filled[is.na(filled)] <- round(avg)
filled
mean(filled)

[1] 3
[1] 8428.571
 [1]  8200  8429  6300 12000  8429  9100  4500  8429 11300  7600
[1] 8428.7