I have a large (millions of observations) dataset and I have used feols to run a linear model. That model has dropped many observations from consideration for missing values. I have recovered the row numbers that were dropped using $obs_selection, but I cannot determine how to use the list that $obs_selection produced to remove the dropped observations from my original dataset.
Ultimately, I would like to remove the dropped observations then join the $residuals to the original data.
I originally tried this (generally - specified in code below):
df[-object$obs_selection]
but this generates an error "Error in -rows_to_delete : invalid argument to unary operator" and is similar to the solution (and error I get) in the answer to this question: How do you retrieve the estimation sample in R?
In the sample data below, there are five observations omitted in the model due to missing values. How would I use fake_lm$obs_selection to remove the dropped observations from my original dataset?
Thank you!
Data:
structure(list(ExamType = c("A", "B", "C", "D", "E", "F", "G",
"A", "B", "C", "D", "E", "F", "G", "A", "B", "C", "D", "E", "F",
"G", "A", "B", "C", "D", "E", "F", "G", "A", "B"), ExamScore = c(1L,
2L, 2L, 3L, 1L, 4L, 4L, 5L, 2L, 1L, 4L, 3L, 2L, 5L, 1L, NA, 3L,
2L, 1L, 2L, 5L, 4L, 4L, 3L, 1L, 2L, 5L, 4L, 3L, 1L), State = c("CA",
"CA", "AL", "AK", "AK", "CA", "AL", "CO", "AL", "CA", "CA", "CA",
"CO", "CO", "AR", "AR", "AK", "CA", "CA", "CT", "AL", "CA", "AK",
"CA", "CA", "AL", "AR", "AR", "CA", "CT"), Male = c(1L, 1L, 0L,
0L, 1L, 0L, 0L, 0L, 1L, 1L, NA, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 1L,
0L, 0L, 1L, 0L, 0L, 0L, 1L, 1L, 0L, 1L, 1L), White = c(1L, 0L,
0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L,
0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L), Black = c(0L,
1L, 0L, NA, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L,
0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 0L), Latinx = c(0L,
0L, 0L, 0L, 1L, 0L, NA, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L,
0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L), X2.Race = c(0L,
0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L,
0L, 0L, 0L, 0L, 0L, 0L, NA, 0L, 0L, 0L, 0L, 0L, 0L)), row.names = c(NA,
30L), class = "data.frame")
Code:
library(fixest)
fake_lm <- feols(ExamScore ~ Male + White + Black + Latinx + X2.Race | State, fake_data)
summary(fake_lm)
#These are the dropped observations
rows_to_delete <- fake_lm$obs_selection
# I would like to clean them from my dataset (fake_data), but this
# generates the error
fake_data[-rows_to_delete]
# Ultimately, once the original dataset only contains those used in the model, I'll add
# residuals as a column in my dataset
fake_data$resid <- fake_lm$residuals
After some pain, I figured this out.
the list of vectors of integers can be cast as a dataframe, and from then on, this becomes a tidyverse question.
Rewriting some of the code from above...
library(tidyverse)
fake_data <- fake_data %>% rowid_to_column()
rows_to_delete <- as.data.frame(fake_lm$obs_selection)
row_to_delete$obsRemoved <- rows_to_delete$obsRemoved * -1
colnames(rows_to_delete) <- c("rowid")
clean_fake_data <- anti_join(fake_data,rows_to_delete,by="rowid")
From here, you can add a column of residuals as desired.