Scraping Lineup Data From Football Reference Using R

I seem to always have a problem scraping reference sites using either Python or R. Whenever I use my normal xpath approach (Python) or Rvest approach in R, the table I want never seems to be picked up by the scraper.

library(rvest)

url = 'https://www.pro-football-reference.com/years/2016/games.htm'

webpage = read_html(url)

table_links = webpage %>% html_node("table") %>% html_nodes("a")
boxscore_links = subset(table_links, table_links %>% html_text() %in% "boxscore")
boxscore_links = as.list(boxscore_links)

for(x in boxscore_links{
  keep = substr(x, 10, 36)
  url2 = paste('https://www.pro-football-reference.com', keep, sep = "") 
  webpage2 = read_html(url2)
  home_team = webpage2 %>% html_nodes(xpath='//*[@id="all_home_starters"]') %>% html_text()
  away_team = webpage2 %>% html_nodes(xpath='//*[@id="all_vis_starters"]') %>% html_text()
  home_starters = webpage2 %>% html_nodes(xpath='//*[(@id="div_home_starters")]') %>% html_text()
  home_starters2 = webpage2 %>% html_nodes(xpath='//*[(@id="div_home_starters")]') %>% html_table()
  #code that will bind lineup tables with some master table -- code to be written later 
}

I'm trying to scrape the starting lineup tables. The first bit of code pulls the urls for all boxscores in 2016, and the for loop goes to each boxscore page with the hopes of extracting the tables led by "Insert Team Here" Starters.

Here's one link for example: 'https://www.pro-football-reference.com/boxscores/201609110rav.htm'

When I run the code above, the home_starters and home_starters2 objects contain zero elements (when ideally it should contain the table or elements of the table I'm trying to bring in).

I appreciate the help!

Solution

I've spent the last three hours trying to figure this out. This is how it shoudl be done. This is given my example but I'm sure you could apply it to yours.

"https://www.pro-football-reference.com/years/2017/" %>%
  read_html() %>%
  html_nodes(xpath = "//comment()") %>% # select comments
  html_text() %>% # extract comment text
  paste(collapse = "") %>% # collapse to single string
  read_html() %>% # reread as HTML
  html_node("table#returns") %>% # select desired node
  html_table()