I'm trying to, ultimately, scrape tables from several different URLs (within the same parent site) in R.
First, I assume I have to scrape the individual game links under "Playoff Series" from https://www.basketball-reference.com/playoffs/NBA_2017.html -- the xpath for that table of links is //*[@id="all_all_playoffs"]
then, I want to scrape tables from each of those individual game links (looks like this: https://www.basketball-reference.com/boxscores/201705170BOS.html) -- the tables I want are the "basic box score stats" for each team.
(I plan on repeating this for several different years, so typing in each URL--like I do below--is not very efficient)
so far, I can only figure out how to scrape tables from one url (or one game) at a time:
games <- c("201705190BOS","201705190BOS","201705210CLE","201705230CLE","201705250BOS")
urls <- paste0("https://www.basketball-reference.com/boxscores/", games, ".html")
get_table <- function(url) {
url %>%
read_html() %>%
html_nodes(xpath = '//*[@id="div_box_cle_basic"]/table[1]') %>%
html_nodes(xpath = '//*[@id="div_box_bos_basic"]/table[1]') %>%
html_table()
}
results <- sapply(urls, get_table)
This works for me, give it a try!
library(rvest)
page <- read_html('https://www.basketball-reference.com/playoffs/NBA_2017.html')
#get all links in the playoff section
playoffs <- page %>%
html_node('#div_all_playoffs') %>%
html_nodes('a') %>%
html_attr('href')
#limit to those that are actually links to boxscores
playoffs <- playoffs[grep('boxscore', playoffs)]
#loop to scrape each game
allGames <- list()
for(j in 1:length(playoffs)){
box <- read_html(paste0('https://www.basketball-reference.com/', playoffs[j]))
#tables are named based on which team is there, get all html id's to find which one we want
atrs <- box %>%
html_nodes('div') %>%
html_attr('id')
#limit to only names that include "basic" and "all"
basicIds <- atrs[grep('basic', atrs)] %>%
.[grep('all', .)]
#loop to scrape both tables (1 for each team)
teams <- list()
for(i in 1:length(basicIds)){
#grab table for team
table <- box %>%
html_node(paste0('#',basicIds[i])) %>%
html_node('.stats_table') %>%
html_table()
#parse table into starters and reserves tables
startReserve <- which(table[,1] == 'Reserves')
starters <- table[2:(startReserve-1),]
colnames(starters) <- table[1,]
reserves <- table[(startReserve + 1):nrow(table),]
colnames(reserves) <- table[startReserve,]
#extract team name
team <- gsub('all_box_(.+)_basic', '\\1', basicIds[i])
#make named list using team name
assign(team, setNames(list(starters, reserves), c('starters', 'reserves')))
teams[[i]] <- team
}
#find game identifier
game <- gsub('/boxscores/(.+).html', '\\1', playoffs[j])
#make list of both teams, name list using game identifier
assign(paste0('game_',game), setNames(list(eval(parse(text=teams[[1]])), eval(parse(text=teams[[2]]))), c(teams[[1]], teams[[2]])))
#add to allGames
allGames <- append(allGames, setNames(list(eval(parse(text = paste0('game_', game)))), paste0('game_', game)))
}
#clean up everything but allGames
rm(list = ls()[-grep('allGames', ls())])
The output is a list of lists. This isn't great, but the data you want is inherently hierarchical: each game has 2 teams and each team has 2 tables (starters and reserves). So, the final object looks like:
-allGames
----Game1
-------Team1
----------Starters
----------Reserves
-------Team2
----------Starters
----------Reserves
----Game2 ...
For example, show the table with data on the starters for Cleveland in the last game of the final with:
> allGames$game_201706120GSW$cle$starters
Starters MP FG FGA FG% 3P 3PA 3P% FT FTA FT% ORB DRB TRB AST STL BLK TOV PF PTS +/-
2 LeBron James 46:13 19 30 .633 2 5 .400 1 4 .250 2 11 13 8 2 1 2 3 41 -13
3 Kyrie Irving 41:47 9 22 .409 1 2 .500 7 7 1.000 1 1 2 6 2 0 4 3 26 +4
4 J.R. Smith 40:49 9 11 .818 7 8 .875 0 1 .000 0 3 3 1 0 2 0 2 25 -2
5 Kevin Love 29:55 2 8 .250 0 3 .000 2 5 .400 3 7 10 2 0 1 0 2 6 -23
6 Tristan Thompson 29:52 6 8 .750 0 0 3 4 .750 4 4 8 3 1 1 3 1 15 -7