Search code examples
rweb-scrapingrvestxml2

Web scraping an HTML table in R taking huge time

Folks, I'm trying to scrape a link which has just around 1000+ records but its taking hours to get them..Wondering if I'm doing anything wrong or ways to load this into a table.

urlString = "https://www.valueresearchonline.com/funds/selector-data/primary-category/1/equity/?tab=snapshot&output=html-data"
urlString <- URLencode(paste0(urlString,""))

#Reading the HTML code from the website and process the text
getHTML <- xml2::read_html(urlString, options = "HUGE")

#This one keeps running endlessly and doesn't load the table
mytable <- data.frame(getHTML %>% html_table(fill = T, trim = T))

Any help would be appreciated. Thanks

Solution

  • The link is a JSON file. You need to read it by jsonlite first. And the HTML data is at html_data node, you read this node by read_html:

    json <- jsonlite::fromJSON("https://www.valueresearchonline.com/funds/selector-data/primary-category/1/equity/?tab=snapshot&output=html-data")
getHTML <- xml2::read_html(json$html_data)
mytable <- data.frame(getHTML %>% html_table(fill = T, trim = T))