Search code examples
rweb-scrapingrcurlwebscarab

Error: Replacement has length zero R


I 'm trying to scrape http://then.gasbuddy.com/.

I'm running the next code in R

 library(RCurl)
 library(XML)
 doc <- htmlTreeParse('http://www.southcarolinagasprices.com/GasPriceSearch.aspx?typ=adv&fuel=A&srch=0&area=All%20Areas&station=All%20Stations&tme_limit=4')
rootNode <- xmlRoot(doc)

((rootNode[[2]][4])[1][[1]])[[15]][[1]][[11]][[1]][[1]][[2]][[8]][[1]][[2]][[1]][[1]][[1]][[1]][[1]][[1]]

#<div class="p1"/>

x <- matrix(, nrow = 20, ncol = 4)

x[1,1] <- xmlValue(((rootNode[[2]][4])[1][[1]])[[15]][[1]][[11]][[1]][[1]][[2]][[8]][[1]][[2]][[1]][[1]][[1]][[1]][[1]][[1]])

But I have this error

replacement has length zero

How can I subtract p1 and put it in a matrix?


Solution

  • You've come up with an interesting way to get around their price obfuscation. Since they didn't restrict scraping in their Terms of Service, here's one way you can scrape the prices:

    library(xml2)
    
    doc <- read_html('http://www.southcarolinagasprices.com/GasPriceSearch.aspx?typ=adv&fuel=A&srch=0&area=All%20Areas&station=All%20Stations&tme_limit=4')
    
    prices <- xml_find_all(doc, xpath="//div[@class='sp_p']")
    
    sapply(prices, function(x) {
      as.numeric(paste(gsub("d", "\\.", 
                            gsub("^p", "", 
                                 unlist(xml_attrs(xml_find_all(x, "./div"))))),
                       collapse=""))
    })
    
    ##   [1] 1.65 1.65 1.65 1.65 1.65 1.65 1.65 1.65 1.65 1.67 1.68 1.69 1.69 1.69 1.69 1.69 1.69 1.69 1.69
    ##  [20] 1.70 1.71 1.72 1.72 1.73 1.73 1.73 1.73 1.73 1.73 1.73 1.73 1.73 1.74 1.74 1.74 1.74 1.74 1.74
    ##  [39] 1.74 1.74 1.74 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.76 1.76
    ##  [58] 1.76 1.76 1.76 1.76 1.76 1.76 1.76 1.76 1.76 1.76 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77
    ##  [77] 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77
    ##  [96] 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.77 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78
    ## [115] 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78
    ## [134] 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79
    ## [153] 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79
    ## [172] 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79
    ## [191] 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79 1.79