Search code examples
rweb-scrapinghttr

HTTR Error while reading from server


I'm doing something that I think should be really simple with r and in particular httr but can't get it to work. I want to visit (http://karpo.gov.bc.ca/royp-bin/phcgi.exe?PH_QKC=WRMSQFELD&PH_APP=RMSprodApp&PH_HTML=WRMSQ_FACI_PLANT.HTM) and submit a plant code and get the result. After reading the documentation I think it should be

res = GET(url = 'http://karpo.gov.bc.ca/royp-bin/phcgi.exe?PH_QKC=WRMSQFELD&PH_APP=RMSprodApp&PH_HTML=WRMSQ_FACI_PLANT.HTM')

res = PUT(url = 'http://karpo.gov.bc.ca/royp-bin/phcgi.exe',  body = list('WEB_FACI_PLANT_VIEW:PLANT' = 160))

But I'm getting an error; "The request failed while reading the Web Server"

Any help?


Solution

  • I used the curlconverter package to build the request:

    library(httr)
    library(rvest)
    library(dplyr)
    
    res <- POST(url = "http://karpo.gov.bc.ca/royp-bin/phcgi.exe", 
                add_headers(Origin = "http://karpo.gov.bc.ca", 
                            `Accept-Encoding` = "gzip, deflate", 
                            `Accept-Language` = "en-US,en;q=0.8", 
                            `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.70 Safari/537.36", 
                            Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 
                            Referer = "http://karpo.gov.bc.ca/royp-bin/phcgi.exe?PH_QKC=WRMSQFELD&PH_APP=RMSprodApp&PH_HTML=WRMSQ_FACI_PLANT.HTM"),
                body=list(PH_HTML = "WRMSQ_FACI_PLANT.HTM", 
                          PH_QKC = "WRMSQ_FACI_PLANT", 
                          PH_APP = "RMSPRODAPP", 
                          PH_SEARCH = "PH_SEARCH_STARTPH_SEARCH_END", 
                          PH_RECORDID = "PH_RECORDID_STARTPH_RECORDID_END", 
                          `WEB_FACI_PLANT_VIEW:FACILITY` = "", 
                          `WEB_FACI_PLANT_VIEW:PLANT` = "160", 
                          PH_ACTION = "Search"), 
                encode="form")
    
    pg <- content(res, as="parsed")
    glimpse(html_table(pg)[[2]])
    ## Observations: 11
    ## Variables: 6
    ## $ FacilityCode   <int> 160, 161, 1647, 2355, 4557, 4582, 5857, 7200, 8294, 9049...
    ## $ FacilityName   <chr> "aitken gas aitkencreek d-044-l/094-a-13 002", "aitken g...
    ## $ LinkEffective  <chr> "1993 06", "1993 06", "2001 04", "2001 03", "2001 03", "...
    ## $ LinkTerminated <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
    ## $ PlantCode      <int> 160, 160, 160, 160, 160, 160, 160, 160, 160, 160, 160
    ## $ PlantName      <chr> "aitken gas aitkencreek d-044-l/094-a-13 002", "aitken g...