Search code examples
rjsonweb-scrapinghttp-posthttr

Scraping Data using R - Form with List in POST


I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.

Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.

Request Data as shown in Safari Network Tab

MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501

HTTR Post to mimic the above

form <- list(
    section = list(1,4),
    period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())

Then the code runs without error and the API does return results but seems to have ignored the specific parameters.

The output from verbose suggests the parameters are being included:

{"section":[1,4],"period":20170501}

Adjustment for Form Type

I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.

Error in vapply(elements, encode, character(1)) : 
  values must be length 1,
 but FUN(X[[1]]) result is length 2

Solution

  • Fixed! I had to use Query instead of Body and add the [] after each item.

    query <- list(
        "section[]" = 1,
        "section[]" = 4,
        "period[]" = 20170501
    )
    
    resp<-POST(URL, query=query, verbose())