I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
{"section":[1,4],"period":20170501}
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())