Search code examples
rpostcurlhttp-posthttr

Download File in R with POST while sending data

I try to download a file, to get it from the server I need to send data at the same time. With curl on the command line it works fine:

curl "https://www.ishares.com/us/product-screener-download.dl" --data "productView=ishares&portfolios=239561-239855"

Unfortunately I don't get it to work with R. I tried with download.file, download.file with libcurl, curl_download and with httr. (download.file with curl or wget does not work as I'm on a window machine.)

What I tried and didn't work with curl:

library("curl")
handle <- new_handle()
handle_setopt(handle, customrequest = "POST")
handle_setform(handle, productView="ishares",portfolios="239561-239855")
curl_download("https://www.ishares.com/us/products/etf-product-list", "./data/ishares-us-etf.xls", handle=handle)

What I tried and didn't work with httr:

library(httr)
POST("https://www.ishares.com/us/products/etf-product-list", body = list(productView="ishares",portfolios="239561-239855"))
Solution

  • After findling a bit around with Fiddler I found out that I need to send the data with postfields and then everything works fine.

    library("curl")
handle <- new_handle()
handle_setopt(handle, customrequest = "POST")
handle_setopt(handle, postfields='productView=ishares&portfolios=239561-239855')
curl_download("https://www.ishares.com/us/product-screener-download.dl", "./data/ishares-us-etf.xls", handle=handle)