I have a file on a private web server I am trying to access. I must first go to a site and login with my credentials and then I can type a URL (there is no link) to access the file, which immediately downloads a csv file to the computer. I am trying to get that csv file to automatically load into R either direct from online or have it automatically download and uploaded from my hard drive
I am going to be refreshing this data 10-15 times a day which is why I need it automatic rather than manually downloading it every time.
I have tried a with several packages and have been impressed with Hadley's package rvest which has shown much easier than some things I have used in the past. I am succeeding in downloading the data:
library(rvest)
fs <- html_session("somewebsite.org")
fs.login <- fs %>% follow_link("Sign In")
login.form <- html_form(fs.login)[[1]]
login.form <-set_values(login.form, userName = "myusername", password = "mypassword")
active.session <- submit_form(fs.login, login.form)
my.data <- jump_to(active.session, "somewebsite.org/report/groups")
I have ran it with a timer several times and it takes an average of 27 seconds which indicates it is downloading the file (roughly the same amount that it takes Google Chrome). The result is a variable class session with 7 elements 43.7 Mb
my.data
somewebsite/report/groups
Status: 200
Type: text/csv
Size: 45856046
My question is how can I access the actual csv file or data in r?
str(my.data)
List of 7
$ handle :List of 2
..$ handle:Class 'curl_handle' <externalptr>
..$ url : chr "somewebsite.org"
..- attr(*, "class")= chr "handle"
$ config :List of 7
..$ method : NULL
..$ url : NULL
..$ headers : NULL
..$ fields : NULL
..$ options :List of 1
.. ..$ autoreferer: int 1
..$ auth_token: NULL
..$ output : NULL
..- attr(*, "class")= chr "request"
$ url : chr "https://somewebsite.org/report/groups"
$ back : chr "https://somewebsite.org/report/groups"
$ forward : chr(0)
$ response:List of 10
..$ url : chr "https://somewebsite.org/report/groups"
..$ status_code: int 200
..$ headers :List of 6
.. ..$ content-disposition: chr "attachment; filename=\"groups-2016-0318-063749.csv\""
.. ..$ content-type : chr "text/csv"
.. ..$ date : chr "Fri, 18 Mar 2016 18:37:49 GMT"
.. ..$ server : chr "Apache-Coyote/1.1"
.. ..$ transfer-encoding : chr "chunked"
.. ..$ connection : chr "Close"
.. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
..$ all_headers:List of 1
.. ..$ :List of 3
.. .. ..$ status : int 200
.. .. ..$ version: chr "HTTP/1.1"
.. .. ..$ headers:List of 6
.. .. .. ..$ content-disposition: chr "attachment; filename=\"groups-2016-0318-063749.csv\""
.. .. .. ..$ content-type : chr "text/csv"
.. .. .. ..$ date : chr "Fri, 18 Mar 2016 18:37:49 GMT"
.. .. .. ..$ server : chr "Apache-Coyote/1.1"
.. .. .. ..$ transfer-encoding : chr "chunked"
.. .. .. ..$ connection : chr "Close"
.. .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
..$ cookies :'data.frame': 6 obs. of 7 variables:
.. ..$ domain : chr [1:6] "somewebsite.org" "#HttpOnly_.site.org" "signin.site.org" ".site.org" ...
.. ..$ flag : logi [1:6] FALSE TRUE FALSE TRUE FALSE TRUE
.. ..$ path : chr [1:6] "/" "/" "/" "/" ...
.. ..$ secure : logi [1:6] FALSE TRUE FALSE FALSE TRUE TRUE
.. ..$ expiration: POSIXct[1:6], format: "2017-03-18 12:37:16" NA NA NA ...
.. ..$ name : chr [1:6] "fs_experiments" "ObSSOCookie" "TS01289383" "TS01b89640" ...
.. ..$ value : chr [1:6] "u%3D-anon-%2Ca%3Dshared-ui%2Cs%3Dac76fc702b255a493a5856b5432b92b4%2Cv%3D0100110011010000000111111111001110101101000000000001100"| __truncated__ "15yUK2dU%2B78GK7o587gtwh3i%2ByORXGD8ne5XJBiGkiHpDAJ3%2F7rQ4Gql6T5DqQIwCg%2FSwSioAMIzzaRxGEFKsCkc%2BGohi1fdWhbR0urah6%2BJikm9lA6"| __truncated__ "01999b7023d69473f53740d0f7f2969d9d79e1a18c7e259f6baf643ce642a330fc0a3604d7" "01999b7023960237ab42ec3f429e5a452fe3559d683a090b19a65cf66ce0c01bc21bdb29bf78f030d36d4eeff4dec21ff185c54b06" ...
..$ content : raw [1:45857717] 69 64 2c 6e ...
..$ date : POSIXct[1:1], format: "2016-03-18 18:37:49"
..$ times : Named num [1:6] 0 0 0.062 0.156 27.425 ...
.. ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ...
..$ request :List of 7
.. ..$ method : chr "GET"
.. ..$ url : chr "https://somewebsite.org/report/groups"
.. ..$ headers : Named chr "application/json, text/xml, application/xml, */*"
.. .. ..- attr(*, "names")= chr "Accept"
.. ..$ fields : NULL
.. ..$ options :List of 4
.. .. ..$ useragent : chr "libcurl/7.43.0 r-curl/0.9.6 httr/1.0.0"
.. .. ..$ cainfo : chr "C:/Users/Thisuser/Documents/R/win-library/3.2/httr/cacert.pem"
.. .. ..$ autoreferer : int 1
.. .. ..$ customrequest: chr "GET"
.. ..$ auth_token: NULL
.. ..$ output : list()
.. .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
.. ..- attr(*, "class")= chr "request"
..$ handle :Class 'curl_handle' <externalptr>
..- attr(*, "class")= chr "response"
$ html :<environment: 0x000000001aad2f60>
- attr(*, "class")= chr "session"
The data are stored in the list item named "content". read_csv from the "readr" package should be able to read it directly.
Try the following:
library(httr)
library(readr)
read_csv(my.data$content)