Search code examples
rdataframercurl

R using getURL data to dataframe


I'm downloading data from the web but then don't know how to change it to a dataframe or anything useful. Does anyone have any suggestions? Here is the code:

library(RCurl) 
myfile = getURL(http://www.stat.ufl.edu/~winner/data/lister_ul.dat,
ssl.verifyhost=FALSE, ssl.verifypeer=FALSE)

If I use this:

A = read.csv(textConnection(myfile), header = F)

then R understands this:

c("1 1 1")

as the first row and not this:

c(1, 1, 1).

This doesn't work b/c I need to use

colnames(A) = c("col1", "col2", "col3")

and can't find a workaround that doesn't involve some tedious work using

unlist(strsplit(A))

Ughh!!

Any suggestions would be appreciated. Or maybe I'll write my own tedious function, if necessary.

gwynn


Solution

  • You are close. Since I don't have RCurl installed but I do have httr (which uses curl), I'll start with that. It's a moot problem, though, since I get to the same table-looking content as you.

    Also, @udden2903's answer is more straight-forward, I'm making an assumption that this is a simplified problem, and that you may have need to continue using an alternative fetching method that read.table(URL) does not allow. (To continue using httr and support some other things such as authentication, read its documentation.)

    library(httr)
    myfile = GET("http://www.stat.ufl.edu/~winner/data/lister_ul.dat")
    str(content(myfile))
    # No encoding supplied: defaulting to UTF-8.
    #  chr "1 1  1\n1 0 11\n0 1  6\n0 0  6\n"
    

    So, content(myfile) is now what your myfile is. The first trick is that your data is not comma-delimited ("csv"), so using read.table is necessary. Second, you nede to specifiy that the first line is not headers.

    x <- read.table(textConnection(content(myfile, encoding = "UTF-8")), header = FALSE)
    x
    #   V1 V2 V3
    # 1  1  1  1
    # 2  1  0 11
    # 3  0  1  6
    # 4  0  0  6
    

    Now just assign your headers.

    colnames(x) <- c("col1", "col2", "col3")
    x
    #   col1 col2 col3
    # 1    1    1    1
    # 2    1    0   11
    # 3    0    1    6
    # 4    0    0    6