Search code examples
rweb-scrapingrvestrcurlhttr

Function to loop through values


I'm trying to perform multiple searches in a website and having a hard time designing a function to plugin in different values and convert the result into a dataframe

below is a set of commands I came up with, but it only helps for individual searches

fn  = 'HARVEY'

ln  = 'ADELSON'

mydf = data.frame(fn,ln);




  root = 'https://npiregistry.cms.hhs.gov/'


  u = paste(root,'registry/search-results-table?','first_name=', mydf$fn, '&last_name=', 
            mydf$ln, sep = "");

            # encode url correctly

            zero <- httr::GET(u);
  tables <- rvest::html_table(content(zero));


tab<-as.data.frame(tables)

Is there a function in r that would plugin different values for the first and the last name and run the set of commands and lastly store all the results in a single dataframe.

Thanks


Solution

  • Generically, you can use Map or mapply to apply your functions to corresponding elements of the columns of a data.frame. The result is returned as a list of data.frames. You can then row bind them (using either plyr::rbind.fill or data.table::rbindlist) to create a single data.frame

    library(httr)
    library(rvest)
    
    fn <- c('HARVEY', "Dollar")
    ln <- c('ADELSON', "Vora")
    mydf <- data.frame(FirstName=fn, LastName=ln);
    root <- 'https://npiregistry.cms.hhs.gov/'
    
    #you can use data.table::rbindlist or plyr::rbind.fill to row bind a list of data.frames
    as.data.frame(data.table::rbindlist(Map(function(fn, ln) {
        u <- paste0(root,'registry/search-results-table?',
            'first_name=', fn, '&last_name=',  ln)
        zero <- GET(u)
        tables <- html_table(content(zero))
    
        as.data.frame(tables)
    }, mydf$FirstName, mydf$LastName), fill=TRUE))