Search code examples
rweb-scrapingrvestrcurlhttr

R - Sending search request on a website


I have a little problem involving posting a search request on a website and displaying the result by using the R packages rvest and httr...I just want to run one search for the name "Acer campestre", and ticking only the box "Match whole words only". Here is my code:

library(httr)
library(rvest)
col = POST(url="http://www.catalogueoflife.org/col",
           encode="form",
           body=list(text="Acer campestre",
                     fossil="0",
                     match="1",
                     submit="Search"))
col_html = read_html(col)
col_table = html_table(col_html,fill=T)

I think I am not too far from the answer, but it seems I always have trouble using this kind a command using html code...Hopes that someone could help me, thanks in advance !


Solution

  • OK,

    I solved it myself in the end, the problem had three sources:

    -Not "fill=T" but "fill=F"

    -Inputs badly called for the post request: not "text" but "key", and not "submit" but "search"...

    -Last but not least: the default URL "http://www.catalogueoflife.org/col" was not the one to use. "http://www.catalogueoflife.org/col/search/all" is the right one to use to actually post request or interact with the web page...Here is the code:

    library(rvest)
    library(httr)
    col = POST(url="http://www.catalogueoflife.org/col/search/all",
               encode="form",
               body=list(key="Acer campestre",
                         fossil="0",
                         match="1",
                         search="Search"))
    col_html = read_html(col)
    col_table = html_table(col_html,fill=F)
    

    Then, give the content of the web page in a nice table !

    Hope it will be of help to someone :)