Search code examples
rrvestrselenium

Extracting links from a drop down menu using rvest/RSelenium

I am trying to scrape a list of place names from a dropdown box. (The dropdown box corresponds to "Zona" in the following link ( https://www.fotocasa.es/es/comprar/viviendas/a-coruna-provincia/todas-las-zonas/l )

I can read in the html data using:

url = 'https://www.fotocasa.es/es/comprar/viviendas/a-coruna-provincia/todas-las-zonas/l'
html_full_page = url %>% 
  read_html()

However, I cannot seem to find the correct xpath or the correct nodes.

html_full_page %>% 
  html_nodes(xpath = '//*[@id="App"]/div[2]/div/div[2]/div[3]/div/div[1]')

I thought the nodes would correspond to the following:

re-GeographicSearchNext-linksList-items

But I cannot seem to collect this data.

What I would like to collect is a small data frame for the dropdown menu.

Place Name    Properties    URL
A bARCALA        69         /es/comprar/viviendas/a-coruna-provincia/a-barcala/l
Arzùa           105         /es/comprar/viviendas/a-coruna-provincia/arzua/l
Barbanza        636         /es/comprar/viviendas/a-coruna-provincia/barbanza/l

Copy element from webpage:

> <div class="re-GeographicSearchNext-linksList-items"><a
> class="re-GeographicSearchNext-linkItem" title="A Barcala"
> href="/es/comprar/viviendas/a-coruna-provincia/a-barcala/l"><span
> class="re-GeographicSearchNext-linkItem-literal">A Barcala</span><span
> class="re-GeographicSearchNext-linkItem-count">69</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Arzúa"
> href="/es/comprar/viviendas/a-coruna-provincia/arzua/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Arzúa</span><span
> class="re-GeographicSearchNext-linkItem-count">105</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Barbanza"
> href="/es/comprar/viviendas/a-coruna-provincia/barbanza/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Barbanza</span><span
> class="re-GeographicSearchNext-linkItem-count">636</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Bergantiños"
> href="/es/comprar/viviendas/a-coruna-provincia/bergantinos/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Bergantiños</span><span
> class="re-GeographicSearchNext-linkItem-count">581</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Comarca de A Coruña"
> href="/es/comprar/viviendas/a-coruna-provincia/comarca-de-a-coruna/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Comarca de A
> Coruña</span><span
> class="re-GeographicSearchNext-linkItem-count">3.701</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Comarca de Betanzos"
> href="/es/comprar/viviendas/a-coruna-provincia/comarca-de-betanzos/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Comarca de
> Betanzos</span><span
> class="re-GeographicSearchNext-linkItem-count">715</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Comarca de Ferrol"
> href="/es/comprar/viviendas/a-coruna-provincia/comarca-de-ferrol/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Comarca de
> Ferrol</span><span
> class="re-GeographicSearchNext-linkItem-count">3.698</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Comarca de Santiago"
> href="/es/comprar/viviendas/a-coruna-provincia/comarca-de-santiago/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Comarca de
> Santiago</span><span
> class="re-GeographicSearchNext-linkItem-count">1.910</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Eume"
> href="/es/comprar/viviendas/a-coruna-provincia/eume/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Eume</span><span
> class="re-GeographicSearchNext-linkItem-count">282</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Fisterra"
> href="/es/comprar/viviendas/a-coruna-provincia/fisterra/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Fisterra</span><span
> class="re-GeographicSearchNext-linkItem-count">198</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Muros"
> href="/es/comprar/viviendas/a-coruna-provincia/muros/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Muros</span><span
> class="re-GeographicSearchNext-linkItem-count">78</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Noia"
> href="/es/comprar/viviendas/a-coruna-provincia/noia/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Noia</span><span
> class="re-GeographicSearchNext-linkItem-count">192</span></a><a
> class="re-GeographicSearchNext-linkItem" title="O Sar"
> href="/es/comprar/viviendas/a-coruna-provincia/o-sar/l"><span
> class="re-GeographicSearchNext-linkItem-literal">O Sar</span><span
> class="re-GeographicSearchNext-linkItem-count">79</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Ordes"
> href="/es/comprar/viviendas/a-coruna-provincia/ordes/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Ordes</span><span
> class="re-GeographicSearchNext-linkItem-count">128</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Ortegal"
> href="/es/comprar/viviendas/a-coruna-provincia/ortegal/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Ortegal</span><span
> class="re-GeographicSearchNext-linkItem-count">248</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Terra de Melide"
> href="/es/comprar/viviendas/a-coruna-provincia/terra-de-melide/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Terra de
> Melide</span><span
> class="re-GeographicSearchNext-linkItem-count">126</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Terra de Soneira"
> href="/es/comprar/viviendas/a-coruna-provincia/terra-de-soneira/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Terra de
> Soneira</span><span
> class="re-GeographicSearchNext-linkItem-count">68</span></a><a
> class="re-GeographicSearchNext-linkItem" title="Xallas"
> href="/es/comprar/viviendas/a-coruna-provincia/xallas/l"><span
> class="re-GeographicSearchNext-linkItem-literal">Xallas</span><span
> class="re-GeographicSearchNext-linkItem-count">20</span></a></div>
Solution

  • We can get the data from dropdown menu data first clicking on element and then extracting required info,

    library(RSelenium)
library(rvest)
rD <- rsDriver(browser="firefox", port=4536L)
remDr <- rD[["client"]]
#navigate 
url = 'https://www.fotocasa.es/es/comprar/viviendas/a-coruna-provincia/todas-las-zonas/l'
remDr$navigate(url)
#click on Zona
remDr$findElement(using = "xpath",'//*[@id="save-results-panel-trigger"]')$clickElement()


html_full_page = remDr$getPageSource()[[1]] %>% read_html() 

names = html_full_page %>% 
  html_nodes('.re-GeographicSearchNext-linkItem') %>% 
  html_text()
  
number = html_full_page %>% 
  html_nodes('.re-GeographicSearchNext-linkItem') %>% html_nodes('.re-GeographicSearchNext-linkItem-count') %>% 
  html_text()

link  = html_full_page %>% 
  html_nodes('.re-GeographicSearchNext-linkItem') %>% 
  html_attr('href')

df = cbind.data.frame(names, number, link)

                      names number                                                           link
1               A Barcala69     69           /es/comprar/viviendas/a-coruna-provincia/a-barcala/l
2                  Arzúa105    105               /es/comprar/viviendas/a-coruna-provincia/arzua/l
3               Barbanza636    636            /es/comprar/viviendas/a-coruna-provincia/barbanza/l
4            Bergantiños581    581         /es/comprar/viviendas/a-coruna-provincia/bergantinos/l