Search code examples
javascriptrweb-scrapingrselenium

Web scraping in R by first navigating through a JavaScript module

I looked up various questions and answers but unfortunately none of the problems I found dealt with a case that is similar to mine. In a typical question, the JavaScript table builds up directly when the website is loaded. In my case, however, I first have to navigate through the JavaScript module and select several criteria before I get the sought-after result.

This is my case: I have to scrape the exchange rates for various currencies from this website www.globocambio.co. To do that, I have (1) to navigate to “I WANT COLOMBIAN PESO”, (2) select the currency (e.g., “Chilean Peso”), (3) and the collection destination (e.g., “El Dorado International Airport”). Only then the respective exchange rate is being loaded. See this screenshot for illustration. I marked the three selection steps red. Green is the data point that I want to scrape for different currencies.

I am not very familiar with JavaScript but I tried to understand what is going on. Here is what I found out:

  1. Using Chrome DevTools, I investigated the Network activity when loading an exchange rate. There is an XHR called “GetPrice” that requests the price using this URL: https://reservations.globocambio.co/DesktopModules/GlobalExchange/API/Widget/GetPrice and using the following Form Data ISOAOrigen=CLP&cantidadOrigen=9000&ISOADestino=COP&cantidadDestino=0&centerId=27&operationType=OperationTypesBuying
  2. I understand that the Form Data contains the information that I initially selected manually:
    1. operationType=OperationTypesBuying: this is the “I WANT COLOMBIAN PESO” option
    2. ISOAOrigen=CLP: this is the “Chilean Peso”
    3. centerId=27: this is the “El Dorado International Airport”

  3. The server responds to my request with the following information:

    {“MonedaOrigen":{"ISOA":"CLP","Nombre":null,"Margen":0.1630000000,"Tramo":0.0,"Fixing":2.9000000000},"CantidadOrigen":9000.00,"MonedaDestino":{"ISOA":"COP","Nombre":null,"Margen":0.0,"Tramo":0.0,"Fixing":0.0},"CantidadDestino":21845.70,"TipoCambio":2.42730000000000000000,"MargenOrigen":0.0,"TramoOrigen":0.0,"FixingOrigen":0.0,"MargenDestino":0.0,"TramoDestino":0.0,"FixingDestino":0.0,"IdCentro":"27","Comision":null,"ComisionTramoSuperior":null,"ComisionAplicada":{"CodigoMoneda":null,"CodigoTipoMoneda":0,"ComisionFija":0.0,"ComisionVariable":0.0,"TramoInicio":0.0,"TramoFin":null,"Orden”:0}}

  4. From this response, "TipoCambio":2.42730000000000000000 is then being written on the website using this line of HTML code: <span id="spTipoCambioCompra">2.427300</span>

  5. This means that "TipoCambio" is the value that I am looking for.

So, I have to communicate somehow via R with the server using the Form Data as input variables. Can anyone tell me how to do this? I mean, understand that I have to combine the URL https://reservations.globocambio.co/DesktopModules/GlobalExchange/API/Widget/GetPrice with the Form Data “ISOAOrigen=CLP&cantidadOrigen=9000&ISOADestino=COP&cantidadDestino=0&centerId=27&operationType=OperationTypesBuying” somehow but I do not know how it works..

Any help will be appreciated!

Update:

I still have no idea how to solve the above issue, yet. However, I try to approach it with small steps. Using RSelenium, I am currently trying to find out how to click on the option “I WANT COLOMBIAN PESO”. My idea was to use the following code:

library(RSelenium)
remDr <- RSelenium::remoteDriver(remoteServerAddr = "localhost",
                                 port = 4445L,
                                 browserName = "chrome")
remDr$open()
remDr$navigate("https://www.globocambio.co/en/home")
webElem <- remDr$findElement("id", "tabCompra") #What is wrong here?
webElem$clickElement() # Click on "I WANT COLOMBIAN PESO"

But I get an error message after executing webElem <- remDr$findElement("id", "tabCompra"):

Selenium message:no such element: Unable to locate element: {"method":"css selector","selector":"#tabCompra"} (Session info: chrome=81.0.4044.113) For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/no_such_element.html ... Error: Summary: NoSuchElement Detail: An element could not be located on the page using the given search parameters. class: org.openqa.selenium.NoSuchElementException Further Details: run errorDetails method

What am I doing wrong here?

Solution

  • I solved my problem using selenium in Python: 

    from selenium import webdriver
driver = webdriver.Firefox(executable_path = '/your_path/geckodriver')

driver.get("https://www.globocambio.co/en/")
driver.switch_to.frame("iframeWidget");

elem = driver.find_element_by_id('tabCompra')
elem.click()

elem = driver.find_element_by_id('inputddlMonedaOrigenCompra')
elem.click()
elem.send_keys(Keys.CLEAR)
elem.send_keys("Chilean Peso")
elem.send_keys(Keys.ENTER)
elem.send_keys(Keys.ARROW_DOWN)
elem.send_keys(Keys.RETURN)

elem = driver.find_element_by_id('info-change-compra')
print(elem.text)