Search code examples
rweb-scrapingrvest

R: Toggle <select> options, submit action and web scrape HTML from site with no <form> tag


How can I manage to get the data from a website that presents multiple options like the ticker of the stock, and the beginning and the end of the period I want the data.

The code that generates this data comes from this line:

<td><input name="button" type="button" class="boton" id="button" value="Buscar" onclick="getInf_Cotizaciones('SIDERC1',document.getElementById('anoIni').value+document.getElementById('mesIni').value+'01',document.getElementById('anoFin').value+document.getElementById('mesFin').value+'01')" /></td>

However the data doesn't show on the HTML source code. How can I get R to download this data.


Solution

  • If you use "Developer Mode" on any modern browser and sort the "timeline" view of the "network resources" (they all have this) by "start time", you'd see that site submits the following URL:

    http://www.bvl.com.pe/jsp/cotizacion.jsp?fec_inicio=20140901&fec_fin=20141001&nemonico=SIDERC1
    

    when posting data based on the <select> box choices. You can, then, use the rvest package to grab the resultant table:

    library(rvest)
    
    pg <- html("http://www.bvl.com.pe/jsp/cotizacion.jsp?fec_inicio=20140901&fec_fin=20141001&nemonico=SIDERC1")
    pg %>% html_table()
    
    ## [[1]]
    ##    Precio fecha actual       NA     NA     NA     NA       NA                NA                   NA Precios fecha anterior             NA
    ## 1     Fecha cotización Apertura Cierre Máxima Mínima Promedio CantidadNegociada MontoNegociado (S/.)          Fechaanterior Cierreanterior
    ## 2           01/10/2014     0.33   0.32   0.33   0.32     0.32        193,148.00            62,707.36             30/09/2014           0.32
    ## 3           30/09/2014     0.33   0.32   0.33   0.32     0.33        542,761.00           177,545.23             29/09/2014           0.34
    ## 4           29/09/2014     0.34   0.34   0.34   0.34     0.34         42,738.00            14,530.92             26/09/2014           0.34
    ## 5           26/09/2014     0.34   0.34   0.34   0.34     0.34        139,829.00            47,503.57             25/09/2014           0.35
    ## 6           25/09/2014     0.35   0.35   0.35   0.35     0.35         56,100.00            19,635.00             23/09/2014           0.35
    ## 7           24/09/2014                                                                                           23/09/2014           0.35
    ## 8           23/09/2014     0.35   0.35   0.35   0.35     0.35         79,800.00            27,900.00             19/09/2014           0.35
    ## 9           22/09/2014                                                                                           19/09/2014           0.35
    ## 10          19/09/2014     0.35   0.35   0.35   0.35     0.35         73,655.00            25,592.70             18/09/2014           0.35
    ## 11          18/09/2014     0.35   0.35   0.35   0.35     0.35         50,000.00            17,500.00             17/09/2014           0.35
    ## 12          17/09/2014     0.35   0.35   0.35   0.35     0.35         94,000.00            32,900.00             16/09/2014           0.36
    ## 13          16/09/2014     0.36   0.36   0.36   0.36     0.36         49,582.00            17,666.87             15/09/2014           0.35
    ## 14          15/09/2014     0.35   0.35   0.35   0.35     0.35         63,900.00            22,365.00             12/09/2014           0.35
    ## 15          12/09/2014     0.35   0.35   0.35   0.35     0.35        100,000.00            35,000.00             11/09/2014           0.36
    ## 16          11/09/2014     0.36   0.36   0.36   0.36     0.36         79,680.00            28,684.80             10/09/2014           0.36
    ## 17          10/09/2014     0.36   0.36   0.36   0.36     0.36        136,169.00            49,020.84             09/09/2014           0.36
    ## 18          09/09/2014     0.35   0.36   0.36   0.35     0.36        420,200.00           151,074.07             08/09/2014           0.35
    ## 19          08/09/2014     0.35   0.35   0.35   0.35     0.35         90,344.00            31,620.40             05/09/2014           0.34
    ## 20          05/09/2014     0.34   0.34   0.34   0.34     0.34        212,500.00            72,250.00             04/09/2014           0.33
    ## 21          04/09/2014     0.33   0.33   0.33   0.33     0.33         12,500.00             4,125.00             03/09/2014           0.34
    ## 22          03/09/2014     0.33   0.34   0.34   0.33     0.33        186,000.00            61,970.00             02/09/2014           0.33
    ## 23          02/09/2014     0.34   0.33   0.34   0.33     0.34        221,613.00            74,654.42             28/08/2014           0.35
    ## 24          01/09/2014                                                                                           28/08/2014           0.35