Search code examples
rxmlweb-scrapingrvestnode-set

R - rvest (webscraping) with unclosed xml nodes, here: problem with html_nodes("br")

I used rvest to extract one part of a webpage (EDIT: this webpage) with this code:

library('rvest')
webpage <- read_html(url("https://www.tandfonline.com/action/journalInformation?show=editorialBoard&journalCode=ceas20"))
people <- webpage %>%
  html_nodes(xpath='//*[@id="8af55cbd-03a5-4deb-9086-061d8da288d1"]/div/div/div') %>%
  html_nodes(xpath='//p')

The result was stored in a xml_nodeset called people:

> people
{xml_nodeset (11)}
 [1] <p> <b>Editors:</b> <br> Dr Xyz Anceschi - <i>University of Glasgow <a href="http://www.gla.ac.uk/schools/soci ...
 [2] <p> <b>Editorial Board:</b> <br> Dr Xyz Aliyev - <i>University of Glasgow</i> <br> Professor Richard Berry < ...
 [3] <p> <b>Board of Management:</b> <br> Professor Xyz Berry (Chair) <i>- University of Glasgow</i> <br> Profes ...
 [4] <p> <b>National Advisory Board:</b> <br> Dr Xyz Badcock <i>- University of Nottingham</i> <br> Professor Cath ...

In people, each element contains various names of people following <br> (which, however, is unclosed: there is no </br>).

I seek to parse each person using this code, but it does not work:

sapply(people,
    function(x)
    {
        x %>%
        html_nodes("br") %>%
        html_text()
    }
)

It only gives me a list of empty results:

[[1]]
 [1] "" ""

[[2]]
 [1] "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

[[3]]
 [1] "" "" "" "" ""

[[4]]
 [1] "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

I assume that the error is grounded in the fact that <br> is an unclosed node in the xml_nodeset. Could it be the case?

If that is so, is there anything I can nevertheless do to extract each person from people?

Solution

  • You can use str_match_all to get all the names which occur between <br> and <i>.

    unlist(sapply(stringr::str_match_all(people, '<br> (.*?)\\s?-?\\s<i>'), 
              function(x) x[, 2]))

# [1] "Dr Luca Anceschi"                "Professor David J. Smith"       
# [3] "Dr Huseyn Aliyev"                "Professor Richard Berry"        
# [5] "Dr Maud Bracke"                  "Dr Eamonn Butler"               
# [7] "Dr Ammon Cheskin"                "Dr Sai Ding"                    
# [9] "Professor Jane Duckett"          "Professor Rick Fawn"  
#...
#...