R XML href scrape from SEC Edgar web site

Question

I checked prior similar questions - no luck... can's seem to get readHTMLTable to read an Edgar web page. I am trying to read this URL:

https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=AAPL&type=10-Q&dateb=&owner=exclude&count=100

...and get all the href links under the "Documents" buttons into a character vector.

The "Documents" links are in a table - from Firefox Inspection tool the first "Documents" href link looks like this:

<div id="seriesDiv" style="margin-top: 0px;">

    <table class="tableFile2" summary="Results">
        <tbody>
            <tr></tr>
            <tr>
                <td nowrap="nowrap"></td>
                <td nowrap="nowrap">
                    <a id="documentsbutton" href="/Archives/edgar/data/320193/000119312516559625/0001193125-16-559625-index.htm">

                         Documents

So I want to get the href link into a character vector for use later.

Problem - XML library is giving me trouble and htmltab library functions do not seem to get recognized in my R instance for some reason.

Here is my code:

library(XML)
EDGARURL <- "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=AAPL&type=10-Q&dateb=&owner=exclude&count=100"
EDGARHREFtables <- readHTMLTable(EDGARURL, as.data.frame = TRUE)

Which results in the following error:

    Warning message:
XML content does not seem to be XML: 'https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=AAPL&type=10-Q&dateb=&owner=exclude&count=100'

What am I missing? Will the XML library's readHTMLTable work on this? And if so how do you extract the href tag for each document?

Answer 1

For simple jobs, the rvest package is a lot easier:

library(rvest)

url <- 'https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=AAPL&type=10-Q&dateb=&owner=exclude&count=100'

        # pull HTML from page  
url %>% read_html() %>%
    # get tags with a certain CSS selector
    html_nodes('#documentsbutton') %>%
    # get the href attribute from each node
    html_attr('href')

# [1] "/Archives/edgar/data/320193/000119312516559625/0001193125-16-559625-index.htm"
# [2] "/Archives/edgar/data/320193/000119312516439878/0001193125-16-439878-index.htm"
# [3] "/Archives/edgar/data/320193/000119312515259935/0001193125-15-259935-index.htm"
# [4] "/Archives/edgar/data/320193/000119312515153166/0001193125-15-153166-index.htm"
# [5] "/Archives/edgar/data/320193/000119312515023697/0001193125-15-023697-index.htm"
# [6] "/Archives/edgar/data/320193/000119312514277160/0001193125-14-277160-index.htm"
# [7] "/Archives/edgar/data/320193/000119312514157311/0001193125-14-157311-index.htm"
# [8] "/Archives/edgar/data/320193/000119312514024487/0001193125-14-024487-index.htm"
# [9] "/Archives/edgar/data/320193/000119312513300670/0001193125-13-300670-index.htm"
# [10] "/Archives/edgar/data/320193/000119312513168288/0001193125-13-168288-index.htm"
# ...