Get the whole List as Json from a List in R

Question

Hello So I am using get request to github api:

commitsPublic <- GET("https://<host-name>/api/v3/search/commits?
q=is:public+org:<ORG-NAME>",
add_headers(Authorization= "token <your-Token>", Accept= 
'application/vnd.github.cloak-preview'))

commitsPublic

And I get:

156 Items (Total Count)

 Content-Type: application/json; charset=utf-8
  Size: 291 kB
{
  "total_count": 156,
  "incomplete_results": false,
  "items": [
    {
  (I removed the Items Since they are not important)

But when I try to convert the Json:

Shows raw data which is not structured and readable

jsonRespText<-content(commitsPublic,as="text") 
jsonRespText

Convert to Json:

toJson <- fromJSON(jsonRespText)
toJson

Returns:

$items[[30]]$score
[1] 1

It returns a List with items up to the number 30 " items[[30]]

and items[[31]] is NULL

So what I am asking is how can I get the 156 Listed items all of the list from Json text. I have another get request which gives me 10.000 Total Commits. But when I convert from Json the list has 30 still. So any help would be grateful thanx!

Answer 1

The Github API paginates to 30 results. Pagination information is in the response Link header.

library(httr)

commitsPublic <- GET("https://api.github.com/search/commits?q=is:public+org:rstudio",
                     add_headers(Accept = 'application/vnd.github.cloak-preview'))

headers(commitsPublic)$link
#> [1] "<https://api.github.com/search/commits?q=is%3Apublic+org%3Arstudio&page=2>; 
#> rel=\"next\", 
#> <https://api.github.com/search/commits?q=is%3Apublic+org%3Arstudio&page=34>; 
#> rel=\"last\""

^{Created on 2019-03-22 by the reprex package (v0.2.1)}

This tells us where the next page is located, and that there are 34 pages in total in this instance.

Reference: https://developer.github.com/v3/guides/traversing-with-pagination/