Search code examples
rjsonrvest

Webscrape from API with Rvest or JsonLite

I am trying to scrape data from this website. I have successfully done it before, but they have changed the site. Is it possible to extract from their API, if they have it at all?

Alternatively, I could do it with RSelenium but it takes a lot of time.

Inpect of the network

I found this is the dev tool, link as such https://www.domstol.no/api/episerver/v3/beramming/ and it does return the complete dataset that I want. This is shown below:

enter image description here

How am I able to request this into R?

Solution

  • Just need to mimic that POST request sending blank param values in the JSON body. None of the headers or cookies are required.

    library(httr2)
library(magrittr)

r <- request("https://www.domstol.no/api/episerver/v3/beramming/") %>% 
  req_body_json(list('To' = '', 'From' = '', 'CaseAbout' = '', 'CaseNumber' = '')) %>% 
  req_perform() %>%
  resp_body_json()


library(purrr)

df <- map_dfr(r, ~ data.frame(.x))

    Or as per OP in comments re df from result of call

    library(httr2)
library(magrittr)
library(dplyr)

df <- request("https://www.domstol.no/api/episerver/v3/beramming/") %>% 
  req_body_json(list('To' = '', 'From' = '', 'CaseAbout' = '', 'CaseNumber' = '')) %>% 
  req_perform() %>%
  resp_body_json(simplifyVector = TRUE) %>% 
  tibble()