Why pdf_text from pdftools reads only the first page of each pdf element in my list of pdfs?

Question

I would like to create a dataframe with all the text and title of ech pdf of my pdfs list. I made one for loop but when I open the resulting dataframe I see that not all the text from each pdf have been processed into text, but only the last page.

Here the code:

#folder
folder <- paste0(getwd(), "/data/pdfs/")

#QC
qc <- sample(1:length(pdf_list), size = 1, replace = F)

#Create a vector of all the files in the folder selected
file_vector <- list.files(all.files = T, path = folder, recursive = T)

#Apply a filter to the file vector of files
pdf_list <- file_vector[grepl(".pdf", file_vector)]

#make the data frame
corpus_raw <- data.frame("title" = c(),"text" = c())

#list of pdf files comprehensive of path
files <- paste0(folder,pdf_list)

#cycle for the text fetching: 
for (i in 1:length(pdf_list)){
    #print i so that I know the loop is working right
    print(i)
    #take the text
    text_from_pdf <- pdf_text(pdf = files[i])
    temp_store_data <- data.frame("title" = gsub(pattern = "\\d\\/", replacement = "", 
                                               x = pdf_list[i], ignore.case = T), 
                                  "text" = text_from_pdf, stringsAsFactors = F)
    # quality control
    if (i == qc[1]){
      print(temp_store_data[i,2])
      write(temp_store_data[i,2], "data/quality_control.txt")
    }
    colnames(temp_store_data) <- c("title", "text")
    corpus_raw <- rbind(corpus_raw, temp_store_data)
}

Can you help me with this? Thank you!

Answer 1

pdf_text creates a vector with one string per page, not a single text string. You are only writing the ith page of your list to the qc text file.

You could try this when reading in the pdf:

text_from_pdf <- paste(pdf_text(pdf = files[i]), collapse = "\n")

This should work providing you do not have whole novels as pdfs to store.