Search code examples
rtidyversetext-miningtidytext

R: Text Mining, create list of words per document

I am reading in the text from a number of PDFs in a directory. Then, I split these texts into single words (tokens) using the tidytext::unnest_tokens()-function. Can someone please tell me, how I can add an additional column to the test-tibble with the name of the file each word comes from?

library(pdftools)
library(tidyverse)
library(tidytext)

files <- list.files(pattern = "pdf$")
content <- lapply(files, pdf_text)
list <- unlist(content, recursive = TRUE, use.names = TRUE)
df = data.frame(text = list)

test <- df %>% tidytext::unnest_tokens(word, text)
Solution

  • You can try the following. Instead of using unlist with all the files, instead pass the entire list of files to map_df from purrr. Then, you can add a column with filename along with the word column.

    library(pdftools)
library(tidyverse)
library(tidytext)

files <- list.files(pattern = "pdf$")

map_df(files, ~ data.frame(txt = pdf_text(.x)) %>%
         mutate(filename = .x) %>%
         unnest_tokens(word, txt))