Search code examples
rtexttext-miningtmcorpus

Keep document ID with R corpus


I have searched stackoverflow and the web and can only find partial solutions OR some that don't work due to changes in TM or qdap. Problem below:

I have a dataframe: ID and Text (Simple document id/name and then some text)

I have two issues:

Part 1: How can I create a tdm or dtm and maintain the document name/id? It only shows "character(0)" on inspect(tdm).
Part 2: I want to keep only a specific list of terms, i.e. opposite of remove custom stopwords. I want this to happen in the corpus, not the tdm/dtm.

For Part 2, I used a solution I got here: How to implement proximity rules in tm dictionary for counting words?

This one happens on the tdm part! Is there a better solution for Part 2 where you use something like "tm_map(my.corpus, keepOnlyWords, customlist)"?

Any help will be greatly appreciated. Thanks much!


Solution

  • In newer versions of tm this is a lot easier with the DataframeSource() function.

    "A data frame source interprets each row of the data frame x as a document. The first column must be named "doc_id" and contain a unique string identifier for each document. The second column must be named "text" and contain a "UTF-8" encoded string representing the document's content. Optional additional columns are used as document level metadata."

    So in this case:

    dd <-data.frame(
        doc_id=10:13,
        text=c("No wonder, then, that ever gathering volume from the mere transit ",
          "So that in many cases such a panic did he finally strike, that few ",
          "But there were still other and more vital practical influences at work",
          "Not even at the present day has the original prestige of the Sperm Whale")
        ,stringsAsFactors=F
     )
    
    Corpus = VCorpus(DataframeSource(dd))