Search code examples
rspecial-characterstmterm-document-matrix

I am trying to create a DocumentTermMatrix while keeping all special characters


Trying to do some text mining with R without removing any special characters. For example in the following "LKC" and "LKC_" should be different words. Instead it is dropping the _ and making it the same word. How can I accomplish this?

library(tm)
special = c("OLAC_ LA LAC LAC_ LAC_E AC AC_ AC_E AC_ET",
                      ")LK )LKC )LKC- LK LKC LKC-",
                      "LAC_ LAC_E LKC LKC-")

bagOfWords <- Corpus(VectorSource(special))


mydocsDTM <- DocumentTermMatrix(bagOfWords, control = list(removePunctuation = FALSE,                     
preserve_intra_word_contractions = FALSE,         
preserve_intra_word_dashes = FALSE,
removeNumbers = FALSE,
stopwords = FALSE,
stemming = FALSE
))

  inspect(mydocsDTM)

Solution

  • Easily done using the quanteda package, after which you can convert to a DocumentTermMatrix, or just keep using quanteda.

    library("quanteda")
    qdfm <- dfm(special, tolower = FALSE, what = "fasterword")
    qdfm
    # Document-feature matrix of: 3 documents, 15 features (57.8% sparse).
    # 3 x 15 sparse Matrix of class "dfm"
    #        features
    # docs    OLAC_ LA LAC LAC_ LAC_E AC AC_ AC_E AC_ET )LK )LKC )LKC- LK LKC LKC-
    #   text1     1  1   1    1     1  1   1    1     1   0    0     0  0   0    0
    #   text2     0  0   0    0     0  0   0    0     0   1    1     1  1   1    1
    #   text3     0  0   0    1     1  0   0    0     0   0    0     0  0   1    1
    
    convert(qdfm, to = "tm")
    # <<DocumentTermMatrix (documents: 3, terms: 15)>>
    # Non-/sparse entries: 19/26
    # Sparsity           : 58%
    # Maximal term length: 5
    # Weighting          : term frequency (tf)