Search code examples
rtokenizereadability

How can I tokenize a string in R?


I am trying to calculate readability, but it seems everything is written to expect either a file path or a Corpus. How do I handle a string?

Error (on the tokenization step):

Error: Unable to locate

I tried:

str<-"Readability zero one.  Ten, Eleven.", "The cat in a dilapidated tophat."
library(koRpus)
ll.tagged <- tokenize(str, lang="en")
readability(ll.tagged,measure="Flesch.Kincaid")

Solution

  • You need to download the language file

    install.koRpus.lang(c("en"))
    library(koRpus.lang.en)
    ll.tagged <- tokenize(str,  format = "obj", lang = "en")
    ll.tagged
       doc_id       token      tag lemma lttr   wclass desc stop stem idx sntc
    1    <NA> Readability word.kRp         11     word <NA> <NA> <NA>   1    1
    2    <NA>        zero word.kRp          4     word <NA> <NA> <NA>   2    1
    3    <NA>         one word.kRp          3     word <NA> <NA> <NA>   3    1
    4    <NA>           .     .kRp          1 fullstop <NA> <NA> <NA>   4    1
    5    <NA>         Ten word.kRp          3     word <NA> <NA> <NA>   5    2
    6    <NA>           ,     ,kRp          1    comma <NA> <NA> <NA>   6    2
                                                 [...]                        
    10   <NA>         cat word.kRp          3     word <NA> <NA> <NA>  10    3
    11   <NA>          in word.kRp          2     word <NA> <NA> <NA>  11    3
    12   <NA>           a word.kRp          1     word <NA> <NA> <NA>  12    3
    13   <NA> dilapidated word.kRp         11     word <NA> <NA> <NA>  13    3
    14   <NA>      tophat word.kRp          6     word <NA> <NA> <NA>  14    3
    15   <NA>           .     .kRp          1 fullstop <NA> <NA> <NA>  15    3