Search code examples
searchindexingporter-stemmer

Porter Stemmer algorithm question

I am implementing a search application. Corpus is large text documents. During file process i'm tokenizing all the words and calling Porter Stemmer algorithm Step1 (http://tartarus.org/~martin/PorterStemmer/csharp2.txt).

Step1 gets rid of plurals and -ed or -ing...

I noticed that a word like 'this' will be stemmed into 'thi'.

Is that normal operation of the algorithm ? Since I wanted to tokenize the word 'this'.

Solution

  • From what you describe, my hunch is that this is considered as plural form in Porter Stemmer algorithm and reduced to thi.

    I do not find an explicit reference to non-plural words ending with s in Porter's paper.

    http://tartarus.org/~martin/PorterStemmer/def.txt