Lucene is limiting the query terms

Question

I'm trying to use Lucene (5.4.1) MoreLikeThis to tag(classify) texts. It's kind of working, but I'm getting poor results, and I think that the problem is related with the Query object.

The example bellow works, but the highest topdoc isn't the one that I expect. By debuging the query object, it shows only content:erro. From a complete portuguese phrase (see into the example) the query was constructed with just one word.

I'm not using stop words or any other kind of filter.

So why lucene is picking just erro as a query term?

To init main objects

Analyzer analyzer = new PortugueseAnalyzer();

Directory indexDir = new RAMDirectory();

IndexWriterConfig config = new IndexWriterConfig(analyzer);
config.setOpenMode(OpenMode.CREATE_OR_APPEND);

To index

try (IndexWriter indexWriter = new IndexWriter(indexDir, config)) {
   FieldType type = new FieldType();
   type.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
   type.setStored(true);
   type.setStoreTermVectors(true);

   Document doc = new Document();
   doc.add(new StringField("id", "880b2bbc", Store.YES));
   doc.add(new Field("content", "erro", type));
   doc.add(new Field("tag", "atag", type));

   indexWriter.addDocument(doc);
   indexWriter.commit();
}

To search

    try (IndexReader idxReader = DirectoryReader.open(indexDir)) {
       IndexSearcher indexSearcher = new IndexSearcher(idxReader);

       MoreLikeThis mlt = new MoreLikeThis(idxReader);
       mlt.setMinTermFreq(0);
       mlt.setMinDocFreq(0);
       mlt.setFieldNames(new String[] { "content" });
       mlt.setAnalyzer(analyzer);

       Reader sReader = new StringReader("Melhorias no controle de sessão no sistema qquercoisa quando expira, ao logar novamente no sistema é exibido o erro "xpto");

       Query query = mlt.like("content", sReader);

       TopDocs topDocs = indexSearcher.search(query, 3);
}

Answer 1

Well, I decided to take a look inside MoreLokeThis class and I found the answer.

The Query query = mlt.like("content", sReader); call the createQueue(Map<String, Int> words) method in MoreLokeThis class.

Inside it, the tokenized terms/words from sReader (that were converted to a Map) are checked against the index.

Only terms/words that are present into the index are used to create a query.

Using the example that I provided, since my index contains only a document with the word erro, this is the only word that is kept from the phrase that I passed.