Search code examples
javaindexinglucenefull-text-searchinverted-index

Is it necessary to save the field itself to generate inverted index in lucene

I am saving a fields id and msg in lucene to support full text search. When user enters a keyword, lucene performs a search and show results and when user clicks on that using the saved id, I open fetch the result from DB with more details to display.

Is it possible for lucene to save just inverted index on msg filed instead of entire msg field and hand me over just the ids so that I can fetch the results from DB to show results ?

By doing this my data will not be redundant.

Solution

  • In Lucene, what you store for document creation/fetch purposes (The purpose that you describe via DB ) and what you index as inverted index for search purpose are two very different things.

    You have not shown any code as how you Index.

    Storing data in Lucene is Optional and folks choose to store data too to avoid additional DB calls etc at the cost of storage space but keeping data in master source results in less sync requirements too. So there is a trade - off.

    In below code snippet, DOC_ID is Indexed and Stored as well while TEXT_FIELD is simply indexed and not stored. 

    Document doc = new Document();
doc.add(new Field("DOC_ID", "DOCONE", new FieldType(
            TextField.TYPE_STORED)));
doc.add(new Field("TEXT_FIELD", "This", new FieldType(
            TextField.TYPE_NOT_STORED)));
writer.addDocument(doc);

    So after searching, let say your hit document is - hitDoc so you will get below, hitDoc.getField("DOC_ID")="DOCONE" and hitDoc.getField("TEXT_FIELD")=NULL

    Both fields are Indexed but only one is stored. Indexed means it can be searched.