Search code examples
solrlucenetokenizen-gram

Front and back EdgeNGrams in Solr

I would like to use EdgeNGramFilterFactory to generate Edge NGrams from front and back. For front I am using

  <filter class="solr.EdgeNGramFilterFactory" maxGramSize="20" minGramSize="4"/>

and for back, I am using

  <filter class="solr.ReverseStringFilterFactory"/>
  <filter class="solr.EdgeNGramFilterFactory" minGramSize="4" maxGramSize="15"/>
  <filter class="solr.ReverseStringFilterFactory"/>

But when they are used together in a single analyzer, the second set of filter factories are acting on the output of the first EdgeNGramFilterFactory.

Is it possible to generate both front and back EdgeNGrams in a single analyzer? Or do I have to create separate analyzers and use copyField to create a field with both the front and back EdgeNGrams?

Update Example schema as requested in comments below

<fieldType name="text_suggest_edge" class="solr.TextField">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
  </analyzer>
</fieldType>

<fieldType name="text_suggest_edge_end" class="solr.TextField">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ReverseStringFilterFactory"/>
    <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
    <filter class="solr.ReverseStringFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ReverseStringFilterFactory"/>
    <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
    <filter class="solr.ReverseStringFilterFactory"/>
  </analyzer>
</fieldType>

<field name="item_name_edge" type="text_suggest_edge" indexed="true" stored="false" multiValued="true"/>
<field name="item_name_edge_end" type="text_suggest_edge_end" indexed="true" stored="false" multiValued="true"/>
<copyField source="item_name" dest="item_name_edge"/>
<copyField source="item_name" dest="item_name_edge_end"/>

Update 2: Including sample input and expected output

Input String

Washington

Required Edge Ngrams

Was, Wash, Washi, ... Washington, ashington, shington, hington ... gton, ton

Solution

  • you could do it in a single analyzer chain if you create your customized version of EdgeNGramFilterFactory (in java, and then plug it into your schema.xml) that creates the additional ngrams from the back.

    Otherwise, you are going to need the copyField into an additional field with a separate chain.

    I honestly thing the first option is too much trouble, but it is possible for sure.