Search code examples
apachesolrtokenize

SOLR 8.1.1 EdgeNGramFilterFactory parsing query


I have a SOLR 4.10.2 core, and I am upgrading to 8.1.1.

I created an 8.1.1 core manually using the default_config set , and then brought over settings into the 8.1.1 schema.

I have adjusted the schema.xml and solrconfig.xml, and I have the core queryable in 8.1.1.

I have a field named Company:

<field name="Company" type="string" indexed="true" stored="true"/>
<field name="IDX_Company" type="text_general" indexed="true" stored="false" multiValued="true" />
<copyField source="Company" dest="IDX_Company"/>

In 4.10.2 when I run the query:

IDX_Company:blue

with debugQuery on, I see the query parsed into pieces (correctly)

"debug": {
    "rawquerystring": "IDX_Company:blue",
    "querystring": "IDX_Company:blue",
    "parsedquery": "(IDX_Company:b IDX_Company:bl IDX_Company:blu IDX_Company:blue)/no_coord",

...

When I run this against 8.1.1, with debugQuery on, I get the following:

"debug":{
    "rawquerystring":"IDX_Company:blue",
    "querystring":"IDX_Company:blue",
    "parsedquery":"IDX_Company:blue",

...

It seems to not be applying the EdgeNGramFilterFactory - the only change I made to the EdgeNGramFilterFactory configuration was to remove the "side" attribute, per the documentation. Also, per the documentation, I replaced the SynonymFilterFactory with SynonmGraphFilterFactory, and added the FlattenGraphFilterFactory.

I have tried removing the FlattenGraphFilterFactory, I have cleared and repopulated the core (reindexed), I have stopped and started SOLR 8.1.1, and no difference.

Here is the definition of text_general I am using in schema.xml

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="true">
      <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="15"/> <!-- RDH - removed side="front"-->
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <!-- RDH SynonymFilterFactory has been deprecated, replace with SynonymGraphFilterFactory -->
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> 
        <!-- RDH https://lucene.apache.org/solr/guide/8_1/filter-descriptions.html
            Flatten Graph Filter
            This filter must be included on INDEX-time analyzer specifications that include at least one graph-aware filter, including Synonym Graph Filter and Word Delimiter Graph Filter.
        -->
        <filter class="solr.FlattenGraphFilterFactory"/>  
        <filter class="solr.LowerCaseFilterFactory"/>
        <!-- strip all punctuation -->
        <filter class="solr.PatternReplaceFilterFactory" pattern="[^\p{L}\p{N} ]" replacement=" " replace="all" /> <!-- RDH -->
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>       
        <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="15"/> <!-- RDH - removed side="front"-->
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <!-- RDH SynonymFilterFactory is deprecated, replace with SynonymGraphFilterFactory -->
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <!-- RDH https://lucene.apache.org/solr/guide/8_1/filter-descriptions.html
            Flatten Graph Filter
            This filter must be included on INDEX-time analyzer specifications that include at least one graph-aware filter, including Synonym Graph Filter and Word Delimiter Graph Filter.
        -->
        <filter class="solr.FlattenGraphFilterFactory"/>  
        <filter class="solr.LowerCaseFilterFactory"/>
        <!-- strip all punctuation -->
        <filter class="solr.PatternReplaceFilterFactory" pattern="[^\p{L}\p{N} ]" replacement=" " replace="all" /> <!-- RDH -->
      </analyzer>
    </fieldType>

Solution

  • Although I was reloading the information by clearing and posting the data into the core, I had neglected to go to the Core Admin page, choose the core, and click on the Reload button.

    Now query is parsed as expected.