solr autosuggest with diacritics

Question

I'm using solr4 with the TermsComponent Autosuggest (as described here) We're doing a regEx "startsWith"-search, that ignores upper/lower-case, the whole searchQuery looks like this:

<solr>/terms
?terms.fl=name
&terms=true
&terms.limit=5
&terms.regex=<term>.*
&terms.regex.flag=case_insensitive
&qt=%2Fterms

Let me give you a few examples what that returns:

test -> Test Listing; test lowercase
Test -> Test Listing; test lowercase

Unfortunately, this solution can't handle diacritics, umlaute, accents .. So the following won't work:

têst -> Test Listing; test lowercase; Têst áccènt
Test -> Test Listing; test lowercase; Têst áccènt

The field is a string - I've tried with a tokenized test_en as well, but without success

<field name="name" type="string" indexed="true" stored="true" required="true" />

What's the best way to enable bidirectional accent-searching for this autosuggest?

---

Edit: Changes for AnalyzerSuggester:

  <searchComponent class="solr.SpellCheckComponent" name="autosuggest">
    <lst name="spellchecker">
      <str name="name">autosuggest</str>
      <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
      <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
      <str name="storeDir">autosuggest</str>
      <str name="buildOnCommit">true</str>
      <str name="field">asug</str>
      <str name="suggestAnalyzerFieldType">text_asug</str>

      <!-- Suggester properties -->
      <bool name="exactMatchFirst">true</bool>
    </lst>
  </searchComponent>
  <requestHandler class="org.apache.solr.handler.component.SearchHandler" name="/autosuggest">
    <lst name="defaults">
      <str name="spellcheck">true</str>
      <str name="spellcheck.dictionary">autosuggest</str>
      <str name="spellcheck.onlyMorePopular">true</str>
      <str name="spellcheck.count">5</str>
      <str name="spellcheck.collate">true</str>
    </lst>
    <arr name="components">
      <str>autosuggest</str>
    </arr>
  </requestHandler>

.

<fieldType name="text_asug" class="solr.TextField" positionIncrementGap="100">
        <analyzer type="index">
        <tokenizer class="solr.KeywordTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.KeywordTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
      </analyzer>
    </fieldType>

Answer 1

The problem here is that the term component works on indexed tokens both for searching and query. So, if you do unicode folding (which is what you do), you will store folded text version. You will match it without accents, but then you will also get it back without accents too.

I can think of two options:

1) Store folded and non-folded term in one field. So somehow get "Têst áccènt" to map to "Test accent|Têst áccènt". You match at the prefix "Test.." and then extract second term on the client. How to do that could be tricky though.

2) Use Suggester instead. This builds on top of spell-checker and - if I read documentation properly - allows to specify alternative field_type, whose analyzers are used during suggester's index/query (using barely-documented queryAnalyzerFieldType parameter in solrconfig.xml). So, your original text gets copied into suggester in folded form. But, presumable, once Suggester matches something, it will return the original form. However, I am not sure. Mostly because, it is advertised as a feature for just born Lucene/Solr 4.1 AnalyzingSuggester. In fact, the article specifically covers your use case:

With an analyzer that folds or normalizes case, accents, etc. (e.g., using ICUFoldingFilter), the suggestions will match irrespective of case and accents. For example, the query "ame..." would suggest Amélie.

The problem is that you need to put together a complete example yourself at this point. There is very little guidance. But it (AnalyzingSuggester) is probably your best bet.