Solr, Special Chars, and Latin to Cyrillic char conversion

Question

I am trying to setup a search engine using Solr (or Lucene) which could have text in both Latin with special chars, (special chars would include Ö or Ç as an example) or Cyrilic chars (examples include Б or б and Ж ж).

Anyway, I am trying to find a solution to allow me to search for words with these charicters in them, but for users who do not have the key on their keyboard...

Example would be (making up words here, hopefully won't offend anyone):

• "BÖÖK" would be found when searching for "book"
• "ЖRAY" would be found when searching for XRAY
• "ЖRAY" would also be found if searching for ZRAY, ZHRAY, or žray (see GOST 16876-71 for info on Transliteration of Cylric to Latin Char.

So, how should I go about this? Some theories I have are:

• allow multiple text fields to be stored for each original string, one in original form, one in the first pass of transliteration (which, for example, would convert Ö to just O and Ж to ž, but also X) and then one in the third form (from the ž to z or zh) -> means I will be storing a LOT of data...
• store in solr as is, and let Solr do the magic -> don't know how well this will work... can't see anything in solr to do this
• Magic bullet I have not found yet...

Any ideas? Anyone tried this before?

Answer 1

Take a look at Solr's Analyzers, Tokenizers, and Token Filters which give you a good intro to the type of manipulation you're looking for.