There is a mismatch between the score for a wildcard match and an exact match
I search for
recording:live OR recording:luve*
And here is the Explain Output from Search
DocNo:0:1.4196585:11111111-1cf0-4d1f-aca7-2a6f89e34b36
1.4196585 = (MATCH) max plus 0.1 times others of:
0.3763506 = (MATCH) ConstantScore(recording:luve*), product of:
1.0 = boost
0.3763506 = queryNorm
1.3820235 = (MATCH) weight(recording:luve in 0), product of:
0.7211972 = queryWeight(recording:luve), product of:
1.9162908 = idf(docFreq=1, maxDocs=5)
0.3763506 = queryNorm
1.9162908 = (MATCH) fieldWeight(recording:luve in 0), product of:
1.0 = tf(termFreq(recording:luve)=1)
1.9162908 = idf(docFreq=1, maxDocs=5)
1.0 = fieldNorm(field=recording, doc=0)
DocNo:1:0.3763506:22222222-1cf0-4d1f-aca7-2a6f89e34b36
0.3763506 = (MATCH) max plus 0.1 times others of:
0.3763506 = (MATCH) ConstantScore(recording:luve*), product of:
1.0 = boost
0.3763506 = queryNorm
In my test I have 5 documents one contains an exact match, another a wildcard match and the other three do not match all. The score of the exact match is 1.4 compared to 0.37 for the wildcard match, thats nearly a factor of 4. With a much larger index the score for an exact match on a rare term compared to a wildcard search would be even higher.
The whole difference is due to the different scoring mechism used for wildcard to exact match, wildcards don't take tf/idf or lengthnorm into account you just get a constant score for each match. Now I'm not bothered about tf or lengthnorm in my data domain it doesnt make much difference but the idf score is a real killer. Because the matching doc is found once in 5 documents its idf contribution is idf squared i.e 3.61
I know this constant score is quicker than calculating the tf*idf*lengthnorm for each wildcard match but it doesn't make sense to me for the idf to contribute so much to the score. I also know I can change the rewrite method but there are two problems with this.
Scoring rewrite methods perform less well because they are calculating idf, tf and lengthnorm. idf is the only value I need.
Ones that do calculate the score dont make much sense either as they would calculate the idf of the matching term even though this isn't what was actually search for and this term could be rarer than what I was actually searching for, possibly boosting it higher than the exact match.
(I could also change the similarity class to override the idf calculation so it always returns 1 but that doesn't make sense because the idf is very useful for comparing exact matches to different words
i.e recording:luve OR recording:luve* OR recording:the OR recording:the*
I would want matches to luve to score higher than matches to the common word the )
So does a rewrite method already exist or is possible for it to just calculate the idf of the term it was trying to match to so for example in this case I search for 'luve' and the wildcard matches on 'luvely' that it would multiple the luvely match by the idf of luve (3.61). This way my wildcard match would be comparable to the exact match and I can just change my query to boost the exact match slightly so exact match would always score higher than wildcard match but not too much higher
i.e
recording:live^1.2 OR recording:luve*
and with this mythical rewrite method this would give (depending on queryNorm):
Okay setting this as the rewrite method for prefix queries seems to work
public static class MultiTermUseIdfOfSearchTerm<Q extends Query> extends TopTermsRewrite<BooleanQuery> {
//public static final class MultiTermUseIdfOfSearchTerm extends TopTermsRewrite<BooleanQuery> {
private final Similarity similarity;
/**
* Create a TopTermsScoringBooleanQueryRewrite for
* at most <code>size</code> terms.
* <p>
* NOTE: if {@link BooleanQuery#getMaxClauseCount} is smaller than
* <code>size</code>, then it will be used instead.
*/
public MultiTermUseIdfOfSearchTerm(int size) {
super(size);
this.similarity = new DefaultSimilarity();
}
@Override
protected int getMaxSize() {
return BooleanQuery.getMaxClauseCount();
}
@Override
protected BooleanQuery getTopLevelQuery() {
return new BooleanQuery(true);
}
@Override
protected void addClause(BooleanQuery topLevel, Term term, float boost) {
final Query tq = new ConstantScoreQuery(new TermQuery(term));
tq.setBoost(boost);
topLevel.add(tq, BooleanClause.Occur.SHOULD);
}
protected float getQueryBoost(final IndexReader reader, final MultiTermQuery query)
throws IOException {
float idf = 1f;
float df;
if (query instanceof PrefixQuery)
{
PrefixQuery fq = (PrefixQuery) query;
df = reader.docFreq(fq.getPrefix());
if(df>=1)
{
idf = (float)Math.pow(similarity.idf((int) df, reader.numDocs()),2);
}
}
return idf;
}
@Override
public BooleanQuery rewrite(final IndexReader reader, final MultiTermQuery query) throws IOException {
BooleanQuery bq = (BooleanQuery)super.rewrite(reader, query);
float idfBoost = getQueryBoost(reader, query);
Iterator<BooleanClause> iterator = bq.iterator();
while(iterator.hasNext())
{
BooleanClause next = iterator.next();
next.getQuery().setBoost(next.getQuery().getBoost() * idfBoost);
}
return bq;
}
}