Search code examples
azure-cognitive-searchazure-qna-maker

QnA maker - Different Results between different azure search Pricing tier


QnAmaker returns different scores even the knowledgebases are completely the same. My 2 knowledgebases are from different qnamakers. I have read confidence-score-differences. So both of them are in the same region. And I tested them in a published environment. By the way, 2 testkb are not the same, neither is price trier.

Is it normal?


Solution

  • Azure Search scores are not to be used as absolute values to be compared between different indexes or even between different queries. They are meant to be used as a relative value to compare results within a single query for ranking purpose. Scores are calculated using various variables, some of those being part of the index statistics.

    Those index statistics (such as document frequencies) are in turn calculated using a process which divide your index into multiple "shards". When you re-create your index (regardless of if you use a different SKU or not), the way your data is distributed in those shards changes, which can result in a difference in index statistics. Usually, in large enough indexes, those statistics stabilize and the differences becomes much less noticeable, however, if you have fewer documents in your index, the difference in statistics between indexes containing the same data can be more noticeable.

    So to answer your question, there's no difference in scoring logic between the various Azure Search SKUs, however, you should expect scoring not to be stable between two different indexes, even if both contains the same documents.