Search code examples
azure-qna-maker

Does QnAMaker Service sort the answers by score if there are multiple?

We are using microsoft's QnAMaker service to build a FAQ bot. I would like to know if the api response has the answers sorted when there are multiple matches with different scores. Does the api return the answer with the top score first?

We tested it with a sample but wanted to make sure that there is no other factor playing a role here. Kindly let me know.

Thanks!

Solution

  • You can read about how QnA Maker processes a user query in the official docs:

    enter image description here

    1. The client application sends the user query to the GenerateAnswer API.
    2. QnA Maker preprocesses the user query with language detection, spellers, and word breakers.
    3. This preprocessing is taken to alter the user query for the best search results.
    4. This altered query is sent to an Azure Cognitive Search Index, which receives the top number of results. If the correct answer isn't in these results, increase the value of top slightly. Generally, a value of 10 for top works in 90% of queries.
    5. QnA Maker uses syntactic and semantic based featurization to determine the similarity between the user query and the fetched QnA results.
    6. The machine-learned ranker model uses the different features, from step 5, to determine the confidence scores and the new ranking order.
    7. The new results are returned to the client application in ranked order.

    Features used include but aren't limited to word-level semantics, term-level importance in a corpus, and deep learned semantic models to determine similarity and relevance between two text strings.

    The Typescript SDK also specifies the following on the generateAnswer method:

    Returns an array of answers sorted by score with the top scoring answer returned first.

    You should also be aware that multiple resources are involved in the QnA Maker ranking process - Azure Cognitive Search and QnA Maker.

    The Cognitive Search resource is used to store the QnA pairs and provide the initial ranking (ranker #1) of the QnA pairs at runtime.

    The QnA Maker resource provides access to the authoring and publishing APIs as well as the natural language processing (NLP) based second ranking layer (ranker #2) of the QnA pairs at runtime.

    The second ranking applies intelligent filters that can include metadata and follow-up prompts.