Azure QnAMaker see all relevant answers in logs

Question

I've started to use QnA Maker and want to be able to view in the logs ALL answers to a question and the score threshold that was sent with the request. Currently I've only been able to see just one answer to a question with this Kusto query.

requests
| where url endswith "generateAnswer"
| project timestamp, id, url, resultCode, duration, performanceBucket
| parse kind = regex url with *"(?i)knowledgebases/"KbId"/generateAnswer"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| extend question = tostring(customDimensions['Question'])
| extend answer = tostring(customDimensions['Answer'])
| extend score = tostring(customDimensions['Score'])
| project timestamp, resultCode, duration, id, question, answer, score, performanceBucket, KbId

Does anyone know if it's possible to see all answers?

Answer 1

I'm using this exact example, and it doesn't appear that it is possible to see a full answer list with the default logging. You would need to log a custom trace (or you could probably do as an event) and use the payload returned by the API call. You also need to set the top parameter in the request, otherwise you will receive only one answer regardless (though I've confirmed setting top does not shown the additional answers in the default app insights logging).

Further, application insights doesn't support arrays (to my knowledge), so you will need to flatten the results using something like flat.

Here is some example code in nodejs. The syntax is a little different for other languages (see link above for custom trace). I'm calling the API directly, but if you're importing a QnA module I should expect this would work the same way.

const qnaResult = await request({
    url: url,
    method: 'POST',
    headers: headers,
    json: {
        question: query,
        top: 3,
        context: qnAcontext
    }
});
flatQnaResult = flatten(qnaResult);
appInsightsClient.trackTrace({message: 'QnA Custom Trace',severity: 1,properties: flatQnaResult});

If you use flat you'll find the relevant answer and score as answers.i.answer and answers.i.score where i is your index. What this will not do is show what threshold you have set. Perhaps that is because I handle that outside of the QnA Maker call to give me more control over the actions (e.g. in some bots, if a threshold is not met in a primary call, I make a call to a separate QnA KB instead of providing default message).

I'm not 100% sure if you will get threshold back in the payload if you set it as part of the call. If not, you could always add it to your flatQnaResults object before logging it to app insights (e.g. flatQnaResults.threshold = YOUR_THRESHOLD).