How do you identify questions that the bot could not answer

Question

My organisation is starting to experiment with the Microsoft bot framework. One of the questions our enterprise architect has asked is as follows:

How do we identify questions that the bot was unable to answer?

I've checked the documentation but I'm still unclear. Can anyone elaborate on the techniques that they use to identify unanswered questions? We feel this is important as it identifies opportunities for further growth.

Answer 1

You can achieve this using a number of techniques. Essentially, what you are trying to do is to store any questions the Bot has not been able to provide an answer for analysis.

You can do this by using the scoring mechanism in the QnAMaker. For example, if the QnAMaker returns a score of zero, an answer doesn't exist, so we need to write that question back to storage for analysis.

You can use a number of storage solutions for this in the Azure stack, such as Application Insights, Cosmos, Blob, SharePoint Lists etc.

In the example below (code trimmed for brevity), I'm using Application Insights to store this information. I have imported the botbuilder-applicationinsights package and have created a simple custom event to capture any responses that score zero against the QnAMaker.

const {
    ApplicationInsightsTelemetryClient,
    ApplicationInsightsWebserverMiddleware
} = require('botbuilder-applicationinsights');

const {
    MessageFactory,
    CardFactory
} = require('botbuilder');

const {
    QnAServiceHelper
} = require('../helpers/qnAServiceHelper');

const {
    CardHelper
} = require('../helpers/cardHelper');

const {
    FunctionDialogBase
} = require('./functionDialogBase');

// Setup Application Insights
settings = require('../settings').settings;
const appInsightsClient = new ApplicationInsightsTelemetryClient(settings.instrumentationKey);

class QnADialog extends FunctionDialogBase {

    constructor() {
        super('qnaDialog');
    }

    async processAsync(oldState, activity) {

        var newState = null;
        var query = activity.text;
        var qnaResult = await QnAServiceHelper.queryQnAService(query, oldState);
        var qnaAnswer = qnaResult[0].answer;
        var qnaNonResponse = qnaResult[0].score;

        var prompts = null;
        if (qnaResult[0].context != null) {
            prompts = qnaResult[0].context.prompts;
        }

        var outputActivity = null;
        if (prompts == null || prompts.length < 1) {
            outputActivity = MessageFactory.text(qnaAnswer);

        } else {
            var newState = {
                PreviousQnaId: qnaResult[0].id,
                PreviousUserQuery: query
            }

            outputActivity = CardHelper.GetHeroCard(qnaAnswer, prompts);
        }

        if (qnaNonResponse === 0) {

                const {
                    NonResponseCard
                } = require('../dialogs/non-response');
                const quicknonresponseCard = CardFactory.adaptiveCard(NonResponseCard);
                outputActivity = ({
                    attachments: [quicknonresponseCard]
                });
                console.log("Cannot find QnA response for" + " " + query);
                appInsightsClient.trackEvent({
                    name: "Non-response",
                    properties: {
                        question: query
                    }
                });
            }
        return ([newState, outputActivity, null]);
    }
}

module.exports.QnADialog = QnADialog;

I can then hook up the query I might use in Application Insights in Power Bi to surface those non-answered questions.

There are multiple ways to achieve this, but this was one I ended up going with.