QnA Api returns response code 200 OK irrespective of no match in the KB

Question

I have been trying to code an automation test for the QnA of my application. Today while running the tests I noticed that all the tests were passing irrespective of wrong inputs I gave to couple of QnA's.

When I checked, I found out that the response code returned by the QnA Api with valid headers like the authentication key was always 200 Ok irrespective of whether the input to the particular QnA was right or wrong. Due to this, all my tests were passing since the step which verified that the response code was 200 Ok always kept passing. The response was also in proper Json format so the formatting test also kept passing.

I understand that this is the reason that the Api keeps returning 200 Ok status code but I believe that ideally it should respond with the status code in respect to the response obtained in the response body. The only difference was that when an invalid input was given to the Api,the 'answer' response body parameter returned an error text message like: "No good match found in KB." This can be seen below.

Whereas in case of a valid input to the BOT, the below response is obtained. Note that the only difference is in the value of the 'answer' response parameter in the response body.

Finally I had to add validators to verify that the tests returning 200 Ok also did not return the error message text in response body 'answer' parameter. But now if the error response message text changes (changes done by the services) in future my tests will fail as well since they have been set to compare with the particular error message.

The way I see it, this is only a temporary solution to the automation code level testing of the QnA Apis. Is there anyway one can modify the response code returned by these Apis to reflect the actual response in the response body. If that can be done then one can stop the tests as soon as the response codes do not match, saving time, effort and improving the performance. If there is any other way to improve this working, I'm open to suggestions.

NOTE: I was using the POST call methods for testing my Apis.

Answer 1

HTTP status codes are just that. They give you information specific to HTTP requests, so for example you might expect a different code if the request failed to reach the server. In the case of your QnA Maker app, the purpose of the app is to search the knowledge base and return a detailed response, knowing that it's possible for the knowledge base to not include the question that was asked. If the QnA Maker app searches the knowledge base and finds no good answer then the app has still done its job and performed correctly, and so if it's able to send the information back that it wasn't able to find an answer then there's no reason for the HTTP status code to indicate anything other than success. This is not a bug.

If you want to determine that no match was found, you should not be using the HTTP status code to do it. You say the "only difference" in the response is that you get an answer like "No good match found in KB," but have another look at that response. There's only one answer and the score is 0, but that may not be as reliable as what you want. The real indicator is that the ID is -1 and the source is null and the questions array is empty, so you could easily do something like this:

if (results.First().Id < 0)
{
    // No good match was found
}

You also have a misunderstanding that the QnA Maker team could possibly change your "no good match" response by modifying some kind of shared service. You have your own QnA Maker app that you're calling the endpoint for, and that's where the message is defined. The only person who can change it is you, as explained here and here.