MS Cognitive service Face API confidence scores changing with releases/learning?

Question

We've gone through an extensive exercise comparing facial recognition/matching providers using our local facial image data sets.

MS Cognitive services came out tops in terms of False Rejects Rate (FRR) for a given False Accept Rate (FAR). We are busy deciding on pass thresholds for different image type matching (selfie vs document, etc.).

The question is, if we are using a specific version (https://{endpoint}/face/v1.0/) and fixed parameters for the Detect and Verify endpoints (recognitionModel = recognition_02 and detectionModel = detection_02), can we expect to see a change in the confidence score for the same two images over time or whenever Microsoft releases a new version? Our concern is that we pick a pass threshold based on our test results and current confidence scores, and then the scores change in future due to machine-learning/releases, meaning we would continuously have to re-adjust our thresholds.

https://westus.dev.cognitive.microsoft.com/docs/services/563879b61984550e40cbbe8d/operations/563879b61984550f3039523a

Answer 1

I think it's a good question about the stability of model function of MS Azure Cognitive Services like Face API. Based on my knowledge for Machine Learning, there are some possible reason that will cause the issue as you said, as below.

1. The structure of Machine Learning Model will be changed with the service upgrading.
2. The upgraded service version start to support a new API with new parameters which be different from the current one.

Sure, I think the two above will absolute possibly happen. However, there are three reason let me believe that will not effect yours too much.

1. MS as a big cloud provider in the marketplace, for the same application scenario, it will make sure the upgraded service return the same output date for the same input data as the previous one, even to upgrade the ML model for improving better performance. Then, MS can keep the regular customers continous to use and pay for their Azure subscriptions.
2. Except some preview services, MS as a success IT company, it will keep its service features compatible with the previous one, like MS Office 365 still be compatible with the older version.
3. If the incompatibility really happen in the new version, I think MS will give the migration guide for users of the older version.

Considering for the worst case, technically speaking, there are many opensource face recognization solutions as the backup for you. It's nothing really matter.