How to calculate confidence score of OCR system?

Question

I am working on an OCR project and I wonder how I can calculate the confidence score of my OCR system.

I have digital multi meter images. There are some measurement results on the screens of devices in images. I want to recognize those values. However, according to my research, I am not sure which OCR confidence calculation technique is fit for my system.

As I understand OCR confidence scores can calculate in character, word and sentence wise. Actually, the last two methods build on character confidence scores. In my case, character wise calculation might be wrong or not sufficient.

For example, I have "40.245 V" text. I got two different recognition result like "40.247 V" and "70.245 V". If I am not wrong both of the result will have same or close confidence scores. Yet "40.247 V" prediction is acceptable and "70.245 V" is not acceptable in my case.

Is there any idea how to calculate confidence score for this case?

Answer 1

When calculating confidence you generate a weighted average of your confidences to give more weight for the first characters & less to the last ones.

#include <iostream>
#include <vector>
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>

using namespace std;

double getWeightedConfidence(vector<pair<char /* character */, double /*confidence of that character */>> word) {
    if (word.empty()) {
        return 1.0;
    }
    
    double confidence = 0;
    
    if (isdigit(word[0].first)) {
        // okay it is a number
        
        double weight = 1;
        double sumOfWeights = 0;
        for (const auto &c : word) {
            confidence += c.second * weight;
            sumOfWeights += weight;
            weight /= 10; // you can decay it by whatever number you want based on how much do you think next digit is less valueble then previous
        }
        
        confidence /= sumOfWeights;
    } else {
        // not a number - just calculate a normal average
        for (const auto &c : word) {
            confidence += c.second;
        }
        
        confidence /= word.size();
    }
    
    return confidence;
}

int main() {
    
    vector<pair<char, double>> number_with_first_digit_wrong;
    number_with_first_digit_wrong.emplace_back('7', 0.1);
    number_with_first_digit_wrong.emplace_back('4', 0.9);
    number_with_first_digit_wrong.emplace_back('6', 0.9);
    number_with_first_digit_wrong.emplace_back('2', 0.9);
    number_with_first_digit_wrong.emplace_back('.', 0.9);
    number_with_first_digit_wrong.emplace_back('9', 0.9);
    
    vector<pair<char, double>> number_with_last_digit_wrong;
    number_with_last_digit_wrong.emplace_back('7', 0.9);
    number_with_last_digit_wrong.emplace_back('4', 0.9);
    number_with_last_digit_wrong.emplace_back('6', 0.9);
    number_with_last_digit_wrong.emplace_back('2', 0.9);
    number_with_last_digit_wrong.emplace_back('.', 0.9);
    number_with_last_digit_wrong.emplace_back('9', 0.1);
    
    
    cout << getWeightedConfidence(number_with_first_digit_wrong) << " " << getWeightedConfidence(number_with_last_digit_wrong) << endl;
    
    return 0;
}

Something simple as that gives result:

0.179999 - when 0.1 is the confidence of first digit (and others are 0.9) 0.899993 - when 0.1 is the confidence of the last digit (and others are 0.9)

You can specify different weights if you consider certain positions more valuable then the others.