Search code examples
matlabmachine-learningclassification

Calculation of sensitivity and specificity from confusion matrix

Consider a case where the number of labelled data as 0 = 1400 and labelled as 1 =100. The data labelled as 0 denote normal operating conditions and data labelled as 1 denote abnormal. An alarm is triggered only for abnormal events.

Assuming the following confusion matrix is obtained for the binary classification 

cmMatrix = 

                    predicted 0  predicted 1
           truth 0    1100 (TN)      300 (FP)
           truth 1    30 (FN)         70 (TP)


cmMatrix = [1100,300;30,70];
acc_0  = 100*(cmMatrix(1,1))/sum(cmMatrix(1,:));
acc_1  = 100*(cmMatrix(2,2))/sum(cmMatrix(2,:));

will give acc_0 = 78.5714 and acc_1 = 70

The confusion matrix is read as out of 1400 normal events, 1100 are correctly identified as normal and 300 are incorrectly identified as abnormal. Then, out of 100 abnormal events, 70 are correctly detected as abnormal whereas 30 are incorrectly detected as abnormal. I want to calculate the sensitivity and specificity for class 1 since that is of primary interest in abnormal event detection. This is how I did

Sensitivity = TP/(TP+FN) = 70/(70+30 ) = 0.70
Specificity = TN/(TN+FP) = 1100/(1100+300) = 0.78

  • Sensitivity would refer to the test's ability to correctly detect abnormal events. Why is Sensitivity so low and different than accuracy acc_1 which is so high (70%).

  • Is this calculation correct and what is the difference between individual class accuracies and sensitivity?

Did I do any mistake in the calculation?

Solution

  • Your calculations are correct. I think the imbalance between sensitivity and specificity is what is confusing you, given your relatively high accuracy. This is normal, and a common problem with classifiers especially when there is not an even split between the two classes. As a thought experiment, imagine that your classifier just arbitrarily decided that every sample was labelled. This would give you an overall 1400/1500 = 0.9333 accuracy, which might seem very good until you investigate the sensitivity and specificity, and it's all due to the difference in the size of the classes.

    Because of this, and the misleadingly high accuracy when you have a very low sensitivity, it might be more representative to show the F1 score: https://en.wikipedia.org/wiki/F1_score