I am trying to understand the correct usage of cntk.metrics.classification_error() and use it to verify a batch of predictions against their ground truths.
The below toy example (based on the Python API docs):
import numpy as np
from cntk.metrics import classification_error
predictions = np.asarray([[1., 2., 3., 4.],[1., 2., 3., 4.],[1., 2., 3., 4.]], dtype=np.float32)
labels = np.asarray([[0., 0., 0., 1.],[0., 0., 0., 1.],[0., 0., 1., 0.]], dtype=np.float32)
classification_error(predictions, labels).eval()
yields the following result:
array([[ 0., 0., 1.],
[ 0., 0., 1.],
[ 0., 0., 1.]], dtype=float32)
Is there a way I can obtain a vector rather than a square matrix which appears inefficient given I would like to process a large batch?
I've tried using the axis keyword when calling classification_error(), but whether I set axis=0 or axis=1 I get an empty result.
This happens because CNTK is trying to be user-friendly and ends up being confused about the types :-) You can tell because the classification error is not even correct.
If you add a little bit of typing information it gets the semantics right.
p = C.input(4)
y = C.input(4)
classification_error(p, y).eval({p:predictions, y:labels})
array([[ 0.],
[ 0.],
[ 1.]], dtype=float32)
We will work on a fix that will prevent the confusion.