tensorflow confusion matrix in Experimenter during evaluation

Question

I've got some troubles during the model evaluation using Tensorflow with the Experimenter API.

I used to work using 2-classes NN, but this time I manage to train a 4-classes one and I need to figure out how to build a confusion matrix in this case. I tried using the tf.confusion_matrix function, but it doesn't work at all.

This is the fragment of code that I used:

if mode == ModeKeys.EVAL:

    eval_metric_ops = {
        'accuracy' : metrics.streaming_accuracy(predictions=predicted_classes, labels=labels),

        # Other metrics...

        'confusion_matrix': tf.confusion_matrix(prediction=predicted_classes, label=labels, num_classes=4)
    }

    return tf.estimator.EstimatorSpec(
        mode=mode,
        predictions=predicted_classes,
        loss=loss,
        eval_metric_ops=eval_metric_ops
    )

And this is the error that I got:

TypeError: Values of eval_metric_ops must be (metric_value, update_op) tuples, given: (<tf.Operation 'test/group_deps' type=NoOp>, <tf.Tensor 'test/accuracy/value:0' shape=() dtype=float32>, <tf.Variable 'test/confusion:0' shape=(4, 4) dtype=int32_ref>) for key: confusion_matrix

I read other answers about to create a confusion matrix in Tensorflow and I understood how to do it, but I think that my question is more related to the Estimator/Experimenter API.

Answer 1

your codes doesn't work because the framework expects eval_metric_ops to be a dictionary containing key with the name of the operation and values of type tuple (result tensor, update_operation for this tensor)

tf.confusion_matrix(prediction=predicted_classes, label=labels, num_classes=4) only returns the expected tensor.

You have to implement your own metric operation like this:

def eval_confusion_matrix(labels, predictions):
    with tf.variable_scope("eval_confusion_matrix"):
        con_matrix = tf.confusion_matrix(labels=labels, predictions=predictions, num_classes=4)

        con_matrix_sum = tf.Variable(tf.zeros(shape=(4,4), dtype=tf.int32),
                                            trainable=False,
                                            name="confusion_matrix_result",
                                            collections=[tf.GraphKeys.LOCAL_VARIABLES])


        update_op = tf.assign_add(con_matrix_sum, con_matrix)

        return tf.convert_to_tensor(con_matrix_sum), update_op



# Add evaluation metrics (for EVAL mode)
eval_metric_ops = {
    "accuracy": tf.metrics.accuracy(labels, predicted_classes),
    "conv_matrix": eval_confusion_matrix(
        labels, predicted_classes)
    }