I'm not sure if I understand tf.metrics and tf.contrib.slim.metrics correctly.
Here is the general flow of the program:
# Setup of the neural network...
# Adding some metrics
dict_metrics[name] = compute_metric_and_update_op()
# Getting a list of all metrics and updates
names_to_values, names_to_updates = slim.metrics.aggregate_metric_map(dict_metrics)
# Calling tf.slim evaluate
slim.evaluation.evaluation_loop(eval_op=list(names_to_updates.values()), ...)
Let's assume I want to compute the accuracy. I have two options: a) Compute the accuracy over all pixels in all images in all batches b) Compute the accuracy over all pixels in one image and take the average of all accuracies over all images in all batches.
For version a) this is what I would write:
name = "slim/accuracy_metric"
dict_metrics[name] = slim.metrics.streaming_accuracy(
labels, predictions, weights=weights, name=name)
Which should be equivalent to:
name = "accuracy_metric"
accuracy, update_op = tf.metrics.accuracy(
labels, predictions, weights=weights, name=name)
dict_metrics[name] = (accuracy, update_op)
Furthermore, it should be pointless or even wrong to add this line
dict_metrics["stream/" + name] = slim.metrics.streaming_mean(accuracy)
Because the accuracy I get from tf.metrics.accuracy is already computed over all the batches via the update_op. Correct?
If I go with option b), I can achieve the effect like this:
accuracy = my_own_compute_accuracy(labels, predictions)
dict_metrics["stream/accuracy_own"] = \
slim.metrics.streaming_mean(accuracy)
Where my_own_compute_accuracy() computes the symbolic accuracy for the labels and predictions tensor but does not return any update operation. In fact, does this version calculate the accuracy over a single image or a single batch? Basically, if I set the batch size to the size of the complete dataset, does this metric then matches the output of slim.metrics.streaming_accuracy?
Lastly, if I add the same update operation twice, will it be called twice?
Thank you!
Yes, the slim streaming accuracy computes the mean of the per-batch accuracy over the entire dataset (if you only do one epoch).
For your accuracy function it depends on how you implement it.