add custom loss to eval_metric_ops

Question

I have defined my own loss function using sequence_loss

loss = tf.contrib.legacy_seq2seq.sequence_loss(logits, labels, weights)

I am hoping to add this to eval_metric_ops so that in my ML engine package I can display the evaluation loss in tensorboard continuously (defaults are just accuracy).I tried adding this as a custom eval_metric_ops

eval_metric_ops = {
    'loss': loss # this has already been coputed for Modes.EVAL
}

However, I get the error "TypeError: Values of eval_metric_ops must be (metric_value, update_op) tuples, given: Tensor("sequence_loss/truediv:0", shape=(), dtype=float32) for key: loss" -- what do I need to do to pass the loss as an eval_metric_op? I guessing that my current loss should be the metric value, but I'm not sure what the update_op should be?

Answer 1

The metric function in your case can be implemented using tf.metrics using an moving average of the loss:

def metric_fn(labels, predict):
   loss = tf.contrib.legacy_seq2seq.sequence_loss(logits, labels, weights)
   mean, op = tf.metrics.mean(loss)
   return mean, op