How to add summaries to seq2seq tutorial?

I'm working with Seq2Seq example in TensorFlow. I can run the training and see the outputs of perplexity on the development set. It's great!

I just want to add summaries (especially scalar_summary such as perplexity on dev set) to the event file and monitor them in TensorBoard. After reading the documentation, I don't understand how to annotate translate.py with summary ops.

Can anybody can help me with simple pseudo-code?

Solution

It looks like translate.py doesn't create a TensorBoard summary log at all. (Part of the reason may be that much of the evaluation happens in Python, rather than in the TensorFlow graph.) Let's see how to add one.

You'll need to create a tf.train.SummaryWriter. Add the following before entering the training loop (here):
```
summary_writer = tf.train.SummaryWriter("path/to/logs", sess.graph_def)
```

You'll need to create summary events for the perplexity in each bucket. These values are computed in Python, so you can't use the usual tf.scalar_summary() op. Instead, you'll create a tf.Summary directly by modifying this loop:

perplexity_summary = tf.Summary()

# Run evals on development set and print their perplexity.
for bucket_id in xrange(len(_buckets)):
  encoder_inputs, decoder_inputs, target_weights = model.get_batch(
      dev_set, bucket_id)
  _, eval_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
                               target_weights, bucket_id, True)
  eval_ppx = math.exp(eval_loss) if eval_loss < 300 else float('inf')
  print("  eval: bucket %d perplexity %.2f" % (bucket_id, eval_ppx))

  bucket_value = perplexity_summary.value.add()
  bucket_value.tag = "peplexity_bucket)%d" % bucket_id
  bucket_value.simple_value = eval_ppx

summary_writer.add_summary(perplexity_summary, model.global_step.eval())

You can add other metrics by constructing tf.Summary values yourself and calling summary_writer.add_summary().