Search code examples
loggingtensorflowtensorboard

No Tensorboard logging in Tensorflow Estimator framework

I am using Estimator to train my net. I want to monitor with Tensorboard. Blogs for Estimator claim things like:

"The training will output information like the global step, loss, 
and accuracy over time on the terminal output. Besides this, the
Experiment and Estimator framework will log certain statistics to 
be visualized by TensorBoard."

https://medium.com/onfido-tech/higher-level-apis-in-tensorflow-67bfb602e6c0

My process does indeed create and event file, but there is nothing in it.

These tags are in checkpoints/1504359209.469093:
audio -
histograms -
images -
scalars -
tensor -

What controls which scalars etc. Estimator writes, and how often it writes them?

Solution

  • Firstly, the documentation is not very clear on this point. To avoid writing a tutorial, here are a few points that you might find helpful, although I should say I am not quite clear on how your code is currently set up.

    • You are not looking to create a FileWriter. Rather, you're just going to add tf.summary.scalar and tf.name_scope annotations to your model_fn.
    • You are not looking to pass anything back from sess.run. That was my first guess, and I can confirm you can keep your Estimator and don't need to restructure.
    • You should be looking at the line that creates your estimator (estimator.Estimator(model_fn=...)). On this line you are specifying your model_dir, which is where you will point tensorboard's logdir parameter. Estimators automatically log to the model_dir, they don't need a FileWriter. You probably already know this since you can see the event output, but I mention it in case it uncovers some inconsistency between what we are doing.
    • You should also look at the RunConfig you pass to your estimator. My contains the key save_summary_steps, and I think that controls how often writes occur.