Search code examples
tensorflowtensorboard

Unable to use Tensorboard in Distributed Tensorflow

I am starting to play with Distributed Tensorflow. I am able to distribute the training in different servers succesfully but I cannot see any summary in tensorboard.

Does anyone know if there are any limitation or caveat with this?

Thanks

Solution

  • There is a caveat, which is that TensorBoard doesn't support replicated summary writers. Other than that, it will work.

    Choose one TensorFlow worker to be the summary writer, and have it write summaries to disk. Then, launch TensorBoard pointing to the summary files that you've saved (the simplest would be to launch TensorBoard on the same server that the summary worker is on - alternatively, you could copy the files off that server onto your machine, etc).

    Note, in the special case where you are using Google Cloud, TensorBoard can read directly from gcs paths.