Search code examples
tensorflowtensorboard

How can I write to Tensorboard even when I interrupt training?


Im using Tensorboard callbacks with update_freq=10000, so I expect metrics written to Tensorboard every 10000 batches.

Since training takes a long time, I occassionally interrupt (like after 200k batches) with KeyboardInterrupt, but when I then look at Tensorboard, no metrics have been written, nothing is loaded, nothing can be seen.

What am I missing or what am I doing wrong? How can I write to Tensorboard in a way that metrics are written even if I interrupt training and not have to wait till the end of the training for Tensorboard to show anything?


Solution

  • Very simple answer: Tensorboard just takes a long time to load and display all the logs, especially after long training.

    Waiting some decent time and then refresh will show all the logs, even of the ones that were interrupted during training.