Search code examples
tensorflowtensorboard

Tensorboard Unble to get first event timestamp for run


I am trying to visualize a training session I trained in a remote server. I used scp to copy the file in my local iMac. I tried to visualize the data by running tensorboard. It runs the tensorboard site but I can't get the visualization. Every chart has a single dot at zero. I get this warning on the terminal.

WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_200/train
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_200/train
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_200/val
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_50/train
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_50/val

Any idea what is going on?


Solution

  • I ran into a similar problem. There are 2 solutions to it -

    • Delete all past tfevents files from the directory and keep the latest one (temporary solution).
    • Create a new directory for building your logs (permanent solution).

    In given below picture, first I build logs in directory 2 which resulted in same error/warnings. Later I changed the directory to 3 and build logs there and tensorboard ran successfully. Problem and its solution