I use tensorboard on pytorch for my experiments as usual: I log necessary data while training and then running tensorboard --logdir my_dir to see them. my_dir is containing folders with all my experiments so I can see all of them on tensorboard screen.
But recently it stopped displaying any data except one experiment (despite the fact that it had previously shown the same experiments!).
And if I specify --logdir for one random experiment, tensorboard shows only first steps of logs..
I searched for solution of this problem via internet and couldn't find any useful information. I want to understand can it be tensorboard fault or this problem is on my side.
It turned out that tensorboard needed time to load the logs. I also noticed that the weight of one log is almost 10 gigabytes! I wonder what exactly gives so much weight to the file... I will try to optimize the amount of memory during future logging