Tensorboard bump in accuracy score

Question

I have been training my Inception ResNet V2 using Tensorflow, and logging the accuracy/loss etc. via TensorBoard.

Now when I resumed training today, almost instantly (in a few global steps) my accuracy bumped from 86% to 97%, when resuming the checkpoint I stopped at earlier.

When looking at the loss plot, it seems to be gradually reducing still, but accuracy had this huge bump. Is there an obvious/logical explanation for this ? I resumed training at epoch 21 (stopped at 20), with 1339 global steps per epoch.

Answer 1

That is because you are using a streaming accuracy, which accumulates all statistics since the beginning of time -- well, of training time.

Until you stopped training, the streaming accuracy was returning the accuracy averaged since the beginning.

When you resumed training, the streaming accuracy op has been reset and now outputs the mean accuracy since you resumed training. It is much higher because it does not average over the earlier, lower values of accuracy, when your model was weak.

I actually posted something yesterday on how to reset streaming metrics from time to time to avoid this continuous accumulation.