How should Tensorboard's fraction of zero values be interpreted?
I am running a cloud ML engine job and my tensorboard plots are showing the fraction of zero values for my hidden layers steadily increasing towards 1 as the number of steps increases. How should this plot be interpreted? I believe it is a good thing as more zero values would suggest that the model is getting more "certain" about the predictions that it is making.
It generally means your regularization technique and/or activation function is forcing activations to zero. You haven't shared details of your model, but this is common when using dropout, especially with relu activation functions.
Models with lots of zero activations tend to generalize better and therefore give better accuracy.
If you want more details, here's a JMLR paper on dropout.
I do have to note that having activations go to zero is sometimes bad, at least for ReLU activation functions. Basically, they can irreverisbly "die". So if you are seeing poor model quality beware. More information here.