I want to measure the entropy of an MNIST image. However most of the intensity values are 0, and tf.log returns inf. I know that I can simply calculate the entropy:
entropy = tf.reduce_mean(-tf.reduce_sum(X * tf.log(X), axis=1))
where X is a placeholder the MNIST image batch, defined as X = tf.placeholder("float", [None, 784]). However, the result is nan (many pixel values are 0, so tf.log will give inf). Is there a way to work around this, or another method to calculate the entropy of one image?
Entropy depends on the probability of each intensity value, not on the value itself. A value with a probability of 0 doesn't enter into the calculations.
Write code to compute the proportion of each intensity value that appears in the photo. Those form your X vector.
I think you misunderstand the concept of entropy. You can look up that part, as well as the calculation.
In general, it's the average amount of surprise you get with any one transmitted sampling of the distribution. Another way to interpret it is to design the best binary coding, given the frequency distribution (Huffman code). The average number of bits transmitted is the entropy.
Now, back to your case. To simplify the numbers, let's consider a 10x10 image, with only four intensity levels, 0-3. 70 of the values are 0, 20 are 1, with six 2's and four 3's filling out the set. In this case, your X array above has only four values: [0.70, 0.20, 0.06, 0.04]. There are no zero or negative values to foul your log calculations.
To do this by hand, let's consider the Huffman coding. The simple-minded case is to encode each value with its 2-bit binary equivalent: 00, 01, 10, 11. This gives us a flat rate of 2 bits per value transmitted.
However, we can do better. If we encode them as
0: 0
1: 10
2: 110
3: 111
Then our average number of bits transmitted is:
0.70*1 + 0.20*2 + 0.06*3 + 0.04*3
= 0.70 + 0.40 + 0.18 + 0.12
= 1.40 bits
... so 1.40 is the entropy for this image. Note that we didn't take the log of any of the values, merely of the frequencies.