I was debugging my program and I've realized that I my loss outputted NaN. These NaN values comes from the fact that I'm computing tf.log(1 + tf.exp(X))
where X is a 2d tensor. Indeed, When a value of X is large enough then tf.exp() returns +Inf and so tf.log(1 + exp(X)) will return +Inf. I was wondering if there exists a neat trick to avoid underflows and overflows in this case.
I have tried:
def log1exp(x):
maxi = tf.reduce_max(x)
return maxi + tf.log(tf.exp(x - maxi) + tf.exp(-maxi))
but it doesn't handle underflows in this case...
Also I've glanced at tf.reduce_logsumexp but it necessarily reduce the tensor along an axis... while I want to keep the same shape!
Finally I know that tf.log(1 + exp(X)) is almost equal to X for large values of X but I think that designing a function that will output X when X > threshold and log(1+exp(X)) otherwise is not very neat.
Thank you
This function is already implemented in tensorflow under the name tf.math.softplus, and takes care of overflows and underflows.