Search code examples
tensorflowscientific-notationnumerical-stability

Tensorflow: Add small number before division for numerical stability


In order to prevent divisions by zero in TensorFlow, I want to add a tiny number to my dividend. A quick search did not yield any results. In particular, I am interested in using the scientific notation, e.g.

a = b/(c+1e-05)

How can this be achieved?


Solution

  • Assuming a, b and c are tensors. The formula you have written will work as expected. 1e-5 will be broadcasted and added on the tensor c. Tensorflow automatically typecasts the 1e-5 to tf.constant(1e-5).

    Tensorflow however has some limitations with non-scalar broadcasts. Take a look at my other answer.