Search code examples
tensorflowderivativegradienttape

Gradient computations in Tensorflow 2.0

Here is my example from Tensorflow 2.0:

import tensorflow as tf

w = tf.Variable([[1.0]])
with tf.GradientTape() as tape_1:
    loss_1 = w * w


with tf.GradientTape() as tape_2:
    loss_2 = w * w * w

grad_1 = tape_1.gradient(loss_1, w)
grad_2 = tape_2.gradient(loss_2, w)
print(grad_1)
print(grad_2)

it returns: 

tf.Tensor([[2.]], shape=(1, 1), dtype=float32)
tf.Tensor([[3.]], shape=(1, 1), dtype=float32)

The above are correct coefficients, but grad_2 should also indicate that we have 3w^2. How can I retrieve w^2 part?

Solution

  • The gradient results do not mean that. If you take your functions, f(w) = w2 and g(w) = w3, their respective derivative functions with respect to w would be f'(w) = 2w and g'(w) = 3w2. What the gradient function gives you is the value of these functions for the current value of w. So, since w is initialized to 1, it gives you f'(1) = 2 and g'(1) = 3. TensorFlow can, in a way, compute the symbolic derivative function, but as a sequence of TensorFlow operations, so it is not straightforward to extract a nice mathematical expression from it. And with eager execution, as you are using, it is not even available, the operations are executed as necessary and intermediates are discarded.