Invalid type tf.complex64 for tf.train.GradientDescentOptimizer()

Question

I am working with complex neural networks. I have created a network that works correctly using:

[...]
gradients = tf.gradients(mse, [weights])[0]
training_op = tf.assign(weights, weights - learning_rate * gradients)
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    sess.run(training_op)

Now when I try to use:

optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)

I get, for line training_op = optimizer.minimize(mse) the following:

ValueError: Invalid type tf.complex64 for weights:0, expected: [tf.float32, tf.float64, tf.float16, tf.bfloat16].

Is this truly not supported for complex? Or am I doing something wrong? I tried the same with real-valued net and it worked correctly so I believe I have the structure correct.

---

New insight:

According to this. Minimize is divided into two parts:

1. compute_gradients
2. apply_gradients

If we test them separately, the error occurs on the compute_gradients method.

So running tf.gradients work but running optimizer.compute_gradients doesn't? This is getting weirder. Anyone knows the reason?

Answer 1

Providing the answer here from the comment section for the benefit of the community.

Since optimizers do not currently support complex types (even if complex gradients are supported, as you mention), you may just use separate variables for real and imaginary parts. You may also consider writing your own optimizer and override the valid_dtypes method like the below example.

def complex_mul_real( c, r ):
    return tf.complex(tf.real(c)*r, tf.imag(c)*r)  

Further, detailed discussion on this topic could be found on this Github issue.