I have a 3D tensor called X, of shape say [2,20,300] and I would like to apply dropout to only the third dimension. However, I want the dropped elements to be the same for the 20 instances (second dimension) but not necessarily for first dimension.
What is the behaviour of the following:
tf.nn.dropout(X[0], keep_prob=p)
Would it only act on the dimension that I want? If so, then for multiple first dimensions, I could loop over them and apply the above line.
See the documentation of tf.nn.dropout:
By default, each element is kept or dropped independently. If noise_shape is specified, it must be broadcastable to the shape of x, and only dimensions with noise_shape[i] == shape(x)[i] will make independent decisions
So it is as simple as:
import tensorflow as tf
import numpy as np
data = np.arange(300).reshape((1, 1, 300))
data = np.tile(data, (2, 20, 1))
data_op = tf.convert_to_tensor(data.astype(np.float32))
data_op = tf.nn.dropout(data_op, 0.5, noise_shape=[2, 1, 300])
with tf.Session() as sess:
data = sess.run(data_op)
for b in range(2):
for c in range(20):
assert np.allclose(data[0, 0, :], data[0, c, :])
assert np.allclose(data[1, 0, :], data[1, c, :])
print((data[0, 0, :] - data[1, 0, :]).sum())
# output something != 0 with high probability#