In Keras, on which weight is the dropout applied?

Question

I am currently trying to find a way to retrieve which weights are "ignored" for a given layer (especially when I use the "training" flag to use dropout during the test phase). Is there an easy way to find it or am I obligated to create a custom dropout layer ?

Answer 1

There is no easy way. Keras' tensorflow backend simply calls tf.nn.dropout which works by generating a random matrix of the size of its input and sets values in the input to zero if the corresponding value in the random matrix is less than the threshold.

Here is the key step, located in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/nn_ops.py:

# Sample a uniform distribution on [0.0, 1.0) and select values larger than
# rate.
random_tensor = random_ops.random_uniform(
    noise_shape, seed=seed, dtype=x.dtype)
keep_prob = 1 - rate
ret = (1 / keep_prob) * math_ops.cast(keep_prob >= random_tensor,
                                      x.dtype) * x

You can't retrieve these results directly through keras since the random multiplication is applied immediately and not saved. However you can try to modify the source code to print or save the result of math_ops.cast(keep_prob >= random_tensor,x.dtype) which contains which weights were used in that use of dropout.