monte-carlo recurrent dropout with lstm

Question

I want to implement mc-dropout for lstm layers as suggested by Gal using recurrent dropout. this requires using dropout in the test time, in regular dropout (masking output activations) I use the functional API with the following layer: intermediate = Dropout(dropout_prob)(inputs, training=True) but I'm not sure how to use that in lieu of the recurrent dropout of LSTM. I would be glad if someone could help me with this matter.

Answer 1

you can use MC dropout with recurrent_dropout in the same way. all you need is to specify training=True in functional API

inp = Input(shape=(10, 1))
x = LSTM(1, recurrent_dropout=0.3)(inp, training=False)
m = Model(inp,x)

X = np.random.uniform(0,1, (1,10,1))

output = []
for i in range(0,100):
    output.append(m.predict(X))

with training=False the result in the output is always the same, this means MC dropout is not applicate

inp = Input(shape=(10, 1))
x = LSTM(1, recurrent_dropout=0.3)(inp, training=True)
m = Model(inp,x)

X = np.random.uniform(0,1, (1,10,1))

output = []
for i in range(0,100):
    output.append(m.predict(X))

in the second example, we set training=True and we can see that output elements are always different.

in conclusion, recurrent_dropout can be activated like a simple dropout layer