An example usage for tf.keras.layers.GaussianDropout in TensorFlow2 for deep GRU network

Question

There are not much example of using

tf.keras.layers.GaussianDropout 

in TensorFlow 2, and I am just converting my code from Tensorflow1.15 to Tensorflow 2, and having some difficulty to understand the new way of coding in TF2.

So, can anyone please guide me thru by giving an example of how to use

tf.keras.layers.GaussianDropout 

in tensorflow2, I planning to implement it into a deep GRU network.

I have tried to append it by putting it as a hidden layer, as shown below, but I got an error saying it expects an input tensor, but my case is a GRU cell:

TypeError: Failed to convert elements of <keras.layers.rnn.gru_v1.GRUCell object at 0x7fad6c09f070> to Tensor. Consider casting elements to a supported type. See https://www.tensorflow.org/api_docs/python/tf/dtypes for supported TF dtypes.

 # TF2 approach to define deep GRU
    # Define the GRU Cell
    for i in range(num_gru_layer):
        gru_cell = tf.keras.layers.GRUCell(units=num_neurons)
        gru_cells.append(gru_cell)

    gru_cells = [tf.keras.layers.GaussianDropout(dropout_rate)(cell) for cell in gru_cells]

I thinking if this possible to do something like normal dropout with a dropout wrapper, as shown below where I can just append it as a hidden layer, but I searched there is no such wrapper for gaussian dropout layer. Appreciate for any guidance.

    for i in range(num_gru_layer):
        gru_cell = tf.keras.layers.GRUCell(units=num_neurons)
        dropout_gru = tf.compat.v1.nn.rnn_cell.DropoutWrapper(gru_cell, input_keep_prob = 1.0-dropout_rate)
        gru_cells.append(dropout_gru)

Answer 1

Ok, I found a way to use tf.keras.layers.GaussianDropout, so jotting this down for future reference, for the case where the TensorFlow code is newly converted from TF1.x to TF2.x.

Based on tf.keras.layers.GaussianDropout documentation,

there are 2 arguments for tf.keras.layers.GaussianDropout, namely "rate" and "seed", which user can assigned accordingly.

While under "Call arguments", it specifies the way we use this API, which mentioning "inputs" and "training". As inspired by the way how we connect one layer to another in one of the examples, thus tf.keras.layers.GaussianDropout can be implemented as below:

tf.keras.layers.GaussianDropout(dropout_rate)(input)

 

where 'input' is the tensor to be fed into the deep GRU network, it will be fed into the Gaussian dropout layer, while the output of tf.keras.layers.GaussianDropout will have the same shape as the input (based on documentation).

For clarity, below is how it will look like in real deep GRU network.

# *************** TF2 approach to define deep GRU ***************
# Define the GRU Cell
for i in range(num_gru_layer):
    gru_cell = tf.keras.layers.GRUCell(units=num_neurons)
    gru_cells.append(gru_cell)

# add the last layer with number of output equals to the output of deep GRU network
gru_cell = tf.keras.layers.GRUCell(units=num_outputs)
gru_cells.append(gru_cell)

# Created stacked cell
stacked_gru = tf.keras.layers.StackedRNNCells(gru_cells)

outputs, states = tf.compat.v1.nn.dynamic_rnn(stacked_gru, *input_placeholder*, dtype=tf.float32)


# *************** Gaussian Dropout Implementation ***************
#below is the way I used to implement Gaussian dropout to the input tensor, which will be fed into deep GRU model, in this case, it is 'model_input'
model_input = [tf.keras.layers.GaussianDropout(dropout_rate)(input) for input in model_input.astype(np.float)]                                         

# Convert the list to a tensor
model_input_tensor = tf.convert_to_tensor(model_input)

# Convert the tensor to a numpy array
model_input_numpy_array = sess.run(model_input_tensor) 

#'model_next_output' is the expected tensor from the output of deep GRU network 
output_val, state_val = sess.run([outputs, states], feed_dict={*input_placeholder*:model_input_numpy_array, *output_placeholder*:model_next_output})