Accessing the elements of an Input Layer in a keras model

Question

I am trying to compile and train an RNN model for regression using Keras Tensorflow. I am using the "Functional API" way for the definition of my model.

I need to have 2 different inputs. The first one (input) is my training data which is an array with the shape: (TOTAL_TRAIN_DATA, SEQUENCE_LENGTH, NUM_OF_FEATURES) = (15000,1564,2). To make it more clear, I have 2 features for every frame of 15000 videos. The videos had initially a different number of frames, so all of them have been padded to have SEQUENCE_LENGTH=1564 frames (by repeating the last row). The second input (lengths) is a vector (15000,) that contains the initial length of each video. It's something like this: lengths = [317 215 576 ... 1245 213 654].

What I am trying to do is concatenate the features in the output of a GRU layer and then multiply them with the appropriate masks to keep only the features corresponding to the initial video lengths. To be more precise, the output of the GRU layer has a shape of (batch_size, SEQUENCE_LENGTH, GRU_UNITS) = (50,1564,256). I have defined a Flatten() layer that reshapes the output of the RNN to (50, 1564*256). So in this step, I want to create a mask array with a shape of (50,1564*256). Each row of the array is going to be the mask for the corresponding sample of the batch.

def mask_creator(lengths,number_of_GRU_features=256,max_pad_len=1564):
  
    masks = np.zeros((lengths.shape[0],number_of_GRU_features*max_pad_len))

    for i, length in enumerate(lengths):   
      masks[i,:] = np.concatenate((np.ones([length * number_of_GRU_features, ]),
                                  np.zeros([(max_pad_len - length) * number_of_GRU_features, ])), axis=0)          
    return masks

#tf.compat.v1.enable_eager_execution()
#tf.data.experimental.enable_debug_mode()
#tf.config.run_functions_eagerly(True)

GRU_UNITS = 256
SEQUENCE_LENGTH = 1564
NUM_OF_FEATURES = 2

input = tf.keras.layers.Input(shape=(SEQUENCE_LENGTH,NUM_OF_FEATURES))
lengths = tf.keras.layers.Input(shape=())  
masks = tf.keras.layers.Lambda(mask_creator, name="mask_function")(lengths)
gru = tf.keras.layers.GRU(GRU_UNITS , return_sequences=True)(input)
flat = tf.keras.layers.Flatten()(gru)
multiplied = tf.keras.layers.Multiply()([flat, masks])
outputs = tf.keras.layers.Dense(7, name="pred")(multiplied )

# Compile
model = tf.keras.Model([input, lengths], outputs, name="RNN")
# optimizer = tf.keras.optimizers.Adam(learning_rate=1e-2)


#Compile keras model
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['MeanSquaredError', 'MeanAbsoluteError']),
              #run_eagerly=True)

model.summary()

To create the masks, I have to somehow access the length vector that I am passing as an input argument to my keras model (lengths = tf.keras.layers.Input(shape=())). For that purpose, I thought about defining a Lamda layer (masks=tf.keras.layers.Lambda(mask_creator, name="mask_function")(lengths)) which calls the mask_creator function to create the masks. The lengths variable is supposed to be a Tensor with a shape of (batch_size,)=(50,) if I am not mistaken. However, I cannot, by any means, access the elements of the lengths as I get different types of errors, like that.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-8e31522694ee> in <module>()
      9 input = tf.keras.layers.Input(shape=(SEQUENCE_LENGTH,FEATURES))
     10 lengths = tf.keras.layers.Input(shape=())
---> 11 masks = tf.keras.layers.Lambda(mask_creator, name="mask_function")(lengths)
     12 gru = tf.keras.layers.GRU(GRU_UNITS , return_sequences=True)(input)
     13 flat = tf.keras.layers.Flatten()(gru)

1 frames
<ipython-input-19-9490084e8336> in mask_creator(lengths, number_of_GRU_features, max_pad_len)
      1 def mask_creator(lengths,number_of_GRU_features=256,max_pad_len=1564):
      2 
----> 3     masks = np.zeros((lengths.shape[0],number_of_GRU_features*max_pad_len))
      4 
      5     for i, length in enumerate(lengths):

TypeError: Exception encountered when calling layer "mask_function" (type Lambda).

'NoneType' object cannot be interpreted as an integer

Call arguments received:
  • inputs=tf.Tensor(shape=(None,), dtype=float32)
  • mask=None
  • training=None

Why is that and how could I fix this?

Answer 1

Try using tf operations only:

import tensorflow as tf

@tf.function
def mask_creator(lengths, number_of_GRU_features=256, max_pad_len=1564):

  ones = tf.ragged.range(lengths * number_of_GRU_features)* 0 + 1
  zeros = tf.ragged.range((max_pad_len - lengths) * number_of_GRU_features) * 0
  masks = tf.concat([ones, zeros], axis=1)         
  return masks.to_tensor()

lengths = tf.constant([5, 10])

tf.print(mask_creator(lengths).shape, summarize=-1)