BERT embeddings in LSTM model error in fit function

Question

I am novice in TensorFlow

I am traying to use BERT embeddings in LSTM model this is my model function

def bert_tweets_model():
    Bertmodel = TFAutoModel.from_pretrained(model_name,output_hidden_states=True)
    
    input_word_ids = tf.keras.Input(shape=(max_length,), dtype=tf.int32, name="input_ids") 
    input_masks_in = tf.keras.Input(shape=(max_length,), name='masked_token', dtype='int32')
    
    with torch.no_grad():
        last_hidden_states = Bertmodel(input_word_ids, attention_mask=input_masks_in)[0]
        
    x = tf.keras.layers.LSTM(100, dropout=0.1, activation='relu',recurrent_dropout=0.3,return_sequences = True)(last_hidden_states)
    x = tf.keras.layers.LSTM(50, dropout=0.1,activation='relu', recurrent_dropout=0.3,return_sequences = True)(x)
    
    x=tf.keras.layers.Flatten()(x)
    
    output = tf.keras.layers.Dense(units = 2, activation='sigmoid')(x)
    
    model = tf.keras.Model(inputs=[input_word_ids, input_masks_in], outputs = output)
    
    return model

with strategy.scope():
    model = bert_tweets_model()
    adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
    model.compile(loss='binary_crossentropy',optimizer=adam_optimizer,metrics=['accuracy'])
    model.summary()


validation_data=[dev_encoded, y_val]
train2=[input_id, attention_mask]

history = model.fit(
    x=train2, y=y_train, batch_size=batch_size,
    epochs=3,
    validation_data=validation_data,
    verbose=2)

I recieved this error in fit function when I tried to input data

"ValueError: Layer "model_1" expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 512) dtype=int32>]"

also,I received these warning massages I do not know what is means.

WARNING:tensorflow:Layer lstm_2 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU. WARNING:tensorflow:Layer lstm_3 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.

can someone help me, thanks in advance.

Answer 1

Regenerating your error

_input1 = tf.random.uniform((1,100), 0 , 10)
_input2 = tf.random.uniform((1,100), 0 , 10)
model(_input1, _input2)

After running this code I am getting the same error...

Layer "model" expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf.Tensor: shape=(1, 100), ...

#Now, the problem is you have to enclose the inputs in the set or list then you have to pass the inputs to the model like this

model((_input1, _input2))

<tf.Tensor: shape=(1, 2), dtype=float32, numpy=array([[0.5324366, 0.3743334]], dtype=float32)>

Remember: if you are using tf.data.Dataset then encolse it then while making the dataset enclose the dataset within the set like this tf.data.Dataset.from_tensor_slices((words_id, words_mask))

Second Problem as you asked

The warning you are getting because, you should be aware that LSTM doesn't run in CUDA GPU it uses the CPU only therefore it is slow, so TensorFlow is just telling you that LSTM will not run under GPU or parallel computing.