Input 0 of layer lstm_9 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 2, 4000, 256]

Question

I try to create model with RNN network but I receive : Input 0 of layer lstm_9 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 2, 4000, 256] error.

INPUT

train_data.shape() = (100,2,4000)

train_labels.shape() =(100,)

labels_values = 0 or 1 (two classes)

MODEL

input = Input(shape=(2,4000)) # shape from train_data
embedded = Embedding(2, 256)(input) 
lstm = LSTM(1024, return_sequences=True)(embedded) # ERROR
dense = Dense(2, activation='softmax')(lstm) 

Answer 1

Your whole concept of designing Keras functional models with embedding layers is wrong, unfortunately.

1. When you are using the embedding layer, it expects 2-d data.

Input shape

2D tensor with shape: (batch_size, sequence_length).

Output shape

3D tensor with shape: (batch_size, sequence_length, output_dim).

Ref: https://keras.io/layers/embeddings/

It takes a sequence of IDs or tokens for the vocabulary. This must be an integer array.

Let's say our vocabulary has len 36, we pass it a list of integer arrays in range (0, 36)

[1, 34, 32, 23] is valid [0.2, 0.5] is not valid

2. Usually, we use Embedding to represent the vectors in reduced space, so output_dim is lower than input_dim, but the opposite can be true too based on design.

3. You need to specify the input_length for the input data.

4. If you use return_sequences = True the temporal dimension will be passed to the next dimension, it's not desired in your case.

5. You have labels in the form (0, 1, 0, 1, 0, 0, ...) and not in one-hot-encoded form, so don't use softmax but sigmoid with 1 unit in the last dense.

This is the somewhat corrected network.

from tensorflow.keras.layers import *
from tensorflow.keras.models import *
import numpy as np
train_data = np.random.randint(0,3, (100, 4000))
y_labels = np.random.randint(0,2, (100,))

input_ = Input(shape=(4000)) # shape from train_data
embedded = Embedding(36, 256, input_length = 4000)(input_) 
lstm = LSTM(256, return_sequences=False)(embedded) # --> ERROR
dense = Dense(1, activation='softmax')(lstm) 

model = Model(input_, dense)
model.summary()

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_6 (InputLayer)         [(None, 4000)]            0         
_________________________________________________________________
embedding_5 (Embedding)      (None, 4000, 256)         9216      
_________________________________________________________________
lstm_5 (LSTM)                (None, 256)               525312    
_________________________________________________________________
dense (Dense)                (None, 1)                 257       
=================================================================
Total params: 534,785
Trainable params: 534,785
Non-trainable params: 0