ValueError: Input 0 is incompatible with layer layer_1: expected ndim=3, found ndim=2

Question

I am trying to build text-summarizer using word Embeddings and encoder-decoder architecture. This is my first shot at Keras and I am not able to understand why layer_1 requires ndim=3. I am not able to figure this out. Below is my code:

vocab_size = 16828
n_embeddings = 200

def model_builder(embeds):
        model = keras.Sequential()
        model.add(Embedding(weights=[embeds], name="embedding_1", input_dim=vocab_size,
                            output_dim=n_embeddings))
        for i in range(3):
            lstm = LSTM(rnn_size, name="layer_%s" %(i))
            model.add(lstm)
            model.add(Dropout(prob, name="drop_%s" %(i)))
        model.add(Dense())
        model.add(Activation('softmax', name="activation"))
        return model

rnn_size = 200
prob = 0.5
encoder = model_builder(embedding)
encoder.compile(loss='categorical_crossentropy', optimizer='rmsprop')
enocder.save_weights('embeddings.pkl', overwrite=True)

I will really appreciate your help. Let me know if you guys need any other information. Thank you in advance.

P.S. Keras backend is Tensorflow.

Answer 1

Your problem lies in these lines:

for i in range(3):
        lstm = LSTM(rnn_size, name="layer_%s" %(i))
        model.add(lstm)
        model.add(Dropout(prob, name="drop_%s" %(i)))

LSTM by default returns only the last step of its predictions - so data is losing its sequential nature. In your example - after the first iteration LSTM outputs a vector instead of a sequence of errors - and that's why an error is raised.

In order to fix that try:

for i in range(2):
        lstm = LSTM(rnn_size, name="layer_%s" %(i), return_sequences=True)
        model.add(lstm)
        model.add(Dropout(prob, name="drop_%s" %(i)))
lstm = LSTM(rnn_size, name="layer_%s" %(i), return_sequences=False)
model.add(lstm)

Another thing which I've noticed is that you are using Dense in an incorrect manner. You should provide the number of output neurons:

model.add(Dense(nb_of_output_neurons))

Cheers.