Keras: Dense vs. Embedding - ValueError: Input 0 is incompatible with layer repeat_vector_9: expected ndim=2, found ndim=3

Question

I have the following network which works fine:

left = Sequential()
left.add(Dense(EMBED_DIM,input_shape=(ENCODE_DIM,)))
left.add(RepeatVector(look_back))

However, I need to replace the Dense layer with the Embedding layer:

left = Sequential()
left.add(Embedding(ENCODE_DIM, EMBED_DIM, input_length=1))
left.add(RepeatVector(look_back))

Then I got the following error when I use Embedding layer:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-119-5a5f11c97e39> in <module>()
     29 left.add(Embedding(ENCODE_DIM, EMBED_DIM, input_length=1))
---> 30 left.add(RepeatVector(look_back))
     31 
     32 leftOutput = left.output

/usr/local/lib/python3.4/dist-packages/keras/models.py in add(self, layer)
    467                           output_shapes=[self.outputs[0]._keras_shape])
    468         else:
--> 469             output_tensor = layer(self.outputs[0])
    470             if isinstance(output_tensor, list):
    471                 raise TypeError('All layers in a Sequential model '

/usr/local/lib/python3.4/dist-packages/keras/engine/topology.py in __call__(self, inputs, **kwargs)
    550                 # Raise exceptions in case the input is not compatible
    551                 # with the input_spec specified in the layer constructor.
--> 552                 self.assert_input_compatibility(inputs)
    553 
    554                 # Collect input shapes to build layer.

/usr/local/lib/python3.4/dist-packages/keras/engine/topology.py in assert_input_compatibility(self, inputs)
    449                                      self.name + ': expected ndim=' +
    450                                      str(spec.ndim) + ', found ndim=' +
--> 451                                      str(K.ndim(x)))
    452             if spec.max_ndim is not None:
    453                 ndim = K.ndim(x)

ValueError: Input 0 is incompatible with layer repeat_vector_9: expected ndim=2, found ndim=3

What additional changes do I need when replacing the Dense layer with an Embedding layer? Thanks!

Answer 1

The output shape of the Dense layer is (None, EMBED_DIM). However, the output shape of the Embedding layer is (None, input_length, EMBED_DIM). With input_length=1, it'll be (None, 1, EMBED_DIM). You can add a Flatten layer after the Embedding layer to remove axis 1.

You can print out the output shape to debug your model. For example,

EMBED_DIM = 128
left = Sequential()
left.add(Dense(EMBED_DIM, input_shape=(ENCODE_DIM,)))
print(left.output_shape)
(None, 128)

left = Sequential()
left.add(Embedding(ENCODE_DIM, EMBED_DIM, input_length=1))
print(left.output_shape)
(None, 1, 128)

left.add(Flatten())
print(left.output_shape)
(None, 128)