how to concatenate pre trained embedding layer and Input layer

Question

normal_input = Input(shape=(56,))

pretrained_embeddings = Embedding(num_words, 200, input_length=max_length, trainable=False,
                                                            weights=[ft_embedding_matrix])

concatenated = concatenate([normal_input, pretrained_embeddings])

dense = Dense(256, activation='relu')(concatenated)

My idea was to create an input with 256 dimension and pass it to a dense layer.

I got the following error.

ValueError: Layer concatenate_10 was called with an input that isn't a symbolic tensor. Received type: . Full input: [, ]. All inputs to the layer should be tensors.

Please help me how to do this.

Answer 1

You need an input to select which embedding you're using.

Since you're using 150 words, your embeddings will have shape (batch,150,200), which is not possible to concatenate with (batch, 56) in any way. You need to transform something somehow to match the shapes. I suggest you try a Dense layer to transform 56 into 200...

word_input = Input((150,))
normal_input = Input((56,))

embedding = pretrained_embeddings(word_input)
normal = Dense(200)(normal_input)

#you could add some normalization here - read below

normal = Reshape((1,200))(normal)
concatenated = Concatenate(axis=1)([normal, embedding]) 

I also suggest, since embeddings and your inputs are from different natures, that you apply a normalization so they become more similar:

embedding = BatchNormalization(center=False, scale=False)(embedding)
normal = BatchNormalization(center=False, scale=False)(normal)

---

Another possibility (I can't say which is best) is to concatenate in the other dimension, transforming the 56 into 150 instead:

word_input = Input((150,))
normal_input = Input((56,))

embedding = pretrained_embeddings(word_input)
normal = Dense(150)(normal_input)

#you could add some normalization here - read below

normal = Reshape((150,1))(normal)
concatenated = Concatenate(axis=-1)([normal, embedding]) 

I believe this is more suited to recurrent and convolutional networks, you add a new channel instead of adding a new step.

---

You could even try a double concatenation, which sounds cool :D

word_input = Input((150,))
normal_input = Input((56,))

embedding = pretrained_embeddings(word_input)
normal150 = Dense(150)(normal_input)
normal201 = Dense(201)(normal_input)

embedding = BatchNormalization(center=False, scale=False)(embedding)
normal150 = BatchNormalization(center=False, scale=False)(normal150)
normal201 = BatchNormalization(center=False, scale=False)(normal201)


normal150 = Reshape((150,1))(normal150)
normal201 = Reshape((1,201))(normal201)
concatenated = Concatenate(axis=-1)([normal150, embedding]) 
concatenated = Concatenate(axis= 1)([normal201, concatenated])