Simple weighted embedding network

I have several embeddings matrices, lets say E1 matrix is for Glove and E2 is for Word2vec.

I would like to build a simple sentiment classifier which takes trainable weighted sum of this embeddings.

For example, for the word "dog" I would like to get x["dog"]*E1["dog"] + y["dog"]*E2["dog"] when x and y are learned parameters (notice that I want to learn x and y for each word I know I need to somehow learn the X vector and Y vector and use merge with "add", but I have no idea how to actually do it, help will be highly appreciated.

My network looks as follows:

embd = Sequential()
embd.add(Embedding(topWords + 2, embedding_vecor_length, 
input_length=max_review_len, weights=[embedding_weights],trainable=False))
sent_model = Sequential()
sent_model.add(embd)
sent_model.add(Conv1D(filters, kernel_size, border_mode='valid', 
activation='relu', input_shape=(max_review_len, embedding_vecor_length)))
sent_model.add(Dense(1, activation='sigmoid'))

Solution

It's been a while I used keras. But I would do it this way:

Glove and Word2Vec you will probably load using gensim library. I assume you know how to load these into the keras embedding layer. If not, please let me know.

Here you will set the both embedding layers to non-trainable.

For X and Y you can create two other embedding layers like you would do for E1 and E2 without supplying weights and setting them to trainable, these will be learned by the network during the training process.

You probably could use also a Dense layer instead, but I think using an embedding layer makes it easier.

For multiplication and adding there is an example how to use it, taken from keras documentation:

import keras

input1 = keras.layers.Input(shape=(16,))
x1 = keras.layers.Dense(8, activation='relu')(input1)
input2 = keras.layers.Input(shape=(32,))
x2 = keras.layers.Dense(8, activation='relu')(input2)
added = keras.layers.Add()([x1, x2])  # equivalent to added = keras.layers.add([x1, x2])

out = keras.layers.Dense(4)(added)
model = keras.models.Model(inputs=[input1, input2], outputs=out)

UPDATE:

For example, for the word "dog" I would like to get x["dog"]*E1["dog"] + y["dog"]*E2["dog"] when x and y are learned parameters (notice that I want to learn x and y for each word I know I need to somehow learn the X vector and Y vector and use merge with "add", but I have no idea how to actually do it, help will be highly appreciated.

So, I haven't tested this, also because I don't have the data, there may be some mistakes in the code - but in general it should probably look like this:

#input
sequence_input = Input(shape=(max_review_len,), dtype='int32')

# loading your Glove embedding
layer_E1 = Embedding(w1.shape[0],w1.shape[1],
                            weights=[w1],
                            input_length=max_review_len,
                            trainable=False)
# loading your Word2Vec embedding
layer_E2 = Embedding(w2.shape[0],w2.shape[1],
                            weights=[w2],
                            input_length=max_review_len,
                            trainable=False)
# applying embeddings
embedded_E1 = layer_E1(sequence_input)
embedded_E2 = layer_E2(sequence_input)

# creating learnable embedding layer X and Y
layer_X = Embedding(vocab_size, embedding_vecor_length, input_length=max_review_len)
layer_Y = Embedding(vocab_size, embedding_vecor_length, input_length=max_review_len)

# your weights X and Y
embedded_X = layer_X(sequence_input)
embedded_Y = layer_Y(sequence_input)

# Multiplying X*E1 (x["dog"]*E1["dog"])
layer_E1_X = keras.layers.Multiply()([embedded_E1, embedded_X])
# Multiplying Y*E2 (y["dog"]*E2["dog"])
layer_E2_Y = keras.layers.Multiply()([embedded_E2, embedded_Y])

# merging the results with add
added = keras.layers.Add()([layer_E1_X, layer_E2_Y])

# …
# some other layers
# … 

your_final_output_layer = Dense(1, activation='sigmoid')(previous_layer)

model = Model(sequence_input, your_final_output_layer)
model.compile(…)
model.fit(…)

Edit: I forgot to apply the X and Y embedding, I added it now.

(Please think of this as a rough idea or outline, you may need to change some things)

Important here is that the dimensions match, so both embeddings E1 and E2 should have the same embedding dimension. embedding_vecor_length must be also the same for this to work.

w1 and w2 in the code are assumed to be your glove and word2vec embeddings loaded with gensim.

So, I hope this is roughly what you wanted to do. :)