I'm building a model using keras in order to learn word embeddings using a skipgram with negative sampling. My input is pair of words: (context_word, target_word) and of course the label 1 for positives and 0 for negative couples. What I need to do is add bias to the model. The bias should be only the bias of the target word for each input and not for both words.
Until now I have the code:
input_u = Input((1,))
input_v = Input((1,))
item_embedding = Embedding(input_dim = items_size,
output_dim = embed_dim,
name = 'item_embedding')
bias_embedding = Embedding(input_dim = items_size,
output_dim = 1,
embeddings_initializer = 'zeros',
name = 'bias_embedding')
u = item_embedding(input_u)
v = item_embedding(input_v)
b_v = bias_embedding(input_v)
dot_p_layer = merge.dot([u, v], axes = 1)
with_bias = merge.add([dot_p_layer, b_v])
flattenned = Flatten()(with_bias)
output_layer = Dense(1,
activation = 'sigmoid',
use_bias = False)(flattenned)
print (output_layer.shape)
model = Model(inputs=[input_u, input_v], outputs=output_layer)
model.compile(loss='binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
However, I can't seem to get it working. The code is running but I'm getting higher loss and lower accuracy than the model without bias. So I'm thinknig I'm doing something wrong. Plus, when I check the sizes I still get the size of my embedding dimension and not embedding dimension + 1
I thought about using another Dense layer (not even sure if it's logical or right to do) in order to add the bias after the dot product but I couldn't really get it working either.
I really would like some help adding bias into the model.
If you want dimension + 1 you're looking for concatenate, not for add.
I don't know the dimension after dot (dot is weird behavior, lol), but if it's 3D (batch, embedding, embedding), you will need to flatten before the concatenation.