cosine similarity for attention decoder in nmt

Question

I am implementing a neural machine translation model, and for the decoder part (with attention mechanism) I would like to calculate the cosine similarity for finding the scores. Here is the function:

score(a,b) = / ||a|| ||b||

In my case:

        a = htilde_t (N, H)

        b = h (S, N, H)

        the output should be (S, N)

My confusion is about their dimensions and I don't know how to solve that in pytorch.

Answer 1

See here: https://pytorch.org/docs/master/nn.html?highlight=cosine#torch.nn.CosineSimilarity

cos = nn.CosineSimilarity(dim=2, eps=1e-6)
output = cos(a.unsqueeze(0),b)

you need to unsqueeze to add a ghost dimension to have both input of same dim:

    Input1: (∗1,D,∗2) where D is at position dim

    Input2: (∗1,D,∗2) , same shape as the Input1

    Output: (∗1,∗2)