Search code examples
deep-learningpytorchtensorboardseq2seqopennmt

Output hidden state in OpenNMT-py

I just have a short question regarding the pytorch version of OpenNMT. There does not seem to be an option to return the hidden state of encoder and decoder in the options. Am I missing a flag or is this not an option in OpenNMT-py?

Solution

  • What do you mean by the encoder and decoder does not return hidden state?

    If you see the RNNEncoder, it returns encoder_final, memory_bank, lengths where the memory_bank represents the hidden state which is of shape seq_len x batch_size x hidden_size. And the encoder_final is in general used by the decoder in a sequence-to-sequence model.

    Now, let's see the RNNDecoder. As we see, the forward() method returns a FlaotTensor and a dictionary of FlaotTensors.

    (FloatTensor, dict[str, FloatTensor]):
* dec_outs: output from the decoder (after attn)
  ``(tgt_len, batch, hidden)``.
* attns: distribution over src at each tgt
  ``(tgt_len, batch, src_len)``.

    Usually, we use the dec_outs in sequence-to-sequence tasks. For example, in natural language generation tasks, we feed the dec_outs to a softmax layer to predict tokens.

    There are several other types of encoders/decoders which you can explore from the following two links.