Keras LSTM autoencoder with embedding layer

Question

I am trying to build a text LSTM autoencoder in Keras. I want to use an embedding layer but I'am not sure how to implement this. The code looks like this.

inputs = Input(shape=(timesteps, input_dim))
embedding_layer = Embedding(numfeats + 1,
                            EMBEDDING_DIM,
                            weights=[data_gen.get_embedding_matrix()],
                            input_length=maxlen,
                            trainable=False)

embedded_sequence = embedding_layer(inputs)
encoded = LSTM(num_units)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(???, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)

sequence_autoencoder.compile(loss='binary_crossentropy', optimizer='adam')

I am not sure how to decode the output into the target sequence (which is obviously the input sequence).

Answer 1

There is no way to implement an inversed embedding layer in the decoder, because the embedding layer is not differentiable. There are probably other ways to work around:

1. construct the autoencoder from the output of the embedding layer, to a layer with a similar dimension. Then use the nearest neighbor or other algorithms to generate the word sequence from there.

2. construct an asymmetric autoencoder, using the time distributed layer and dense layers to reduce the dimension of LSTM output.

Hopefully this helps.