How to join an encoder and a decoder

Question

I have built the following encoder-decoder architecture, and the encoder and decoder both work fine separately:

from tensorflow.keras.layers import LSTM, Input, Reshape, Lambda
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K

WORD_TO_INDEX = {"foo": 0, "bar": 1}

MAX_QUERY_WORD_COUNT = 10
QUERY_ENCODING_SIZE = 15

# ENCODER
query_encoder_input = Input(shape=(None, len(WORD_TO_INDEX)), name="query_encoder_input")
query_encoder_output = LSTM(QUERY_ENCODING_SIZE, name="query_encoder_lstm")(query_encoder_input)
query_encoder = Model(inputs=query_encoder_input, outputs=query_encoder_output)

# DECODER
query_decoder_input = Input(shape=(QUERY_ENCODING_SIZE,), name="query_decoder_input")
query_decoder_reshape = Reshape((1, QUERY_ENCODING_SIZE), name="query_decoder_reshape")(query_decoder_input)
query_decoder_lstm = LSTM(QUERY_ENCODING_SIZE, name="query_decoder_lstm", return_sequences=True, return_state=True)
recurrent_input, state_h, state_c = query_decoder_lstm(query_decoder_reshape)
states = [state_h, state_c]
query_decoder_outputs = []
for _ in range(MAX_QUERY_WORD_COUNT):
    recurrent_input, state_h, state_c = query_decoder_lstm(recurrent_input, initial_state=states)
    query_decoder_outputs.append(recurrent_input)
    states = [state_h, state_c]
query_decoder_output = Lambda(lambda x: K.concatenate(x, axis=1), name="query_decoder_concat")(query_decoder_outputs)
query_decoder = Model(inputs=query_decoder_input, outputs=query_decoder_output)

But when I try to join them together to create an autoencoder, I get an odd error and I don't know why.

# AUTOENCODER
# apply the reshape layer to the output of the encoder
query_autoencoder_output = query_decoder.layers[1](query_encoder_output)
# rebuild the autoencoder by applying each layer of the decoder to the output of the encoder
for decoder_layer in query_decoder.layers[2:]:
    # this fails and I don't know why
    query_autoencoder_output = decoder_layer(query_autoencoder_output)
# the code never gets here
query_autoencoder = Model(inputs=query_encoder_input, outputs=query_autoencoder_output)

This throws the error:

ValueError: Shape must be rank 3 but is rank 2 for '{{node query_decoder_concat/concat_1}} = ConcatV2[N=3, T=DT_FLOAT, Tidx=DT_INT32](query_decoder_lstm/PartitionedCall_11:1, query_decoder_lstm/PartitionedCall_11:2, query_decoder_lstm/PartitionedCall_11:3, query_decoder_concat/concat_1/axis)' with input shapes: [?,1,15], [?,15], [?,15], [].

This is the template I used for my decoder. (See the "What if I don't want to use teacher forcing for training?" section.)

I relied on these StackOverflow questions (especially the last one) to figure out how to combine the models together.

What does this error mean and how can I fix it?

Answer 1

You can treat a model as a layer * essentially *. With an autoencoder, it'll be as straightforward as something like this:

autoencoder = Sequential([encoder, decoder])