Using categorical_crossentropy for a sequence of images

Question

I have a model that accepts a sequence of images as an input (None, n_step, 128, 128) (instead of a single image) where n_step is a fixed number 10. And I am using categorical_crossentropy for classification of four class problem. But I have an error as shown below

ValueError: A target array with shape (1342, 10, 4) was passed for an output of shape (None, 1, 4) while using as loss `categorical_crossentropy`. This loss expects targets to have the same shape as the output.

I understand from the error, that it is expecting a single image at a time. Is there any way where I can use this loss for a sequence of images?

Output form the model would also be a set of 10 labels

Edit:

n_steps = 10
feature_count = 4
def create_model():

    trajectory_input = Input(shape=(n_steps, feature_count), name='trajectory_input')
    image_input  = Input(shape=(128, 128, n_steps), name='image_input')

    x_aware = (Conv2D(32, kernel_size=(3, 3), activation='relu'))(image_input)
    x_aware = (BatchNormalization())(x_aware)
    x_aware = (MaxPooling2D(pool_size=(2, 2)))(x_aware)
    x_aware = (Dropout(0.25))(x_aware)

    x_aware = (Conv2D(64, kernel_size=(3, 3), activation='relu'))(x_aware)
    x_aware = (BatchNormalization())(x_aware)
    x_aware = (MaxPooling2D(pool_size=(2, 2)))(x_aware)
    x_aware = (Dropout(0.25))(x_aware)

    x_aware = (Conv2D(64, kernel_size=(3, 3), activation='relu'))(x_aware)
    x_aware = (BatchNormalization())(x_aware)
    x_aware = (MaxPooling2D(pool_size=(2, 2)))(x_aware)
    x_aware = (Dropout(0.25))(x_aware)

    x_aware = (Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x_aware)
    x_aware = (BatchNormalization())(x_aware)
    x_aware = (Dropout(0.25))(x_aware)
    x_aware = Reshape((1, 12544))(x_aware)

    x = (Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(trajectory_input)
    x = Reshape((1, 32*n_steps))(x)

    x = concatenate([x, x_aware])
    x = (Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)
    x = (Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)

    x_reg = (Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)
    x_class = (Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)

    x_reg = Reshape((2, 4))(x_reg)

    output_regression = (Dense(2, name='main_output'))(x_reg)
    output_class = (Dense(4, name='classification_output', activation='softmax'))(x_class)

    adam = Adam(lr=learning_rate)
    model = Model(inputs=[trajectory_input, image_input], outputs=[output_regression, output_class])
    model.compile(optimizer=adam, loss={'main_output': 'mse','classification_output': 'categorical_crossentropy'}, metrics={'main_output': [euc_dist_1, euc_dist_2], 'classification_output': 'accuracy'})
    model.summary()
    return model

Inputs are the images and their related information for regression task and outputs are labels and the next predictable value.

Answer 1

Problem was with the input given to the Dense and Reshape layer of classification task. Since the input was in (10,128,128), number of units should be 10*num_class. So changing this

x_reg = (Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)
x_class = (Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)

x_reg = Reshape((2, 4))(x_reg)

output_regression = (Dense(2, name='main_output'))(x_reg)
output_class = (Dense(4, name='classification_output', activation='softmax'))(x_class)

to this resolved the issue

x_reg = (Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)
x_class = (Dense(40, activation='relu', kernel_regularizer=regularizers.l2(0.001)))(x)

x_reg = Reshape((2, 4))(x_reg)
x_class = Reshape((10,4))(x_class)

output_regression = (Dense(2, name='main_output'))(x_reg)
output_class = (Dense(4, name='classification_output', activation='softmax'))(x_class)