Is there a way to merge layers in keras with different sample/batch sizes and a question about merged layers

Question

I have looked for an answer my specific question but have not come across anything so posting here. Any help would be much appreciated!

I have two questions about merging layers in keras. I have built a model that concatenates a layer between two models before an output. Because of the nature of my data, the inputs (sample sizes) from both sides will be different, however merging layers would be beneficial as two sources of data will both belong to the same classes in the data and so it would be nice to see how classification is taking place when two models share data before the output.

I would like to know if there is a way of using different sample sizes from both sides as i get an array length error when fitting the model if the data are different sizes.

The way i have counteracted this is to make the left branch of the model (whose input sample size is smaller than the right side data)the same length as the right side by simply duplicating data until the right dimensions are reached, i.e,

train left: 140 samples, train right 180 samples... for train left, i add rows 1:40 to the bottom of the data to make it 180 samples.

Is this a sufficient way to counteract this problem?

My code for the model is as follows:

left_branch_input = Input(shape=(5078,), name='Left_input')
left_branch_output = Dense(64, activation='relu', name="middle_layer")(left_branch_input)
left_branch_output_2=Dense(32, activation="relu", name="second_layer")(left_branch_output)
left_branch_output_3=Dense(100, activation="relu", name="penultimate_layer")(left_branch_output_2)

right_branch_input = Input(shape=(5078,), name='right_input')
right_branch_output = Dense(64, activation='relu', name="middle_layerR")(right_branch_input)
right_branch_output_2=Dense(32, activation="relu", name="second_layerR")(right_branch_output)
right_branch_output_3=Dense(100, activation="relu", name="penultimate_layerR")(right_branch_output_2)

concat = concatenate([left_branch_output_3, right_branch_output_3], name='Concatenate')

final_model_output = Dense(24, activation='sigmoid')(concat)
final_model = Model(inputs=[left_branch_input, right_branch_input], outputs=final_model_output,
                    name='Final_output' )


final_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=["accuracy"])


sum_vector = np.array(y_train.astype(bool) + y_trainSC.astype(bool), dtype=int)

historyMerged=final_model.fit([X_train, X_trainSC], sum_vector,
          epochs=200,
          verbose=1,
          validation_split=0.15,
          callbacks=[EarlyStopping(monitor='val_loss', patience=5)], shuffle=True)

The model compiles and fits and the performance is good, however as a sanity check I would just like to ask if the approach of appending samples (by duplicated[length required] is ok? Is there a way of ignoring the sample size from the different branches?

As this is a binary_crossentropy multi-label problem, the orders of the sample inb oth data sets do not overlap perectly. For example, row is organ 1 age 1 for the left branch and for the right branch the first sample is organ 2 age 2. Does this matter? or is the model working by the class according to the one-hot encoded data as this is what should actually matter right? the class it exists in as oppose to the order... the model.predict function actually gets the data right for both data sets, so presumably, for the data that is the same in left and right branches, the merge layers are using weights from both branches to find similarities and features that improve classification:

Finally, I have a question about the model summary for the merged layer.

Any help and advice on this would be much appreciated! I am pleased with the performance thus far but want to be absolutely sure that the order as well duplicated data is a valid way of approaching this problem.

Thanks!

Answer 1

It seems that your question is how to concatenate tensors with different shapes. This is a simple example you can run in Colab:

import tensorflow as tf
keras = tf.keras

from keras.models import Model
from keras.layers import *

in_a = Input(shape=(10,))
in_b = Input(shape=(20,))
join = Concatenate(axis=1)([in_a, in_b])
h1 = Dense(10)(join)
out = Dense(1, activation='sigmoid')(h1)
model = Model([in_a, in_b], out)
model.compile('adam', 'mse')
model.summary()

The key is the "axis" parameter to Concatenate.

I'm not sure I understand your second question. The features in left and right branches do not need to correlate at all. These could be completely different.