Concatenation of ResNet (2D images) with fully-connected network (1D input)

Question

I'm using a pre-built ResNet in Keras (TensorFlow 2) in the following way:

from tensorflow.keras.applications.resnet50 import ResNet50
base_model = ResNet50(weights=None, include_top=False, input_shape=(39,39,3))
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dropout(0.5)(x)
output_tensor = Dense(self.num_classes, activation='softmax')(x)
cnn_model = Model(inputs=base_model.input, outputs=output_tensor)
opt = Adam(lr=0.001)
cnn_model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy', tf.keras.metrics.AUC()])

The inputs to the model (base_model.input) are 39 x 39 x 3 images. In addition, I now also want to provide a 20 dimensional vector with additional information to the model (i.e. 20 x 1). I can do this in two different ways:

1. Appending the 20 dimensional vector after the GlobalAveragePooling2D step.
2. Creating an additional Fully-connected network for the 20-dimensional vector and concatenating the output of this fully-connected network to the above ResNet after the GlobalAveragePooling2D step. Here ideally both networks are trained at the same time but I don't know if that is possible.

Can I adjust my models for both options or does that not work?

Answer 1

This should do it, Comment out the Dense Layer to concatenate them after the global average pooling.

from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.layers import GlobalAveragePooling2D, Dropout, Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
import tensorflow as tf


base_model = ResNet50(weights=None, include_top=False, input_shape=(39, 39, 3))
x1 = base_model.output
x1 = GlobalAveragePooling2D()(x1)
x1 = Dropout(0.5)(x1)

input_2 = tf.keras.layers.Input(shape=(20, 1))
x2 = tf.keras.layers.Flatten()(input_2)
# comment this if needed.
x2 = tf.keras.layers.Dense(16, activation='relu')(x2)

x = tf.keras.layers.Concatenate()([x1, x2])

output_tensor = Dense(self.num_classes, activation='softmax')(x)
cnn_model = Model(inputs=[base_model.input, input_2], outputs=output_tensor)
opt = Adam(lr=0.001)
cnn_model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy', tf.keras.metrics.AUC()])
print(cnn_model.summary())