I would like to have an image embedding to understand which images the network is seing as closer and which one seemed to be very different for him. First, I wanted to use Tensorboard callbacks in Keras, but the documentation are not clear enough for me, and I could not find any useful examples to reproduce it. Hence, to make sure to understand what I am doing, I preferred to make the embedding myself.
To do so, I planed to download the model already trained on my data, removed the last layers (last dropout and dense layer), and predict on the validation images to get the features associated to each images. Then I would simply do a PCA on these features and plot the images according to their first three principal component values.
But I think I misunderstood something, as when I remove the last layers the model predictions are still of the size of the number of classes, but to me it should be of the size of the last layer, which is 128 in my case.
Below is the code for clarification (where I just put the lines which seems useful to answer the question, but do not hesitate to ask for more details):
#model creation
base_model = applications.inception_v3.InceptionV3(include_top=False,
weights='imagenet',
pooling='avg',
input_shape=(img_rows, img_cols, img_channel))
#Adding custom Layers
add_model = Sequential()
add_model.add(Dense(128, activation='relu',input_shape=base_model.output_shape[1:],
kernel_regularizer=regularizers.l2(0.001)))
add_model.add(Dropout(0.60))
add_model.add(Dense(2, activation='sigmoid'))
# creating the final model
model = Model(inputs=base_model.input, outputs=add_model(base_model.output))
Then I trained the model on a dataset having two classes, and loade the model plus its weight to produce features:
model = load_model(os.path.join(ROOT_DIR,'model_1','model_cervigrams_all.h5'))
#remove the last two layers
#remove dense_2
model.layers[-1].pop()
#remove dropout_1
model.layers[-1].pop()
model.summary() # last alyer output shape is : (None, 128), so the removal worked
#predict
model.predict(np.reshape(image,[1,image.shape[0],image.shape[1],3])) #output only two values
Where am I wrong? Would you have any recommendations?
Solution when you add custom layers using Keras functional API:
If you use Keras functional API to add your custom layers, the follwoing solutions works properly:
# base model creation
base_model = applications.inception_v3.InceptionV3(include_top=False,
weights='imagenet',
pooling='avg',
input_shape=(150, 150, 3))
# adding custom Layers
x = Dense(128, activation='relu',input_shape=base_model.output_shape[1:],
kernel_regularizer=regularizers.l2(0.001))(base_model.output)
x = Dropout(0.60)(x)
out = Dense(2, activation='sigmoid')(x)
# creating the final model
model = Model(inputs=base_model.input, outputs=out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
Here is how to extract the activations of custom layers by defining a new model:
# construct a new model to get the activations of custom layers
new_model = Model(model.inputs, [model.layers[-3].output,
model.layers[-2].output,
model.layers[-1].output])
# predict one one random input sample
inp = np.random.rand(1, 150, 150, 3)
output = new_model.predict([inp])
# verify that's what we want
print(output[0].shape) # shape of first dense layer output, prints: (1, 128)
print(output[1].shape) # shape of dropout layer output, prints: (1, 128)
print(output[2].shape) # shape of second dense layer output, prints: (1, 2)
Alternatively, you can define a Keras function:
from keras import backend as K
func = K.function(inputs=model.inputs + [K.learning_phase()],
outputs=[model.layers[-3].output,
model.layers[-2].output,
model.layers[-1].output])
# usage of the defined function:
# the inputs should be a *list* of input arrays
# plus 1 or 0 for the train/test mode
sample_input = np.random.rand(1, 150, 150, 3)
# train mode
output = func([sample_input, 1])
# test mode
ouput = func([sample_input, 0])
Note that you need to use K.learning_phase() since the model contains layers such as BatchNormalization and Dropout that behave differently in test and train modes.
NOTE: The above solutions DO NOT work properly if you use Sequential class to add your custom layers. That's because when using add_model(base_model.output) in construction of model, the whole add_model is stored as one layer of the model. You can verify this by running model.summary() or print(model.layers[-1]). And there is no way to access the output of middle layers of this sequential model. Of course, you can use model.layers[-1].layers[1].output (which is the dropout layer):
new_model = Model(model.inputs, model.layers[-1].layers[1].output)
new_model.predict(...)
however, it would complain that the graph is disconnected since the original input of sequential model has not been fed:
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("dense_7_input:0", shape=(?, 2048), dtype=float32) at layer "dense_7_input". The following previous layers were accessed without issue: []
Actually, I expected that inner layers of sequential model (i.e. model.layers[-1].layer[1:]) have additional inbound and outbound nodes but it seems this is not the case. I don't know whether I am missing something here or it is somehow a bug or impossible-to-do in Keras.
Side note: Actually, using pop() on layers attribute of model object does not work since you need to update some of the internal attributes of the model (although, only for sequential models a built-in pop() method has been implemented).