I am working a transfer learning problem. When I create a new model from just the Mobilenet, I set a dropout.
base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(200,200,3), dropout=.15)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(10, activation='softmax')(x)
I save models as I train using model_checkpoint_callback. As I train I find where overfitting is happening and adjust the amount of frozen layers and the learning rate. Can I also adjust dropout when I save a loaded model again?
I saw this answer but there are no actual dropout layers in Mobilenet, so this
for layer in model.layers:
if hasattr(layer, 'rate'):
print(layer.name)
layer.rate = 0.5
doesn't do anything.
In the past, you had to clone the model for the new dropout to take. I haven't tried it recently.
# This code allows you to change the dropout
# Load model from .json
model.load_weights(filenameToModelWeights) # Load weights
model.layers[-2].rate = 0.04 # layer[-2] is my dropout layer, rate is dropout attribute
model = keras.models.clone(model) # If I do not clone, the new rate is never used. Weights are re-init now.
model.load_weights(filenameToModelWeights) # Load weights
model.predict(x)
credit to
If the model doesn't have dropout layers to even begin with, as with Keras's pretrained mobilenet, you'll have to add them with methods. Here's one way you could do it.
For adding in a single layer
def insert_single_layer_in_keras(model, layer_name, new_layer):
layers = [l for l in model.layers]
x = layers[0].output
for i in range(1, len(layers)):
x = layers[i](x)
# add layer afterward
if layers[i].name == layer_name:
x = new_layer(x)
new_model = Model(inputs=layers[0].input, outputs=x)
return new_model
For systematically adding a layer
def insert_layers_in_model(model, layer_common_name, new_layer):
import re
layers = [l for l in model.layers]
x = layers[0].output
layer_config = new_layer.get_config()
base_name = layer_config['name']
layer_class = type(dropout_layer)
for i in range(1, len(layers)):
x = layers[i](x)
match = re.match(".+" + layer_common_name + "+", layers[i].name)
# add layer afterward
if match:
layer_config['name'] = base_name + "_" + str(i) # no duplicate names, could be done different
layer_copy = layer_class.from_config(layer_config)
x = layer_copy(x)
new_model = Model(inputs=layers[0].input, outputs=x)
return new_model
Run like this
import tensorflow as tf
from tensorflow.keras.applications.mobilenet import MobileNet
from tensorflow.keras.layers import Dropout
from tensorflow.keras.models import Model
base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(192, 192, 3), dropout=.15)
dropout_layer = Dropout(0.5)
# add single layer after last dropout
mobile_net_with_dropout = insert_single_layer_in_model(base_model, "conv_pw_13_bn", dropout_layer)
# systematically add layers after any batchnorm layer
mobile_net_with_multi_dropout = insert_layers_in_model(base_model, "bn", dropout_layer)
By the way, you should absolutely experiment, but it's unlikely you want additional regularization on top of batchnorm for a small net like mobilenet.