I am building a network that is re-using the region proposal network weights from the matterport Mask R-CNN repo (mask_rcnn_coco.h5), and only the RPN. The code is very similar to what's in the original repository:
def rpn_graph(self, feature_map, anchors_per_location=3, anchor_stride=1):
"""Builds the computation graph of Region Proposal Network.
feature_map: backbone features [frames, height, width, depth]
anchors_per_location: number of anchors per pixel in the feature map
anchor_stride: Controls the density of anchors. Typically 1 (anchors for
every pixel in the feature map), or 2 (every other pixel).
Returns:
rpn_class_logits: [frames, H * W * anchors_per_location, 2] Anchor classifier logits (before softmax)
rpn_probs: [frames, H * W * anchors_per_location, 2] Anchor classifier probabilities.
rpn_bbox: [frames, H * W * anchors_per_location, (dy, dx, log(dh), log(dw))] Deltas to be
applied to anchors.
"""
# Shared convolutional base of the RPN
shared = KL.Conv2D(512, (3, 3), padding='same', activation='relu',
strides=anchor_stride, name='rpn_conv_shared', trainable=False)(feature_map)
# Anchor Score. [batch, height, width, anchors per location * 2].
x = KL.Conv2D(2 * anchors_per_location, (1, 1), padding='valid',
activation='linear', name='rpn_class_raw', trainable=False)(shared)
# Reshape to [batch, anchors, 2]
rpn_class_logits = KL.Lambda(lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 2]))(x)
# Softmax on last dimension of BG/FG.
rpn_probs = KL.Activation("softmax", name="rpn_class_xxx")(rpn_class_logits)
# Bounding box refinement. [batch, H, W, anchors per location * depth]
# where depth is [x, y, log(w), log(h)]
x = KL.Conv2D(anchors_per_location * 4, (1, 1), padding="valid", activation='linear',
name='rpn_bbox_pred', trainable=False)(shared)
# Reshape to [batch, anchors, 4]
rpn_bbox = KL.Lambda(lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 4]))(x)
return [rpn_class_logits, rpn_probs, rpn_bbox]
def build_rpn_model(self):
"""Builds a Keras model of the Region Proposal Network.
It wraps the RPN graph so it can be used multiple times with shared
weights.
Returns a Keras Model object. The model outputs, when called, are:
rpn_class_logits: [batch, H * W * anchors_per_location, 2] Anchor classifier logits (before softmax)
rpn_probs: [batch, H * W * anchors_per_location, 2] Anchor classifier probabilities.
rpn_bbox: [batch, H * W * anchors_per_location, (dy, dx, log(dh), log(dw))] Deltas to be
applied to anchors.
"""
input_feature_map = KL.Input(shape=[None, None, self.DEPTH], name="input_rpn_feature_map")
outputs = self.rpn_graph(
input_feature_map,
self.ANCHORS_PER_LOCATION,
self.ANCHOR_STRIDE
)
model = KM.Model([input_feature_map], outputs, name="rpn_model")
self.load_pre_trained_rpn_weights(model)
return model
def load_pre_trained_rpn_weights(self, model):
"""
We load the weights for the region proposal network from the pre-trained mask-rcnn model
:param model: Keras model for the region proposal network
:return:
"""
weights = h5py.File(self.PATH_TO_MASK_RCNN_WEIGHTS, mode='r')
saving.load_weights_from_hdf5_group_by_name(weights, [model])
weights.close()
Then I initialize the RPN in init to use later on:
...
self.region_proposal_network = self.build_rpn_model()
...
Training is running well and I can see the checkpoint is saved. But then, when the model is loaded for evaluation I get (using tf.estimator.train_and_evaluate)
RuntimeError: Graph is finalized and cannot be modified.
Any tips that could help me out?
I found what was causing the problem.
The estimator runs training then evaluation. During training we load the pretrained weights, then after training, a checkpoint is created. The weights are then loaded from the checkpoint for evaluation at which point the weights are locked (for good reason). But then I was trying to load the weights on top of the frozen model, hence the issue.
TL;DR: I ended up loading the weights only during the train phase.