Why is the deeplab v3+ model confused about pixels outside image boundary?

Question

I'm using the google research github repository to run deeplab v3+ on my dataset to segment parts of a car. The crop size I've used is 513,513 (default) and the code adds a boundary to images smaller than that size (correct me if I'm wrong).

example!

The model seems to be performing poorly on the added boundary. Is there something I'm supposed to correct or will the model do fine with more training ?

Update: Here's the tensorboard graphs for training. Why is the regularization loss shooting like that? The output seems to be improving, can someone help me making inferences from these graphs?

Answer 1

Is there something I'm supposed to correct or will the model do fine with more training ?

its Ok, don't mind the boundary

---

To inference you can use this code

import cv2
import tensorflow as tf
import numpy as np
from PIL import Image

from skimage.transform import resize



class DeepLabModel():
    """Class to load deeplab model and run inference."""

    INPUT_TENSOR_NAME = 'ImageTensor:0'
    OUTPUT_TENSOR_NAME = 'SemanticPredictions:0'
    INPUT_SIZE = 513


    def __init__(self, path):
        """Creates and loads pretrained deeplab model."""
        self.graph = tf.Graph()

        graph_def = None
        # Extract frozen graph from tar archive.

        with tf.gfile.GFile(path, 'rb')as file_handle:
            graph_def = tf.GraphDef.FromString(file_handle.read())

        if graph_def is None:
            raise RuntimeError('Cannot find inference graph')

        with self.graph.as_default():
            tf.import_graph_def(graph_def, name='')

        self.sess = tf.Session(graph=self.graph)


    def run(self, image):
        """Runs inference on a single image.

        Args:
          image: A PIL.Image object, raw input image.

        Returns:
          seg_map: np.array. values of pixels are classes
        """

        width, height = image.size

        resize_ratio = 1.0 * self.INPUT_SIZE / max(width, height)
        target_size = (int(resize_ratio * width), int(resize_ratio * height))

        resized_image = image.convert('RGB').resize(target_size, Image.ANTIALIAS)

        batch_seg_map = self.sess.run(
           self.OUTPUT_TENSOR_NAME,
           feed_dict={self.INPUT_TENSOR_NAME: [np.asarray(resized_image)]})

        seg_map = batch_seg_map[0]        

        seg_map = resize(seg_map.astype(np.uint8), (height, width), preserve_range=True, order=0, anti_aliasing=False)


        return seg_map

the code is based on this file https://github.com/tensorflow/models/blob/master/research/deeplab/deeplab_demo.ipynb

model = DeepLabModel(your_model_pb_path)

img = Image.open(img_path)

seg_map = model.run(img)

---

to get your_model_pb_path you need to export your model to .pb file you can do it using export_model.py file in Deeplab repo https://github.com/tensorflow/models/blob/master/research/deeplab/export_model.py

if you were training xception_65 version

python3 <path to your deeplab folder>/export_model.py \
  --logtostderr \
  --checkpoint_path=<your ckpt> \
  --export_path="./my_model.pb" \
  --model_variant="xception_65" \
  --atrous_rates=6 \
  --atrous_rates=12 \
  --atrous_rates=18 \
  --output_stride=16 \
  --decoder_output_stride=4 \
  --num_classes=<NUMBER OF YOUR CLASSES> \
  --crop_size=513 \
  --crop_size=513 \
  --inference_scales=1.0 

<your ckpt> is a path to your trained model checkpoint you can find checkpoints in the folder that you passed as argument --train_logdir when training

you need to include only model name and number of iterations in path, or in other words you will have in your training folder, for example, files model-1500.meta, model-1500.index and model-1000.data-00000-of-00001 you need to discard everything that goes after ., so the ckpt path will be model-1000

please make sure that atrous_rates are the same as you used to train the model

if you were training mobilenet_v2 version

python3 <path to your deeplab folder>/export_model.py \
  --logtostderr \
  --checkpoint_path=<your ckpt> \
  --export_path="./my_model.pb" \
  --model_variant="mobilenet_v2" \
  --num_classes=<NUMBER OF YOUR CLASSES>  \
  --crop_size=513 \
  --crop_size=513 \
  --inference_scales=1.0

more you can find here https://github.com/tensorflow/models/blob/master/research/deeplab/local_test_mobilenetv2.sh https://github.com/tensorflow/models/blob/master/research/deeplab/local_test.sh

---

You can visualize results using this code

img_arr = np.array(img)

# as may colors as you have classes
colors = [(255, 0, 0), (0, 255, 0), ...] 

for c in range(0, N_CLASSES):
    img_arr[seg_map == c] = 0.5 * img_arr[seg_map == c] + 0.5 * np.array(colors[c])

cv2.imshow(img_arr)
cv2.waitKey(0)