How to add two layers within a CNTK Sequential

Question

I'm currently reimplement my TensorFlow implementation of Jonathan Longs FCN8-s using CNTK. While TensorFlow is meanwhile very familar to me I'm very inexperienced in using Microsofts CNTK yet. I read a few CNTK Github tutorials but now I'm at the point where I want to add pool4_score with the upscore layer. In TensorFlow I would simply use tf.add(pool4_score, upscore1) but in CNTK I have to use Sequentials (correct?) So my code looks like:

with default_options(activation=None, pad=True, bias=True):
    z = Sequential([
        For(range(2), lambda i: [
            Convolution2D((3,3), 64, pad=True, name='conv1_{}'.format(i)),
            Activation(activation=relu, name='relu1_{}'.format(i)),
        ]),
        MaxPooling((2,2), (2,2), name='pool1'),

        For(range(2), lambda i: [
            Convolution2D((3,3), 128, pad=True, name='conv2_{}'.format(i)),
            Activation(activation=relu, name='relu2_{}'.format(i)),
        ]),
        MaxPooling((2,2), (2,2), name='pool2'),

        For(range(3), lambda i: [
            Convolution2D((3,3), 256, pad=True, name='conv3_{}'.format(i)),
            Activation(activation=relu, name='relu3_{}'.format(i)),
        ]),
        MaxPooling((2,2), (2,2), name='pool3'),

        For(range(3), lambda i: [
            Convolution2D((3,3), 512, pad=True, name='conv4_{}'.format(i)),
            Activation(activation=relu, name='relu4_{}'.format(i)),
        ]),
        MaxPooling((2,2), (2,2), name='pool4'),

        For(range(3), lambda i: [
            Convolution2D((3,3), 512, pad=True, name='conv5_{}'.format(i)),
            Activation(activation=relu, name='relu5_{}'.format(i)),
        ]),
        MaxPooling((2,2), (2,2), name='pool5'),

        Convolution2D((7,7), 4096, pad=True, name='fc6'),
        Activation(activation=relu, name='relu6'),
        Dropout(0.5, name='drop6'),

        Convolution2D((1,1), 4096, pad=True, name='fc7'),
        Activation(activation=relu, name='relu7'),
        Dropout(0.5, name='drop7'),

        Convolution2D((1,1), num_classes, pad=True, name='fc8')

        ConvolutionTranspose2D((4,4), num_classes, strides=(1,2), name='upscore1')
        # TODO:
        # conv for pool4_score with (1x512) and 21 classes
        # combine upscore 1 and pool4_score
    ])(input)

I read that there is a combine method .. But I found no examples how to use it within the sequential. So how would I implement the tf.add method using CNTK?

Thanks a lot!

Answer 1

You can use C.plus or +, in this case you will need to split your sequence in order to get to the layer that you want to add.

For example the below:

z = Sequential([Convolution2D((3,3), 64, pad=True),
                MaxPooling((2,2), (2,2))])(input)

Is equivalent to:

 z1 = Convolution2D((3,3), 64, pad=True)(input)
 z2 = MaxPooling((2,2), (2,2))(z1)

You can now do z1 + z2.