Does PyTorch have implicit functions for element-wise product and sum?

Question

I am trying to implement RCAN for super resolution in Tensorflow by Yulun Zhang et al (the original code published with paper is implemented in PyTorch: https://github.com/yulunzhang/RCAN).

I am trying to understand how they have implemented RCAB. By looking at the diagram they have published of their network architecture, it seems pretty straight forward how the nueral network is built. But the code doesn't seem to match it.

According to the diagram here: https://raw.githubusercontent.com/yulunzhang/RCAN/master/Figs/RCAB.PNG

Each RCAB should have following structure:

Residual Channel Attention Block(RCAB){

--0) Conv2D
--1) Relu
--2) Conv2D
--3) Channel Attention Layer{
----0)Global pooling
----1)Conv2D
----2)Relu
----3)Conv2D
----4)Sigmoid
----5)Element Wise Product (Input of this layer/function would be the output from the Conv2D layer 3)
--}
--4) Element Wise Sum (Input of this layer/function would be the input of layer 1)
}

However, when I print the PyTorch model in the paper's GitHub repo, RCAB looks like this: (see https://github.com/yulunzhang/RCAN/blob/master/RCAN_TrainCode/experiment/model/Network_RCAN_BIX2_G10R20P48-2018-07-15-20-14-55.txt for the full printed model)

(0)RCAB(
          (body): Sequential(
            (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): ReLU(inplace)
            (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (3): CALayer(
              (avg_pool): AdaptiveAvgPool2d(output_size=1)
              (conv_du): Sequential(
                (0): Conv2d(64, 4, kernel_size=(1, 1), stride=(1, 1))
                (1): ReLU(inplace)
                (2): Conv2d(4, 64, kernel_size=(1, 1), stride=(1, 1))
                (3): Sigmoid()
              )
            )
          )
        )

There seems to be no mention of Element Wise sum and Element Wise product in the RCABs of the models that are published along the paper. Signmoid layer is the last layer in each RCAB.

So my question is: does Pytorch have some implicit way of declaring thes element wise sum/product layers? or is it the case that the publishers of the code/model simply haven't added any such layer and therefore have not followed the model architecture diagram that they published?

Answer 1

If you look at their actual model file you can find the elementwise sum (implemented as just +): https://github.com/yulunzhang/RCAN/blob/master/RCAN_TrainCode/code/model/rcan.py

I believe the elementwise product is handled the same way. These are not exactly "part of the model" in the PyTorch sense. They are not created in __init__ and are kind of dynamic, only revealing its behavior during the forward pass. Static model analysis could not have revealed them (and thus not show in the txt).