Search code examples
pythonpytorchbatch-normalization

Is batch normalisation a nonlinear operation?

Can batch normalisation be considered a nonlinear operation like relu or sigmoid?

I have come across a resnet block like:

: IBasicBlock(
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (prelu): PReLU(num_parameters=512)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )

so I wonder if batchnormalisation can act as non linear operation

Solution

  • Batch Normalization is non-linear in the data points xi on a per batch training basis, but is linear during inference in the scaling parameters (as these are fixed).