I have used SSD300(imagenet pre-trained model) for detection and recognition for two class classification:[Basketball-ChainerCV] (https://github.com/atom2k17/Basketball-ChainerCV/blob/master/basketballproject.py). The training and predictions are good. But when I am using SSD512(imagenet pre-trained model) while training, I get the following error:
/usr/local/lib/python3.6/dist-
packages/chainer/functions/connection/convolution_2d.py in
_forward_cudnn(self, x, W, b, y)
226 cuda.cudnn.convolution_forward(
227 x, W, b, y, pad, stride, dilation, self.groups,
228 auto_tune=auto_tune, tensor_core=tensor_core)
229 return y,
230
cupy/cudnn.pyx in cupy.cudnn.convolution_forward()
cupy/cudnn.pyx in cupy.cudnn._find_algorithm_fwd()
cupy/cuda/memory.pyx in cupy.cuda.memory.alloc()
cupy/cuda/memory.pyx in cupy.cuda.memory.MemoryPool.malloc()
cupy/cuda/memory.pyx in cupy.cuda.memory.MemoryPool.malloc()
cupy/cuda/memory.pyx in cupy.cuda.memory.SingleDeviceMemoryPool.malloc()
cupy/cuda/memory.pyx in cupy.cuda.memory.SingleDeviceMemoryPool._malloc()
cupy/cuda/memory.pyx in cupy.cuda.memory._try_malloc()
OutOfMemoryError: out of memory to allocate 1073741824 bytes (total
12092240384 bytes)
I am using Google Colab GPU environment. Any pointers towards solving this issue is appreciated.
As suggested in the comments by @corochann this memory issue is solved when trainer.run() was executed taking batch_size as 4 from 32. So taking a smaller batch_size was the solution here.