Stepwise decrease GPU utility followed by out of memory error
I am running a 3D image segmentation deep learning training pipeline on a GCloud VM and am noticing a stepwise decrease of the GPU utility after about 25 epochs and an out of memory error after 32 epochs. Since such a training pipeline is basically the same loop over the data over and over again, and since all other main metrics do not show such a pattern change, I don't understand why the first epochs are fine and it then suddenly occurs.
Could this be some kind of memory leak on the GPU? Could GCloud apply some kind of throttling based on the GPU temperature?
Some context info:
- I am using Julia 1.9.0 with FastAI.jl 0.5.1, Flux.jl 0.13.16, CUDA.jl 4.2.0
- The VM is Ubuntu 22.04 x86_64 with CUDA toolkit 12.1, NVIDIA driver 530.30.02, and an NVIDIA Tesla T4 GPU with 16GB RAM
- The model is a residual U-Net with approximately 9.5 million parameters, the input data are 3D Float32 images with size (96, 96, 96) and I am using a batch size of 2.
Some things I've tried:
I can reproduce the behaviour reliably, it happens every time after the same amount of epochs
If I decrease the input image size, it still happens but later (epoch 60)
If I decrease the model size it happens earlier (this I especially don't understand)
I've set
JULIA_CUDA_MEMORY_POOLtononeand added a callback after each epoch that executesGC.gc(true)andCUDA.reclaim()
The problem was resolved by changing my optimiser from Flux.Nesterov to Optimisers.Nesterov as suggested here. Apparently the Flux optimisers gathers some kind of state whereas the ones from Optimisers.jl do not.
- My C code works fine but crashes right before the end (0xC0000374)
- What uses are there for "placement new"?
- Managing memory with recursion and for loops (C++20)
- Is there a way to stop a memory leak with a queue and a custom struct with a character array?
- Amazon EC2 instance : Cannot create temp file for here document. No space left on device
- Reducing memory consumption of mysql on ubuntu@aws micro instance
- With .NET 9 escape analysis, are struct and class now equal?
- Android Out of Memory Exception while HTML unescaping a String
- Choice between vector::resize() and vector::reserve()
- iPhone OS Memory Warnings. What Do The Different Levels Mean?
- How do I free a GTK 4 widget in C?
- Does "automatic storage" now include registers?
- What is thrashing? Why does it occur?
- Twice number of page faults for access to zero-initialized memory
- Understanding JVM Memory Allocation and Java Out of Memory: Heap Space
- Why allocate memory? (C++)
- Canonical example of stack destruction in a C program
- Is there any way to flash a raspberry Pi Pico with a firmware contained in a pointer inside it?
- Compelling examples of custom C++ allocators?
- Error while deleting 3D dynamic array in FreePascal
- Is there a way to limit the memory GHCi can have?
- Virtual memory management in Linux
- Why is the '&' operator needed for an int variable but not for an array in C?
- Efficient linked list design for a C file manager
- Confusion between stack, call stack and memory heap in Javascript
- Why does Go not throw an error when dereferencing a dangling pointer immediately after a variable goes out of scope?
- Can some explain the performance behavior of the following memory allocating C program?
- serial memory pages (virtual memory) and TLB hit
- Memory Alignment in C/C++
- Memory Lifetime within a process