Search code examples
pythonmachine-learningartificial-intelligencegoogle-colaboratory

CUDA out of memory Dreambooth

A few days ago I ran this known DreamBooth Google Colab with my face and I was able to get some results. But yesterday I wanted to retrain it again to make it better (tried using the same photos again), and right now, it throws this out of memory exception:

image.png

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 12.24 GiB already allocated; 501.75 MiB free; 13.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF`

I was wondering if someone had the same problem and know why is this happening now and not before. I'm linking my colab to show it to you.

Solution

  • I´ve just reduced the resolution and was able to run it.

    Go to

    Setting up all training args

    And change

    resolution=512

    to

    resolution=256

    Sure, you´ll loose quality. But at least it works with the free colab version.

    UPDATE: just tried with 480 and works as well. Just be sure to restart the environment to free the GPU memory first.