How to clearing Tensorflow-Keras GPU memory?

Question

I am fitting a model in a for loop, but I am getting an error that my GPU's memory is full. I am using Keras in Anaconda Spyder IDE. My GPU is a Asus GTX 1060 6gb.

I have also used codes like: K.clear_session(), gc.collect(), tf.reset_default_graph(), del custom_model but none of them worked. GPU properties say's 98% of memory is full:

Nothing flush GPU memory except numba.cuda.close() but will not allow me to use my GPU again. The only way to clear it is restarting kernel and rerun my code.

I am looking for any script code to add my code allow me to use my code in for loop and clear GPU in every loop.

Answer 1

Wrap up the model creation and training part in a function then use subprocess for the main work. When training is done, subprocess will be terminated and GPU memory will be free.

something like:

import multiprocessing

def create_model_and_train( ):
      .....
      .....

p = multiprocessing.Process(target=create_model_and_train) 
p.start() 
p.join()

Or you can create below function and call it before each run:

from keras.backend.tensorflow_backend import set_session
from keras.backend.tensorflow_backend import clear_session
from keras.backend.tensorflow_backend import get_session
import tensorflow
import gc

# Reset Keras Session
def reset_keras():
    sess = get_session()
    clear_session()
    sess.close()
    sess = get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    print(gc.collect()) # if it does something you should see a number as output

    # use the same config as you used to create the session
    config = tensorflow.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    set_session(tensorflow.Session(config=config))