Fitting multiple Keras models consecutively leads to GPU OOM

Question

I want to fit and save three models consecutively on the same data (the NIH-Chest set - https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community). Even though I can do this separately I am wondering if I can do this from a list (see code below). At this point however this leads to a GPU memory saturation and consequently an OOM error. This is caused by the dataset not being flushed from the GPU memory and trying to reload it as well for the second and third fit.

All the solutions relating clearing the GPU memory either involve closing Cuda or restarting Keras or the kernel...which in causes fit to be erased (no surprise there).

I am wondering, is it possible to fit multiple models consecutively from a list or should I just go back to basics and fit and save every model separate?

Small code snippet (for a quick test run > small# epochs and validation);

epochs = 2
i = 0
for model in model_list:
    i += 1
    model.fit_generator(train_data, epochs = epochs, validation_data = validation_data, validation_steps = 100)
    model.save('model_{}'.format(i))

Answer 1

You can use del(model) at the end of the loop. This would delete the memory of the model which you would not intend to use later.