Search code examples
tensorflowkerasmulti-gpu

My Keras network with multi_gpu_model uses only 1 GPU

I have a problem when trying to use Keras with three GPUs.

My psuedocode is as follows: 

import keras
import keras.models as M
from keras.utils import multi_gpu_model 

i = M.Input(None,None,6) 
o1,o2,o3 = my_Network(i)

net = M.Model(inputs = i, outputs = [o1,o2,o3])
net = multi_gpu_model(net,gpus = 3) 

net.compile( ~~~~~ ) 
net.fit(~~~~~ )

My code is training my network, however, only one GPU is utilised.

My configuration is as follows:

keras : 2.3.1

tensorflow : 2.1.0

Cuda : 10.0

windows : 10

GPU : Tesla 100 x 3 (VRAM : 32GB x 3 )

What is the mistake?

Solution

  • Something that you must consider is the batch size when executing fit. You aren't showing this here but you need to make sure that you are giving it a batch size that is divisible by 3 in order to parallelize it across your 3 GPUs. If you are giving it a batch size of 1, for example, it will not be able to distribute the training across the GPUs.

    You did not provide much information but based on your execution of multi_gpu_model, I don't see anything clearly wrong.