Search code examples
pythontensorflowkerasdeep-learning

Using keras fit/fit_generator with max_queue_size, workers and use_multiprocessing

I am confused on how to you use the max_queue_size, workers and use_multiprocessing in Keras Documentation

Can someone please show an example of how would you use them if you had

  • 1xGPU(Nvidia Quadro p1000)
  • 6 core CPU with 12 logical processors

Here is how I use it based on unscientific guesses for those three fields.

classifier.fit_generator(training_set,
                         steps_per_epoch = 8000,
                         epochs = 25,
                         validation_data = test_set,
                         validation_steps = 2000/32,
                         max_queue_size = 10,
                         use_multiprocessing = False,
                         workers=1)
Solution

  • MAX_QUEUE_SIZE --> Increase this parameter if, when you monitor your GPU usage, the GPU is idling(waiting for batches). Ideally, the GPU should wait at little as possible for the CPU to fetch data. Waiting for batches means that GPU memory consumption is not at a constant peak, say 95%). Case in point : when you monitor the GPU memory usage, you see big spikes (5% usage, 95% usage, 5% usage, 95% usage). The time_difference between 95% and 5% usage is actually the time when the GPU is idling. Increase this queue_size if you such usage discrepancies.

    USE_MULTIPROCESSING --> May generate errors on Windows(to me it did not happen, but I saw other posts in which, due to multiprocessing issues it may freeze), works fine on Linux based systems. Set this to true if you want to use multiple processes to fetch data to your CPU. e.g.

     ..,use_multiprocessing = True, workers = 4)

    In my experience, these parameters work together; i.e. if you want to increase your performance in terms of speed, you could try to increase all of them.