Search code examples
pythonmultiprocessingpathos

Shutting down resources with python ThreadPool on completion

I have a script that uses, pathos ProcessPool to do a function data() parallel on a number of data points.

The function called data() calculates and sends the result to a single object called Database and since a lot of pushes are sent to the db, I added a simple queue, when the queue size is greater than 500, it gets pushed to the db.

The problem occurs when the all the data points have been mapped but there is still some amount of data in the queue of class Database (in each process, process 1 might have 100 process 2 might have 200).

I wrote a function to flush on exit, but how do I call that function on all the processes ?

Solution

  • There is Manager in multiprocessing that let's you access data stored by the processes in one go, instead of using queue.

    You can read about it here