Search code examples
pythonmultiprocessingterminate

Mulitprocessing, terminate and corrupt queues

Whilst looking at the python doc for multiprocessing.terminate() I came across the following:

Terminate() If this method is used when the associated process is using a pipe or queue then the pipe or queue is liable to become corrupted and may become unusable by other process. Similarly, if the process has acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock.

Which basically says if you terminate a process which is using a queue, pipe or similar you run the risk of the structure becoming corrupt.

I have a couple of questions about this,

  1. What will happen to another process trying to retrieve the data from the PIPE, Queue or similar if a corruption occurs?
  2. How can a process check to see if there is corruption?
  3. Can the deadlock be resolved in any way if you know another process has been terminated?

I understand you should always try to not use terminate, but this is for that situation where you cannot do anything else but this

Solution

  • Okay, so it was not the nicest solution but I have firstly handled the exceptions as best as possible. I did not want to run the risk of having corruption so I now restart the application in an instance where there is the possibility of a corruption occurring to reduce the chance of possible issues.

    Thanks @MarwanAlsabbagh for your suggestion.