I have a code the reads a very large text file and process each line in Pools.
In the case of the elif I need to sleep the whole process for 120 seconds, in other words, I want all other Pools created to pause. But after 120 seconds all Pools should resume working.
The code's functionality is similar to this:
from multiprocessing import Pool
import sys
sys.tracebacklimit = 0
def req(line):
if "@" not in line:
# (some function for processing here)
return line
elif "somestring" in line:
#HERE I NEED TO SLEEP ALL POOLS
else:
# (some function for processing)
return line
if __name__ == "__main__":
pool = Pool(20)
with open("list.txt") as source_file:
# chunk the work into batches of 20 lines at a time
pool.map(req, source_file, 35)
As is said by @abarnert, you should use an Event object as followed:
from multiprocessing import Pool
import sys
from threading import Event, Timer
sys.tracebacklimit = 0
# Setup clojure environment
def reqgen():
ev_stop = Event()
def req(line):
# Wait at the start
if ev_stop.is_set():
ev_stop.wait()
if "somestring" in line:
#HERE I NEED TO SLEEP ALL POOLS
# Clear the internal flag, make all workers await
ev_stop.clear()
# An alarm to reset the internal flag,
# which will make all workers get back to work
Timer(120, lambda: ev_stop.set()).start()
# Regular work
return req(line)
else:
# (some function for processing)
return line
return req
if __name__ == "__main__":
pool = Pool(20)
with open("list.txt") as source_file:
# chunk the work into batches of 20 lines at a time
pool.map(reqgen(), source_file, 35)