I would like to start 3 distinct functions at the same time on machine with multiple cores. Each function launches an extensive data query to separate databases using the same module (cx_Oracle) and each function is called with some arguments. How could this be accomplished?
I have read a bit on parallel computing with Python but ProcessPoolExecutor always returns BrokenProcessPool error. The script is saved (reported as an issue in other SE item).
if __name__ == '__main__'
with ProcessPoolExecutor(3) as executor:
future_1 = executor.submit(query_1, arg1)
future_2 = executor.submit(query_2, arg2)
future_3 = executor.submit(query_3, arg3)
result_1 = future_1.result()
result_2 = future_2.result()
result_3 = future_3.result()
BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Have you seen something similar? A reference to other thread or an example of three distinct calls with arguments is enough for me. I have also tried multiprocessing but not successfully.
The cx_Oracle sample Threads.py shows running two statements in different threads:
. . .
def TheLongQuery():
. . .
def DoALock():
. . .
thread1 = threading.Thread(None, TheLongQuery)
thread1.start()
thread2 = threading.Thread(None, DoALock)
thread2.start()
thread1.join()
thread2.join()