Run multiple versions of the same function independently in python

Question

I currently have a for loop that creates a pandas dataframe on each iteration and launches a function where that dataframe is the argument for the function:

for ii in hold_:
        
 temp_df = runners_df[runners_df['market_id'] == ii]
 from odds import odds
 odds(temp_df ) 

Is there a way I can run the all the versions of the function called in the loop independently but concurrently. I want to add a variable time delay within the function, so some versions of the function in the loop may not finish for 5 hours, while some will finish almost instantaneously. Can I use threading for this?

Thanks!

Answer 1

Threading would be the first thing to try. The code below should approximately work in your case. It creates a thread for each task. Depending on what your odds does, your code could be further enhanced with muliprocessing, at the cost of increased complexity.

from odds import odds
from threading import Thread


threads: list[Thread] = []
for ii in hold_:
    temp_df = runners_df[runners_df['market_id'] == ii]
    th = Thread(target=odds, args= (temp_df,), name=f"Thread #{ii}")
    threads.append(th)
    th.start()

for th in threads:
    th.join()