Search code examples
pythonmultithreadingpool

How to use pool of threads in python, in this case


Suppose I want to process strings in a file, in each line, which are not dependent between each other. How I usually do:

def process_line(filelines, i):
    #process lines
import threading
threads = []
lines = open(file).readlines()
for i in range(0,len(lines)):
    threads.append(threading.Thread(target=process_line, args=(lines,i,))

[t.start() for t in threads]
[t.join() for t in threads]

Solution

  • from concurrent.futures import ThreadPoolExecutor
    
    with ThreadPoolExecutor() as executor:
        [executor.submit(proccess_line, lines, i) for i in range(len(lines))]
    

    ThreadPoolExecutor docs: https://docs.python.org/3/library/concurrent.futures.html

    Note that it's better to use processes instead of threads for these kinds of tasks (replace ThreadPoolExecutor with ProccessPoolExecutor