How to set shell=True for a Subprocess.run with a concurrent.future Pool Threading Executor

Question

I try to use the concurrent.future multithreading in Python with subprocess.run to launch an external Python script. But I have some troubles with the shell=True part of the subprocess.run().

Here is an example of the external code, let's call it test.py:

#! /usr/bin/env python3

import argparse

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-x', '--x_nb', required=True, help='the x number')
    parser.add_argument('-y', '--y_nb', required=True, help='the y number')
    args = parser.parse_args()

    print('result is {} when {} multiplied by {}'.format(int(args.x_nb) * int(args.y_nb),
                                                         args.x_nb,
                                                         args.y_nb))

In my main python script I have:

#! /usr/bin/env python3

import subprocess
import concurrent.futures
import threading
...

args_list = []
for i in range(10):
    cmd = './test.py -x {} -y 2 '.format(i)
    args_list.append(cmd)

# just as an example, this line works fine
subprocess.run(args_list[0], shell=True)

# this multithreading is not working       
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    executor.map(subprocess.run, args_list)

The problem here is that I can't pass the shell=True option to the executor.map.

I have already tried without success:

args_list = []
for i in range(10):
    cmd = './test.py -x {} -y 2 '.format(i)
    args_list.append((cmd, eval('shell=True'))

or

args_list = []
for i in range(10):
    cmd = './test.py -x {} -y 2 '.format(i)
    args_list.append((cmd, 'shell=True'))

Anyone has an idea on how to solve this problem?

Answer 1

I don't think the map method can call a function with keyword args directly but there are 2 simple solutions to your issue.

Solution 1: Use a lambda to set the extra keyword argument you want

The lambda is basically a small function that calls your real function, passing the arguments through. This is a good solution if the keyword arguments are fixed.

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    executor.map(lambda args: subprocess.run(args, shell=True), args_list)

Solution 2: Use executor.submit to submit the functions to the executor

The submit method lets you specify args and keyword args to the target function.

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    for args in args_list:
        executor.submit(subprocess.run, args, shell=True)