Run a bash command with variables as a python subprocess

Question

I have this shell command:

$ docker run -it --env-file=.env -e "CONFIG=$(cat /path/to/your/config.json | jq -r tostring)" algolia/docsearch-scraper

And I want to run it as a python subprocess. I thought I'll only need an equivalent of the jq -r tostring, but if I use the config.json as a normal string the " don't get escaped. I also escaped them by using json.load(config.json). With the original jq command the " don't get escaped either and it's just returning the json string.

When I use the json returned as a string in python subprocess i get always a FileNotFoundError on the subprocess line.

@main.command()
def algolia_scrape():
    with open(f"{WORKING_DIR}/conf_dev.json") as conf:
        CONFIG = json.load(conf)
        subprocess.Popen(f'/usr/local/bin/docker -it --env-file={WORKING_DIR}/algolia.env -e "CONFIG={json.dumps(CONFIG)}" algolia/docsearch-scraper')

Answer 1

You get "file not found" because (without shell=True) you are trying to run a command whose name is /usr/local/bin/docker -it ... when you want to run /usr/local/bin/docker with some arguments. And of course it would be pretty much a nightmare to try to pass the JSON through the shell because you need to escape any shell metacharacters from the string; but just break up the command string into a list of strings, like the shell would.

def algolia_scrape():
    with open(f"{WORKING_DIR}/conf_dev.json") as conf:
        CONFIG = json.load(conf)
    p = subprocess.Popen(['/usr/local/bin/docker', '-it',
        f'--env-file={WORKING_DIR}/algolia.env',
        '-e', f'CONFIG={json.dumps(CONFIG)}',
        'algolia/docsearch-scraper'])

You generally want to save the result of subprocess.Popen() because you will need to wait for the process when it terminates.