How can you create an os.environ object with a modified environment, e.g. after loading many different modules with "module load"?

Question

I have a python script that calls an application using subprocess. I am calling this application many times, currently I am doing something along the lines of

out, err = subprocess.Popen(f"module load {' '.join(my_module_list)} && ./my_binary", stdout=sp.PIPE, stderr=sp.STDOUT, shell = True).communicate()

to run my program. Ideally I would like to first generate a modified os.environ object that already contains all the paths to the modules I am loading, and then pass it to subprocess.Popen under the env argument. However, since the printenv command doesn't output a python dictionary format, I'm not sure how to access all the modifications that modules load makes to the environment variables. Is there a good, clean way to create the required modified os.environ object?

Answer 1

I'd be tempted to call python in the subprocess and dump from os.environ in it

python -c 'import os; print(os.environ)'

Once you know what you're after, you can pass a dict directly to subprocess's env arg to set custom environmental vars, which could be something like

custom_env = os.environ.copy()
custom_env["foo"] = "bar"

subprocess.Popen(
    ...
    env=custom_env,
)