Search code examples
pythonsubprocesscondavirtual-environment

Running script in Conda environment from within script in another Conda environment

I have two scripts, let's call them master.py and subject.py. master.py is supposed to run in Python3, while subject.py, in Python2, therefore I created two separate Conda environments for them which we'll call env-master and env-subject. However, master.py is supposed to run subject.py as a subprocess. Something like:

subprocess.run('conda run -n env-subject python2 /path/to/subject.py')

Is it actually possible to make such a thing work? Because, as far as I understand, only one conda environment can be active at any one time. I wonder if there are workarounds for basically using master.py to run multiple batches of subject.py while it simultaneously analyzes the output files of any executions which already completed, and never running more than N subject.pys at once.

Solution

  • Works fine for me. Here's an example:

    Scripts

    so-py23-mix.py

    import sys
import subprocess

print("This is the main process")
print(sys.version)

subprocess.run(['conda', 'run', '-n', 'so-py2', 'python', 'so-py2-sub.py'])

    so-py2-sub.py

    import sys

print "This is the subprocess"
print sys.version

    Running

    shell commands

    mamba create -yn so-py2 python=2
mamba create -yn so-py3 python=3

mamba activate so-py3

python so-py23-mix.py

    output

    This is the main process
3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:09:13) [Clang 13.0.1 ]
This is the subprocess
2.7.15 | packaged by conda-forge | (default, Mar  5 2020, 14:58:04)
[GCC Clang 9.0.1 ]

    Running 2

    Also can get exactly the same behavior without activating, but instead launching through conda run:

    shell

    mamba create -yn so-py2 python=2
mamba create -yn so-py3 python=3

conda run -n so-py3 --live-stream python so-py23-mix.py

    results

    This is the main process
3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:09:13) [Clang 13.0.1 ]
This is the subprocess
2.7.15 | packaged by conda-forge | (default, Mar  5 2020, 14:58:04)
[GCC Clang 9.0.1 ]

    Note that my base environment is Python v3.8.