How to compile multiple subprocess python files into single .exe file using pyinstaller

Question

I have a similar question to this one:Similar Question. I have a GUI and where the user can input information and the other scripts use some of that information to run.I have 4 different scripts for each button. I run them as a subprocess so that the main gui doesn’t act up or say that it’s not responding. This is an example of what I have since the code is really long since I used PAGE to generate the gui.

###Main.py#####
import subprocess

def resource_path(relative_path):
    #I got this from another post to include images but I'm also using it to include the scripts"
    try:
        # PyInstaller creates a temp folder and stores path in _MEIPASS
        base_path = sys._MEIPASS
    except Exception:
        base_path = os.path.abspath(".")

    return os.path.join(base_path, relative_path)
Class aclass:
    def get_info(self):
        global ModelNumber, Serial,SpecFile,dateprint,Oper,outputfolder
        ModelNumber=self.Model.get()
        Serial=self.SerialNumber.get()
        outputfolder=self.TEntry2.get()
        SpecFile= self.Spec_File.get()

        return ModelNumber,Serial,SpecFile,outputfolder

    def First(self):
        aclass.get_info(self)                          #Where I use the resource path function
        First_proc = subprocess.Popen([sys.executable, resource_path('first.py'),str(ModelNumber),str(Serial),str(path),str(outputfolder)])
        First_proc.wait()


#####First.py#####
import numpy as np
import scipy 
from main import aclass

ModelNumber    = sys.argv[1]
Serial         = sys.argv[2]
path           = sys.argv[3]
path_save      = sys.argv[4]

and this goes on for my second,third, and fourth scripts.

In my spec file, I added:

a.datas +=[('first.py','C\\path\\to\\script\\first.py','DATA')]
a.datas +=[('main.py','C\\path\\to\\script\\main.py','DATA')]

this compiles and it works, but when I try to convert it to an .exe, it crashes because it can't import first.py properly and its own libraries (numpy,scipy....etc). I've tried adding it to the a.datas, and runtime_hooks=['first.py'] in the spec file...and I can't get it to work. Any ideas? I'm not sure if it's giving me this error because it is a subprocess.

Answer 1

Assuming you can't restructure your app so this isn't necessary (e.g., by using multiprocessing instead of subprocess), there are three solutions:

• Ensure that the .exe contains the scripts as an (executable) zipfile—or just use pkg_resources—and copy the script out to a temporary directory so you can run it from there.
• Write a multi-entrypoint wrapper script that can be run as your main program, and also run as each script—because, while you can't run a script out of the packed exe, you can import a module out of it.
• Using pkg_resources again, write a wrapper that runs the script by loading it as a string and running it with exec instead.

The second one is probably the cleanest, but it is a bit of work. And, while we could rely on setuptools entrypoints to some of the work, trying to explain how to do this is much harder than explaining how to do it manually,¹ so I'm going to do the latter.

---

Let's say your code looked like this:

# main.py
import subprocess
import sys
spam, eggs = sys.argv[1], sys.argv[2]
subprocess.run([sys.executable, 'vikings.py', spam])
subprocess.run([sys.executable, 'waitress.py', spam, eggs])

# vikings.py
import sys
print(' '.join(['spam'] * int(sys.argv[1])))

# waitress.py
import sys
import time
spam, eggs = int(sys.argv[1]), int(sys.argv[2]))
if eggs > spam:
    print("You can't have more eggs than spam!")
    sys.exit(2)
print("Frying...")
time.sleep(2)
raise Exception("This sketch is getting too silly!")

So, you run it like this:

$ python3 main.py 3 4
spam spam spam
You can't have more eggs than spam!

---

We want to reorganize it so there's a script that looks at the command-line arguments to decide what to import. Here's the smallest change to do that:

# main.py
import subprocess
import sys
if sys.argv[1][:2] == '--':
    script = sys.argv[1][2:]
    if script == 'vikings':
        import vikings
        vikings.run(*sys.argv[2:])
    elif script == 'waitress':
        import waitress
        waitress.run(*sys.argv[2:])
    else:
        raise Exception(f'Unknown script {script}')
else:
    spam, eggs = sys.argv[1], sys.argv[2]
    subprocess.run([sys.executable, __file__, '--vikings', spam])
    subprocess.run([sys.executable, __file__, '--waitress', spam, eggs])

# vikings.py
def run(spam):
    print(' '.join(['spam'] * int(spam)))

# waitress.py
import sys
import time
def run(spam, eggs):
    spam, eggs = int(spam), int(eggs)
    if eggs > spam:
        print("You can't have more eggs than spam!")
        sys.exit(2)
    print("Frying...")
    time.sleep(2)
    raise Exception("This sketch is getting too silly!")

And now:

$ python3 main.py 3 4
spam spam spam
You can't have more eggs than spam!

---

A few changes you might want to consider in real life:

• DRY: We have the same three lines of code copied and pasted for each script, and we have to type each script name three times. You can just use something like __import__(sys.argv[1][2:]).run(sys.argv[2:]) with appropriate error handling.
• Use argparse instead of this hacky special casing for the first argument. If you're already sending non-trivial arguments to the scripts, you're probably already using argparse or an alternative anyway.
• Add an if __name__ == '__main__': block to each script that just calls run(sys.argv[1:]), so that during development you can still run the scripts directly to test them.

I didn't do any of these because they'd obscure the idea for this trivial example.

---

_{1 The documentation is great as a refresher if you've already done it, but as a tutorial and explanatory rationale, not so much. And trying to write the tutorial that the brilliant PyPA guys haven't been able to come up with for years… that's probably beyond the scope of an SO answer.}