pyinstaller - collect local package

Question

I have a Python application which I want to bundel with pyinstaller.

My folder structure is like this:

.
├── docs
│   ├── manual
├── main.py
├── README.md
├── requirements.txt
├── setup.sh
├── venv
│   └── ...
└── mypackage
    ├── package1
    ├── package2
    ├── bindata1.bin
    ├── bindata2.bin
    └── __init__.py

mypackage is a python package which also contains some binary data and does some dynamic importing by scanning some submodules/namespaces:

def iter_namespace(ns_pkg):
    return pkgutil.iter_modules(ns_pkg.__path__, ns_pkg.__name__ + ".")


APPS = {}

for finder, name, ispkg in iter_namespace(apps):
    if not ispkg:
        continue
    package = importlib.import_module(name + '.app')
    APPS[package.APPNAME] = package.App

This works fine when running the python script directly, but not with the frozen version. My first approach was this command: pyinstaller main.py --clean --noconfirm --onefile --windowed --hidden-import pyvisa_py So I tried to collect the package by appending --collect-all mypackage, but I get those warnings:

115 INFO: PyInstaller: 5.13.0
115 INFO: Python: 3.10.6
116 INFO: Platform: Linux-5.19.0-46-generic-x86_64-with-glibc2.35
116 INFO: wrote /opt/mysoftware/main.spec
117 INFO: Removing temporary files and cleaning cache in /root/.cache/pyinstaller
121 WARNING: Unable to copy metadata for mypackage: The 'mypackage' distribution was not found and is required by the application
121 WARNING: collect_data_files - skipping data collection for module 'mypackage' as it is not a package.
121 WARNING: collect_dynamic_libs - skipping library collection for module 'mypackage' as it is not a package.
142 INFO: Determining a mapping of distributions to packages...
760 WARNING: Unable to determine requirements for mypackage: The 'mypackage' distribution was not found and is required by the application
760 INFO: Extending PYTHONPATH with paths
['/opt/mysfoftware']
898 INFO: checking Analysis
898 INFO: Building Analysis because Analysis-00.toc is non existent
898 INFO: Initializing module dependency graph...
...

How can I get Pyinstaller to collect my package so that dynamic loading and accessing binary data works?

Answer 1

Copied rokm's from Github discussion, used solution 1:

Yes, that's a limitation of how --collect-submodules mypackage, --collect-data mypackage, --collect-binaries mypackage work. They add calls to collect_submodules(), collect_data_files(), and collect_dynamic_libs() at the top of the generated .spec file, and pass the resulting lists to the Analysis instance.

However, the package search paths are extended with location of your entry-point acript only during instantiation of Analysis. Therefore, those collect_ calls cannot find a local (non-installed) copy of mypackage.

There are three ways to deal with this (aside from installing mypackageto your venv):

1. write a hook for mypackage. I.e., create a directory called pyinstaller-hooks, and create a hook file called hook-mypackage.py inside it, with the following content:

# pyinstaller-hooks/hook-mypackage.py
from PyInstaller.utils.hooks import collect_submodules, collect_data files 
hiddenimports = collect_submodules('mypackage') 
datas = collect_data_files('mypackage') 
binaries = collect_dynamic_libs('mypackage')

and then pass the additional hooks directory to PyInstaller via --additional-hooks-dir ./pyinstaller-hooks. This is how things were done before --colllect-* command-line "shortcuts" were added; and it works because hooks are ran during Analysis instantiation, so after the package search path has been extended with location of the entry-point script.

2. Use PYTHONPATH environment variable to add the parent directory of mypackage to python's global search path, before running PyInstaller.
3. Edit the generated spec file and append the parent directory of mypackage to sys.path before the collect_* calls are made (within the context of code executed in the .spec file, there is a special variable called SPECPATH that points to the parent directory of the spec; you can use that as an anchor to path to parent directory of mypackage). Then you build your application using the modified .spec file instead of the original .py file (which would re-generate the spec and overwrite the changes).