Bundling python app compiled with cython with pyinstaller

Question

Problem

I have an application which is bundled with pyinstaller. Now a new feature request is, that parts are compiled with cyphon to c libraries.

After the compilation inside the activated virtual environment (poetry) the app runs as expected.

BUT, when I bundle it with pyinstaller the executable afterwards can't find packages which are not imported in the main.py file. With my understanding, this is totally fine, because the Analysis stage of the pyinstaller can't read the conntent of the compiled c code ( In the following example modules/test/test.py which is available for the pyinstaller as modules/test/test.cpython-311-x86_64-linux-gnu.so).

Folder overview:

├── compile_with_cython.py
├── main.py
├── main.spec
├── main_window.py
├── poetry.lock
└── pyproject.toml

main.py

import sys
from PySide6.QtWidgets import QApplication
from main_window import MainWindow

if __name__ == '__main__':
    app = QApplication(sys.argv)
    mainWin = MainWindow()
    mainWin.show()
    sys.exit(app.exec_())

main_window.py

MVP PySide6 Application which uses tomllib to load some toml file

import sys
from PySide6.QtWidgets import QApplication, QMainWindow, QPushButton, QDialog, QVBoxLayout, QTextEdit
from PySide6.QtCore import Slot

class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        ... 

Error code

./main
Traceback (most recent call last):
  File "main.py", line 12, in <module>
  File "modules/test/test.py", line 3, in init modules.test.test
ModuleNotFoundError: No module named 'tomllib'
[174092] Failed to execute script 'main' due to unhandled exception!

Answer 1

Problem

The main problem pyinstaller faces is that it can't follow imports of files/modules compiled by cython. Therefore, it can only resolve and package files & libraries named in main.py, but not in main_window.py. To make it work, we need to specify all imports that are hidden from pyinstaller.

I have found two suitable solutions for using pyinstaller with cython compiled binaries.

Solution 1:

Add any import needed by any script to the main python file, e.g:

# imports needed by the main.py file
import argparse
import logging
import sys
import time

# dummy imports (needed by the main_window.py file)

import tomllib
import pydantic

This will work, but is only suitable for small projects. Moreover the stated imports will be deleted by various linters because the imports are not really used by this file...

Solution 2

I found the following in the pyinstaller documentation, to get it to work I changed my `.spec' file as follows:

a = Analysis(
    ['main.py'],
    pathex=[],
    binaries=[],
    datas=[],
    hiddenimports=['tomllib', 'pydantic'], 

Bonus

Since the code above was clearly just an example, and I had a project with hundreds of Python files and libraries, I came up with the following code to automatically generate the contents of the `hiddenimports' variable each time the pipeline builds the package:

def find_all_hidden_imports(directory_path: Path) -> set:
    imports_set = set()
    for file_path in directory_path.rglob('*.py'):
        if ".venv" not in str(file_path):
            imports_set.update(get_imports_of_file(file_path))
    return imports_set


def get_imports_of_file(file_path: Path) -> set:
    imports_set = set()

    with open(file_path, 'r', encoding='utf-8') as file:
        content = file.read()
        try:
            tree = ast.parse(content)
            for node in ast.walk(tree):
                if isinstance(node, ast.Import):
                    for name in node.names:
                        imports_set.add(name.name)
                elif isinstance(node, ast.ImportFrom):
                    if node.module is not None:
                        imports_set.add(node.module)
        except SyntaxError:
            print(f"Syntax error in file: {file_path}")

    return imports_set

This set is then converted to the correct list format string and this string is then inserted into the current .spec file...