py2exe: how to include additional resource files from dependancy

I'm trying to build a project that has a dependency on jsonschema. On building and running the project with py2exe I get this error at runtime:

INFO:runtime:Analyzing the code
INFO:runtime:Found 527 modules, 27 are missing, 0 may be missing

  27 missing Modules
  ------------------
? __main__                            imported from bdb, pdb
? _frozen_importlib                   imported from importlib, importlib.abc, zipimport
? _frozen_importlib_external          imported from importlib, importlib._bootstrap, importlib.abc, zipimport
? _posixshmem                         imported from multiprocessing.resource_tracker, multiprocessing.shared_memory
? _winreg                             imported from platform
? annotationlib                       imported from attr._compat
? asyncio.DefaultEventLoopPolicy      imported from -
? dummy.Process                       imported from multiprocessing.pool
? fqdn                                imported from jsonschema._format
? idna                                imported from jsonschema._format
? importlib_metadata                  imported from attr, jsonschema
? importlib_resources                 imported from jsonschema._utils
? isoduration                         imported from jsonschema._format
? java.lang                           imported from platform
? jsonpointer                         imported from jsonschema._format
? org.python.core                     imported from copy, pickle
? os.path                             imported from ctypes._aix, distutils.file_util, os, pkgutil, py_compile, sysconfig, tracemalloc, unittest, unittest.util
? pep517                              imported from importlib.metadata
? readline                            imported from cmd, code, pdb
? requests                            imported from jsonschema.validators
? resource                            imported from test.support
? rfc3339_validator                   imported from jsonschema._format
? rfc3986_validator                   imported from jsonschema._format
? rfc3987                             imported from jsonschema._format
? typing_extensions                   imported from attr._compat, jsonschema.protocols
? uri_template                        imported from jsonschema._format
? webcolors                           imported from jsonschema._format
Building 'C:\Users\James\Development\Wave-Venture\py2exe_playground\build\Example.exe'.

On running the .exe:

Traceback (most recent call last):
  File "__main__.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "jsonschema\__init__.pyc", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "jsonschema\protocols.pyc", line 33, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "jsonschema\validators.pyc", line 388, in <module>
  File "jsonschema\_utils.pyc", line 61, in load_schema
  File "zipfile.pyc", line 2327, in read_text
  File "zipfile.pyc", line 2315, in open
  File "zipfile.pyc", line 1511, in open
  File "zipfile.pyc", line 1438, in getinfo
KeyError: "There is no item named 'jsonschema/schemas/draft3.json' in the archive"

If I look in the jsonschema package in the Pythons site-packages, I can see these files

How can I configure py2exe to see these and bundle them? I found this answer on stack overflow, though it appears to no longer work and I cannot find the equivalent hook in the latest versions of py2exe.

I'll include a minimal example project that produces this error below.

py2exe_playground.zip

Thanks for any help.

Solution

Update

Thanks to @freebie's own advice!

Using monkeypatch can avoid of modifying py2exe's source code in the first step of the solution.

Below is the modified build.py from py2exe_playground:

...

from _pytest import monkeypatch

def hook_jsonschema(finder, module):
    import jsonschema
    jsonschema_dir = os.path.dirname(jsonschema.__file__)
    files = glob.glob(f"{jsonschema_dir}\\schemas/*.json", recursive=True)
    finder.add_datadirectory("jsonschema\\schemas", f"{jsonschema_dir}\\schemas", recursive=True)
    for f in files:
        finder.add_datafile_to_zip(f"jsonschema\\schemas\\{os.path.basename(f)}", f)

py2exe.hooks.hook_jsonschema = hook_jsonschema

...

py2exe.freeze(
    console=[],
    windows=[
        {
            "dest_base": NAME,
            "script": "app/main.py",
        },
    ],
    options={
        "dist_dir": DEST,
        "includes": [],
        "packages": ["jsonschema"]   # Edit here
    },
    data_files=None,
    version_info={
        "project_name": NAME,
        "version": META["tool"]["poetry"]["version"],
        "description": META["tool"]["poetry"]["description"],
    },
)

The main issue is that the data_file specified by py2exe is not being packaged into library.zip, as mentioned in this post. Although modifying the original Python package code is not ideal, since the goal is just to create a runnable exe, you might consider the following approach of modifying the source code by adding your own hook_jsonschema.

Here's the steps

Step 1: Add your own `hook_jsonschema` into py2exe

Use this function to add the missing file to the dependencies. If successfully added, you will see it in build/library.zip. Note that the hook function name must be hook_<package-name>.

hooks.py located under the path of the py2exe package in your virtual environment.

e.g. D:\StackOverFlow\78901337-py2exe-how-to-include-additional-resource-files-from-dependancy\py2exe_playground\.venv\Lib\site-packages\py2exe\hooks.py

hooks.py

def hook_jsonschema(finder, module):
    import jsonschema
    # Find package path and files in your env
    jsonschema_dir = os.path.dirname(jsonschema.__file__)
    files = glob.glob(f"{jsonschema_dir}\\schemas/*.json", recursive=True)
    
    # Add Path
    finder.add_datadirectory("jsonschema\\schemas", f"{jsonschema_dir}\\schemas", recursive=True)
    
    # Add needed files
    for f in files:
        finder.add_datafile_to_zip(f"jsonschema\\schemas\\{os.path.basename(f)}", f)

Step 2: Add `options: {"packages": ["jsonschema"]}`

Here is the build.py in your py2exe_playground.zip

build.py

py2exe.freeze(
    console=[],
    windows=[
        {
            "dest_base": NAME,
            "script": "app/main.py",
        },
    ],
    options={
        "dist_dir": DEST,
        "includes": [],
        "packages": ["jsonschema"]   # Edit here
    },
    data_files=None,
    version_info={
        "project_name": NAME,
        "version": META["tool"]["poetry"]["version"],
        "description": META["tool"]["poetry"]["description"],
    },
)

After above modifying, run

$poetry run python build.py

It's then successfully packaging Example.exe without error.log.

py2exe: how to include additional resource files from dependancy

Step 1: Add your own hook_jsonschema into py2exe

Step 2: Add options: {"packages": ["jsonschema"]}

Step 1: Add your own `hook_jsonschema` into py2exe

Step 2: Add `options: {"packages": ["jsonschema"]}`