How do you install a generated openapi client to a zipapp?

Question

So I generated an openapi client using openapi-generator.

And I now want to install it to my zippapp.

There must be something wrong with my overal structure, though.

I have an app folder. It contains my __main__.py file, along with everything else I need.

It's also where I pip install my dependencies into. (Not sure if that's the correct way to go about it, but it worked for other dependencies, so ...)

So after generating the openapi client, I

OUT_ABSOLUTE="<path to generated openapi client>"
APP="${PWD}/app"

sudo chown -R "${USER}" "${OUT_ABSOLUTE}"

pip install -t "${APP}" "${OUT_ABSOLUTE}" --upgrade

this says it succeeds, with

Building wheels for collected packages: openapi-client
  Building wheel for openapi-client (pyproject.toml) ... done
  Created wheel for openapi-client: filename=openapi_client-1.0.0-py3-none-any.whl size=1666754 sha256=f7beef08b8727cad41d0b23f4eb18b523d75e05c953b52d8079870ad0fa9e79b
  Stored in directory: /tmp/pip-ephem-wheel-cache-05c68l5b/wheels/15/66/d5/db16f91fb5af2f414682e9db528a1a63d5611d8afcfafdb1df
Successfully built openapi-client
Installing collected packages: urllib3, typing-extensions, six, annotated-types, python-dateutil, pydantic-core, pydantic, openapi-client
Successfully installed annotated-types-0.7.0 openapi-client-1.0.0 pydantic-2.8.2 pydantic-core-2.20.1 python-dateutil-2.9.0.post0 six-1.16.0 typing-extensions-4.12.2 urllib3-2.0.7

after which I compose and run my app

python -m zipapp app -o my_app.pyz
chmod +x my_app.pyz

python my_app.pyz

This is where things stop going the way I want them to.

After pip installing the openapi client, my app folder looks like this:

As you can see, the pydantic_core._pydantic_core package is present.

And yet, trying to run the app gets python to complain that

Traceback (most recent call last):
  File "/home/user1291/anaconda3/envs/my_app/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user1291/anaconda3/envs/my_app/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/user1291/code/my_app/my_app.pyz/__main__.py", line 5, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/my_app.py", line 5, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/openapi_client/__init__.py", line 20, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/openapi_client/api/__init__.py", line 4, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/openapi_client/api/aa_sequences_api.py", line 15, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/pydantic/__init__.py", line 404, in __getattr__
  File "/home/user1291/anaconda3/envs/my_app/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/user1291/code/my_app/my_app.pyz/pydantic/validate_call_decorator.py", line 8, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/pydantic/_internal/_validate_call.py", line 7, in <module>
  File "/home/user1291/code/my_app/my_app.pyz/pydantic_core/__init__.py", line 6, in <module>
ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'

I appreciate the irony of getting a "not found" error on line 404, but I'd rather python finds its stuff.

How do I get this to work?

Answer 1

There is a caveat with zipapp packaging that you can't (easily) use C extensions. One of your dependencies uses an extension module, we can see that in the filename _pydantic_core.cpython-310-x86_64-linux-gnu.so.

The complications of zipapp packaging, and some workarounds, are mentioned in the docs:

Caveats

If your application depends on a package that includes a C extension, that package cannot be run from a zip file (this is an OS limitation, as executable code must be present in the filesystem for the OS loader to load it). In this case, you can exclude that dependency from the zipfile, and either require your users to have it installed, or ship it alongside your zipfile and add code to your __main__.py to include the directory containing the unzipped module in sys.path. In this case, you will need to make sure to ship appropriate binaries for your target architecture(s) (and potentially pick the correct version to add to sys.path at runtime, based on the user’s machine).

You may also want to consider third-party packaging tools such as shiv or pex, they workaround this problem by creating a self-extracting executable. In this way the limitations of the stdlib zipimporter are avoided.