Search code examples
pythonpython-c-api

Failed to load c-extensions in a shiv package


I maintain a project which I bundle into a shiv package and distribute at work. I've included all necessary package dependencies into the shiv package, and this includes numpy and PIL. It is safe to assume everyone is using Windows and python 3.6+.

It seems that the C-extensions for numpy and PIL do not come along with the rest of the packages because if I give the pyz to someone who already has the same versions of these two packages in their environment as the ones the zipapp was built with it works as intended, but if they have any other version it fails to load those extensions but does load all of the other packages from the zipapp as expected. I can also exclude numpy and PIL from the zipped site-packages but this defeats the purpose of the zipapp because it relies on the user's environment which I can't trust to be updated.

I am building my shiv package with shiv --compressed --compile-pyc --site-packages "path_to_site-packages" -o my_output.pyz -e the_package:__run_main

From reading the shiv documentation I thought that --compile-pyc was the missing key but using this or not returned the same result. Any recommendations would be greatly appreciated

It seems that this might be related to this unanswered question Python - Pyinstaller Numpy C-extensions error


Solution

  • At long last I've found the cause to my issue. I am using Anaconda to manage my python environments and when creating the shiv package "path_to_site-packages" folder was the one that Anaconda creates. I can not explain exactly why this was the issue, but my assumption is that Anaconda manages where the C-libraries are stored differently than pip.

    The resolution to my problem came when I made my own dedicated site-packages folder for the shiv installation by pip installing everything I needed with --target to that folder. Once I did that, building the shiv package from that folder fully resolved the missing C-library problem. I hope this helps someone!