Search code examples
pythonsetuptoolseggwafbuild-tools

How can I make a Python extension module packaged as an egg loadable without installing it?

I'm in the middle of reworking our build scripts to be based upon the wonderful Waf tool (I did use SCons for ages but its just way too slow).

Anyway, I've hit the following situation and I cannot find a resolution to it:

  • I have a product that depends on a number of previously built egg files.
  • I'm trying to package the product using PyInstaller as part of the build process.
  • I build the dependencies first.
  • Next I want to run PyInstaller to package the product that depends on the eggs I built. I need PyInstaller to be able to load those egg files as part of it's packaging process.

This sounds easy: you work out what PYTHONPATH should be, construct a copy of sys.environ setting the variable up correctly, and then invoke the PyInstaller script using subprocess.Popen passing the previously configured environment as the env argument.

The problem is that setting PYTHONPATH alone does not seem to be enough if the eggs you are adding are extension modules that are packaged as zipsafe. In this case, it turns out that the embedded libraries are not able to be imported.

If I unzip the eggs (renaming the directories to .egg), I can import them with no further settings but this is not what I want in this case.

I can also get the eggs to import from a subshell by doing the following:

  • Setting PYTHONPATH to the directory that contains the egg you want to import (not the path of the egg itself)
  • Loading a python shell and using pkg_resources.require to locate the egg.

Once this has been done, the egg loads as normal. Again, this is not practical because I need to be able to run my python shell in a manner where it is ready to import these eggs from the off.

The dirty option would be to output a wrapper script that took the above actions before calling the real target script but this seems like the wrong thing to do: there must be a better way to do this.

Solution

  • Heh, I think this was my bad. The issue appear to have been that the zipsafe flag in setup.py for the extension package was set to False, which appears to affect your ability to treat it as such at all.

    Now that I've set that to True I can import the egg files, simply by adding each one to the PYTHONPATH.

    I hope someone else finds this answer useful one day!