Python 3: setup.py: pip install that does everything (build_ext + install)

Question

I'm learning how to use distutils, and there's something I don't understand, and I wish someone could explain this to me.

I already am successful in creating tar.gz packages that can be installed with

pip install mypackage.tar.gz

I did this with setup.py, with a simple script that runs the function setuptools.setup(), which I call using python3 setup.py sdist.

What I wanna learn now: How to include building extensions in this.

My problem: I couldn't find any comprehensive text that explains how a pip install of a package that has a build_ext class can get it to build, then install.

If we look at this example, for the famous tool cx_freeze package, we see:

• There's an inherited build_ext class
• There's a method build_extension()
• in setup(), the variable cmdclass, which contains a dict that contains the class build_ext

My question: What gets cx_freeze to build the extension and then install it? Is having a cmdclass defined with build_ext enough to achieve this?

Answer 1

After many tests, I learned that this is related to how pip works, not how setup.py works. It turns out that after writing your setup.py file and using pip to install, this is what happens:

1. pip creates a temporary file, in Linux it's in /tmp, and on Windows it's in the temp dir of the user.
2. pip downloads/extracts the package to that temp directory (whether from a tar.gz or from an online source or from a repository)
3. pip runs the following operations in order:
   • setup.py install
   • setup.py build
   • setup.py install_lib
   • setup.py build_py
   • setup.py build_ext

And all this depends on whether you have stuff defined in the cmdclass parameter of setup(). For example, build_ext will run only if you have build_ext defined in cmdclass AND you have ext_modules defined in the parameters of your setup() call. So ext_modules, for example, is expected to be a list of Extension(), which contains all the information about every extension. The function in the class build_extension(self,ext) will be executed on every element of that list.

All these classes that go to cmdclass (I use build and build_py) have a method called run() that you should override to put in your custom building procedure.

After all that is done, pip installs (or copies) the packages defined in setup() (which are basically directories in that temp) to your Python directory, which is the end of the installation, and deletes temp files.

There's more details to all this, but I guess this is good for a starter. I saw none of this explained anywhere comprehensively. So I hope this helps and saves people the empirical testing I had to do to learn this.