Search code examples
pythondistutils

What is the relationship between setup(name) and Extension(name)

I am writing (actually generating) a setup.py script for building a single Python extension with several C++ sources. It is unclear to me what is the relationship between the name specified as the name parameter to distutils.core.setup and the name specified as the name parameter to distutils.core.Extension. So when I have this:

distutils.core.setup(
  name = 'Abc',
  ext_modules = [
    distutils.core.Extension(
      name = 'Xyz',
      sources = ['a.cpp', 'b.cpp']
    )
  ]
)

What is the relationship between Abc and Xyz, especially with respect to:

  • Each other
  • The name which will be used to import the extension in Python scripts which use it
  • The name of the generated .pyd (or .so) file

I read this in the documentation of distutils:

  • Abc is "The name of the package"
  • Xyz is "the full name of the extension, including any packages — ie. not a filename or pathname, but Python dotted name"

Unfortunately, I am not able to decipher my answer from this (possibly because Python is not my primary language and I only use it occasionally).

In case it's relevant, the context is that I am creating a CMake wrapper to incorporate building Python extensions into my project's CMake framework.

Solution

  • The biggest source of misunderstanding is that the word "package" is heavily overloaded. There are 4 different names in the game — the name of the directory being used for development (the one that contains setup.py), the name of the directory contained __init__.py and other importable modules, the name of distribution at PyPI, and the name of the extension. Quite often the first 3 are the same or similar but that's not required.

    The name of the development directory can be any, its name doesn't play any role. Of course it's convenient to name it properly but that's only convenience.

    The name of the directory with Python files name the package to be imported. Once the package is named for import the name usually stuck and cannot be changed.

    The name of the distribution gives one a page at PyPI and the name of distribution files (source distribution, eggs, wheels). It's the name one puts in setup(name='distribution') call.

    Extensions are python modules so the name of an extension must be proper module names. But the name one puts in Extension() must be full python path to the module including the top-level package name. In your example it's

    Extension(
  name = 'Abc.Xyz',
  sources = ['a.cpp', 'b.cpp']
)

    This produces an extension that can be imported as

      import Abc.Xyz

    or

      from Abc import Xyz

    or

      from Abc.Xyz import <internal name>

    Let me show detailed real example. I've been maintaining a templating library called CheetahTemplate. I develop it in the development directory called cheetah3/. The distribution at PyPI is called Cheetah3; this is the name I put into setup(name='Cheetah3'). The top-level module is Cheetah hence one does import Cheetah.Template or from Cheetah import Template; that means that I have a directory cheetah3/Cheetah/.

    The library has an extension _namemapper. The name used in Extension() is Cheetah._namemapper.