Package C source files along with a Python package

Question

I have a C library I am writing, and my goal is to be able to package and distribute that C library via a python package on PyPI. The concept is, it's a combination of Python code, an extension module, and a C library. I want the user to be able to install and build the C library at runtime via specific Python functions using distutils.ccompiler, after they install the package. However, I am running into an issue where even though the C source files are listed in MANIFEST.in, they do not appear when I run the python setup.py build command.

Here is a layout of my directory

home -
     setup.py
     MANIFEST.in
     package -
             ctools  -
                     __init__.py
             src     -
                     some_funcs.c
             include -
                     some_funcs.h
     pyext -
            pymain.c

My MANIFEST.in file looks like this

recursive-include package *.c *.h
include LICENSE.txt
include README.md

my distutils setup looks like

macro_defs = []

if os.name == 'nt':
  macro_defs.append(('_CRT_SECURE_NO_WARNINGS', '1'))

# This goes under the main package
# Will be linked with the C library later on 
core_module = distutils.core.Extension('package.core',
                    define_macros = macro_defs,
                    include_dirs = ['include'],
                    sources = ['pyext/pymain.c'])

distutils.core.setup(name='package',
      version='0.0.1',
      description='A library for searching and analyzing  data',
      author='Me',
      author_email='me@home.come',
      url='https://github.com/some/repo',
      download_url='https://github.com/some/repo/archive/master.zip',
      license = 'MIT',
      keywords = keyword_list,
      classifiers = classifers_list,
      long_description = open('README.md').read(),
      packages=['package', 'package.ctools'],
      ext_modules=[core_module],
     )

The python setup.py sdist works fine as intended, but when I run the build command, it doesn't copy over the files under package/src or package/include.

When the user installs my package from pip, I want those C source and header files to be embedded in their installed python package. How can I get this to work?

To be clear, I would like the result of my build command to create a layout identical to my source package.

Answer 1

This can be accomplished with the data_files argument of the distutils.core.setup function. This keyword argument takes a list of tuples, where the first argument of each tuple is the desired partial path of the installed location, and the second argument is a last of paths to the files desired to be installed under the first element's directory name. Despite the name data_files, one can use it to install any files that are not involved in the build process of the python package being distributed.

For example, a use of data_files might look like

from distutils.core import setup

setup(
   ...,
   data_files=[("csrc", ["src/main.c", 
                         "src/helper.c"]),
               ("cinclude", ["include/helper.h"])]
)

directories and files specified under data_files will be installed under the sys.prefix or under the site.USER_BASE locations. Such as in the case of windows, those might look like the following

>>> import sys
>>> sys.prefix
'C:\\Users\\foobar\\AppData\\Local\\Programs\\Python\\Python37'
>>> import site
>>> site.USER_BASE
'C:\\Users\\foobar\\AppData\\Roaming\\Python'

Which of the two the data_files get installed at depends on if the --user option is specified when using pip.