distutils.core vs setuptools with c++ extension

Question

I am creating a python package with a c++ extension. I am trying to do this with setuptools as that seems to be the preferred solution. Given that it allows for setup_requires, install_requires, I agree and use it for my python only packages. However, I am unable to get it to work when I have a c++ extension module. Then I resort to distutils.core to get it to work. I would like to know how to get it to work using setuptools. My setup script looks like

from setuptools import setup
import shutil
import os

# folder where .so is being build by cpp compilation 
so_src = os.path.join(dir, 'cpp/build/')
# folder where .so should live in python package
so_des = os.path.join(dir, 'package_py/cpp/')

# extension module
lib_files = ['cpp_py.so']

# copy shared lib
for f in lib_files:
  shutil.copyfile(so_src + f, so_des + f)

# set-up script
setup(
      name=DISTNAME
    , version=FULLVERSION
    , description= DESCRIPTION
    , author= AUTHOR
    , author_email= EMAIL
    , maintainer= AUTHOR
    , maintainer_email= EMAIL
    , long_description=LONG_DESCRIPTION
    , packages=['package_py',
            'package_py.cpp']
    , package_dir={'package_py.cpp': 'package_py/cpp'}
    , package_data={'package_py.cpp': lib_files}
    )

This results in a package_py-0.0.1-py3.6.egg file in my python site-packages. The package only works when used from the installation folder.

Changing the first line to use distutils.core instead of setuptools

from distutils.core import setup # only change, remainder is the same!!
import shutil
import os

# folder where .so is being build by cpp compilation 
so_src = os.path.join(dir, 'cpp/build/')
# folder where .so should live in python package
so_des = os.path.join(dir, 'package_py/cpp/')

# extension module
lib_files = ['cpp_py.so']

# copy shared lib
for f in lib_files:
  shutil.copyfile(so_src + f, so_des + f)

# set-up script
setup(
      name=DISTNAME
    , version=FULLVERSION
    , description= DESCRIPTION
    , author= AUTHOR
    , author_email= EMAIL
    , maintainer= AUTHOR
    , maintainer_email= EMAIL
    , long_description=LONG_DESCRIPTION
    , packages=['package_py',
            'package_py.cpp']
    , package_dir={'package_py.cpp': 'package_py/cpp'}
    , package_data={'package_py.cpp': lib_files}
    )

I get a package_py folder (with .so file) and package_py-0.0.1-py3.6.egg-info in site-packages. Now the module works in all folders.

As I would like to extend the python package to also use setup_requires, instal_requires I want to really use setuptools. How can I get the package to work in all folders using setuptools instead of distutils.core

Answer 1

If you're using a C extension, you should use Extension from either setuptools or distutils as well, it's not a python package, so I'm surprised it's even managing to install. If you plan on distributing the package to others, you shouldn't compile the extension beforehand, but have it compiled during the package installation so that the user's system compiles it appropriately (i.e., an .so file doesn't help a Windows user who needs it compiled into a .dll file, etc).

Try something like this:

from setuptools import setup, Extension
import shutil
import os

# I don't know how you want to build your extension or your file structure, 
# so removing the build stuff.

your_modulename=Extension('_extensionname',
                          sources=['path/to/extension.cpp', 'more/file/paths'],
                          language='c'
                         )


# set-up script
setup(
      name=DISTNAME
    , version=FULLVERSION
    , description= DESCRIPTION
    , author= AUTHOR
    , author_email= EMAIL
    , maintainer= AUTHOR
    , maintainer_email= EMAIL
    , long_description=LONG_DESCRIPTION
    , ext_modules=[your_modulename]
    , packages=['package_py']
    )

Hope this helps or at least gets you on the right track.