Error exporting symbol when building Python C Extension in Windows

Question

I'm working on porting a Python module to Windows. I have a toy example as follows.

The folder structure is:

foo/
  libfoo/
    foo.c
  setup.py

setup.py

from setuptools import setup, Extension

sources = ['libfoo/foo.c']

foo = Extension('libfoo',
                 sources = sources,
                 define_macros = None,
                 include_dirs = ['./libfoo'],
                 libraries = None,
                 library_dirs = None,
                 )

setup(name          = 'foo',
      ext_modules      = [foo],
      install_requires = ['setuptools'],
)

libfoo/foo.c (for completeness)

#include <stdio.h>

void foo() {
  printf("Hello World!");
}

When I attempt to install the package, I encounter an error.

C:\Users\user\foo>python setup.py install
running install
running bdist_egg
running egg_info
creating foo.egg-info
writing requirements to foo.egg-info\requires.txt
writing foo.egg-info\PKG-INFO
writing top-level names to foo.egg-info\top_level.txt
writing dependency_links to foo.egg-info\dependency_links.txt
writing manifest file 'foo.egg-info\SOURCES.txt'
reading manifest file 'foo.egg-info\SOURCES.txt'
writing manifest file 'foo.egg-info\SOURCES.txt'
installing library code to build\bdist.win32\egg
running install_lib
running build_ext
building 'libfoo' extension
creating build
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
creating build\temp.win32-2.7\Release\libfoo
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -I./libfoo -IC:\Python27\include -IC:\Python27\PC /Tclibfo
o/foo.c /Fobuild\temp.win32-2.7\Release\libfoo/foo.obj
foo.c
creating build\lib.win32-2.7
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo
/INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:i
nitlibfoo build\temp.win32-2.7\Release\libfoo/foo.obj /OUT:build\lib.win32-2.7\l
ibfoo.pyd /IMPLIB:build\temp.win32-2.7\Release\libfoo\libfoo.lib /MANIFESTFILE:b
uild\temp.win32-2.7\Release\libfoo\libfoo.pyd.manifest
LINK : error LNK2001: unresolved external symbol initlibfoo
build\temp.win32-2.7\Release\libfoo\libfoo.lib : fatal error LNK1120: 1 unresolv
ed externals
error: command 'c:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\l
ink.exe' failed with exit status 1120

It seems that the distutils package (in this case setuputils) will always export one symbol from a shared extension and that is "init + extension_name" [Link].

It is specified in the Windows Linker "EXPORT" option [Link] but it can't find the symbol.

Help?

EDIT: The C code does not use the Python C API i.e. "#include ". This is because the goal of the project is to take an existing C library and encase in a Python wrapper via a Python extension. The package works on Unix/Linux.

Answer 1

After lots of playing around, I was able to understand the issue.

Solution: in Windows, you must define an initialization function for your extension because setuptools will build a .pyd file, not a DLL

To resolve this in Python 2, you define the initlibfoo() method in the source.

#include <Python.h>

PyMODINIT_FUNC initlibfoo(void) {
    // do stuff...
}

To resolve this in Python 3, you define the PyInit_libfoo() method in the source.

#include <Python.h>

PyMODINIT_FUNC PyInit_libfoo(void) {
    // do stuff...
}

so now, foo.c will look like:

#include <stdio.h>
#include <Python.h>

void foo() {
  printf("Hello World!");
}

PyMODINIT_FUNC initlibfoo(void) // Python 2.7
//PyMODINIT_FUNC PyInit_libfoo(void) // Python 3.5
{
    // do stuff...
}

Explanation

When compiling Python extensions for Windows, the Extension class will tell the linker to export a function for other programs to call. That can be seen in the terminal output:

c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo
/INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:i
nitlibfoo build\temp.win32-2.7\Release\libfoo/foo.obj /OUT:build\lib.win32-2.7\l
ibfoo.pyd /IMPLIB:build\temp.win32-2.7\Release\libfoo\libfoo.lib /MANIFESTFILE:b
uild\temp.win32-2.7\Release\libfoo\libfoo.pyd.manifest

(emphasis on /EXPORT:initlibfoo)

Now this is a required function for C/C++ Extensions so that they can be imported as modules. In other words, when you import libfoo Python will call the function initlibfoo on the .pyd as part of the initialization of the module. The error encountered here is due to the fact that the linker is told to export the function initlibfoo however it can't find the symbol in the C object files because it was never defined in the source!

Surely, one should be able to decline the option of export a symbol right? Apparently not. Referring back to the Extension class documentation, there's an argument export_symbols. I tried passing None to it and the linker option was still being used. It seems there's no way to circumvent this linker option.

NOT RECOMMENDED: if for some reason, needing to have Python.h bothers you, there's a way around this. You still need to define the "init" methods above but you can do:

void initlibfoo() {} //Python 2.7
void PyInit_libfoo() {} //Python 3.5

But now, you can't use the library via import libfoo. You could use the ctypes module and load it with ctypes.PyDLL('/path/to/pyd'). Essentially, you used Python to build a DLL for you, in which case, it does but it builds a special DLL known as a .pyd file. Then to use it, you must load it in via ctypes module.