Python crash when boost/c++ is imported if anaconda was used

Question

I'm trying to build c++ engine interfaces for python, using boost/python. The python import fails in an anaconda environment. I've stripped the c++ down to the following (simplepython.cpp), which still fails:

#include <boost/python/module.hpp>
#include <boost/python/class.hpp>

namespace R3 {
  class Coordinate {
  public:
    Coordinate() : x(0), y(0), z(0) {}
    Coordinate(int _x, int _y, int _z) : x(_x), y(_y), z(_z) {}
    double getX() const { return x;}
    double getY() const { return y;}
    double getZ() const { return z;}
  private:
    double x, y, z;
  };
}

BOOST_PYTHON_MODULE(simplepython) {
  // An established convention for using boost.python.
  using namespace boost::python;
  Py_Initialize();

  class_<R3::Coordinate>("R3Coordinate", init<>())
    .def(init<int, int, int>())
    .def("getX", &R3::Coordinate::getX)
    .def("getY", &R3::Coordinate::getY)
    .def("getZ", &R3::Coordinate::getZ)
  ;
}

trying it:

$> python -c ‘import sys; print(sys.executable); import simplepython

... SystemError: initialization of _heapq did not return an extension module

Sys/build details:

Mac OS Mojave, 10.14.6 Python version 3.7.4, Conda 4.8.3 C++ et al: Apple clang version 11.0.0 (clang-1100.0.33.17)

conda install was: conda install -c anaconda boost

Boost version: boost-1.67.0

Build:

g++ -I/opt/anaconda3/include/python3.7m -I/opt/anaconda3/include/python3.7m -I/opt/anaconda3/include -O2 -fPIC -std=c++11 -Iinclude -c -o objs/simplepython.o src/simplepython.cpp

g++ -Wl,-rpath,/opt/anaconda3/lib -shared -o lib/simplepython.so objs/simplepython.o -L/opt/anaconda3/lib -Llib -lpython3.7m -ldl -framework CoreFoundation -lboost_python37 -lboost_numpy37

Full results:

/opt/anaconda3/bin/python
Error processing line 1 of /opt/anaconda3/lib/python3.7/site-packages/matplotlib-3.1.1-py3.7-nspkg.pth:

Fatal Python error: initsite: Failed to import the site module
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site.py", line 168, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/opt/anaconda3/lib/python3.7/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
  File "/opt/anaconda3/lib/python3.7/contextlib.py", line 5, in <module>
    from collections import deque
  File "/opt/anaconda3/lib/python3.7/collections/__init__.py", line 24, in <module>
    import heapq as _heapq
  File "/opt/anaconda3/lib/python3.7/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site.py", line 579, in <module>
    main()
  File "/opt/anaconda3/lib/python3.7/site.py", line 566, in main
    known_paths = addsitepackages(known_paths)
  File "/opt/anaconda3/lib/python3.7/site.py", line 349, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/opt/anaconda3/lib/python3.7/site.py", line 207, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/opt/anaconda3/lib/python3.7/site.py", line 178, in addpackage
    import traceback
  File "/opt/anaconda3/lib/python3.7/traceback.py", line 3, in <module>
    import collections
  File "/opt/anaconda3/lib/python3.7/collections/__init__.py", line 24, in <module>
    import heapq as _heapq
  File "/opt/anaconda3/lib/python3.7/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

Thanks for any help!

Answer 1

It turns out the error isn't specifically with Py_Initialize(), but with the symbolic linkage between program and python runtime. The following solved it completely:

In your link line, replace -lpython<version>m with -undefined dynamic_lookup.

CMake: The link line is available in /build/mybinary/CMakeFiles/myextension_py.dir/link.txt. For now it seems that the link command has to be manually edited in CMake version 3.17.2.

Credits: Nehal J Wani originally reported this solution, see his answer to this question.

NOTE: I had been using $(shell python3-config --libs) to derive these -l paths, which unfortunately includes -lpython<version>m; and from version 3.8 on it will not. Due to this inconsistency I will simply use my own expanded data for the time being.

Credit and thanks: Nehal J Wani