Search code examples
pythonc++linuxpython-3.4python-c-api

Embedding Python in Linux program


I am currently trying to embed Python in my C++ application to give the user advanced scripting possibilities. My program is working great on Windows so far (it is fully working), and now I'm trying to do the same on GNU/Linux (Debian 7 for now) but this is giving me much more trouble than I expected. First, I download the python.tar.gz and compile it from source with enable-shared option to get the fPIC option:
./configure --enable-shared --prefix=/opt/python
make && make altinstall
Then, I install numpy thanks to pip : python3.4 -m pip install numpy. Easy.

Last, I copy the installation to another location (yes, it should be deployed anywhere), in my home directory to be precise, and name it python_install. This copy seems to give me much pain.

On the code side, I call Py_Initialize which is working good. Here is what I have done so far :

#include "Python.h"

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "numpy/arrayobject.h"

#include <iostream>

int InitNumpy()
{
    import_array();
}

int main()
{
    std::string python_home = "/home/xxxx/dev/test-python/python_install";
    setenv("PYTHONHOME", python_home.c_str(),1 );

    Py_Initialize();

    std::cout << "Importing Numpy... ";
    int cr = InitNumpy();
    std::cout << cr << std::endl;

    return 0;
}

ImportError: numpy.core.multiarray failed to import
I think the error is pretty clear, Python can't find any libraries including numpy. But I have tried everything from setting PYTHONPATH to playing with -Wl,-rpath to set additional directories.... even PySys_SetPath... What is working on Windows is failing here on Linux. Any ideas would be welcome ! Thanks.

EDIT : Here is the makefile I used (corrected):

CC=g++
CFLAGS= -Ipython_install/include/python3.4m -Ipython_install/lib/python3.4/site-packages/numpy/core/include -Wno-unused-result  -DNDEBUG -g -fwrapv -O3 -Wall
#Wrong
#LDFLAGS= -Lpython_install/lib/python3.4/config-3.4m -lpython3.4m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-rpath,\$${ORIGIN}/python_install/lib
#Right
LDFLAGS= -Lpython_install/lib/ -lpython3.4m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-rpath,\$${ORIGIN}/python_install/lib
EXEC=test-python

all: $(EXEC)

test-python: test-python.o
    $(CC) -o $@ $^ $(LDFLAGS)

%.o: %.cpp
    $(CC) -o $@ -c $< $(CFLAGS)

.PHONY: clean mrproper

clean:
    rm -rf *.o

mrproper: clean
    rm -rf $(EXEC)

Options are coming directly from the python3.4m-config executable...


Solution

  • The short answer is : don't link statically on Python. All Python modules link dynamically on libpython3.4.so, so the C++ program has to do the same. To sum up Python embedding :

    1. Link dynamically on Python (cf corrected Makefile). Use rpath to locate the python .so.
    2. In the C++ code, set PYTHONHOME environment variable to the Python installation or use Py_SetPythonHome() function.
    3. Call Py_Initialize();
    4. If you want to import custom module, add the module path to Python path.

      PyObject *sys = PyImport_ImportModule("sys");
      PyObject *path = PyObject_GetAttrString(sys, "path");
      PyList_Append(path, PyUnicode_FromString(sys_path.toUtf8().data()));