Search code examples
pythonc++python-3.xembedding

How to handle string for embedding properly in Python 3

I am trying to use python embedding with Python 3 but unfortunately there are some strange outcomes when using it. Here is the source code:

Scenario 1

/tmp/main.cpp

#include <python3.6/Python.h>
//#include <python2.7/Python.h>
using namespace std;

int main()
{
    Py_Initialize();
    PyObject* sysPath = PySys_GetObject("path");
    PyObject *path = PyBytes_FromString("/tmp");
    int result = PyList_Append(sysPath, path);
    PyRun_SimpleString("from hello import hello\n"); // there is a /tmp/hello.py
    Py_Finalize();
    return 0;
}

If I compile and run the above code in Python 2.7 like this, it did not show any errors:

g++ -L/usr/lib/python2.7/config-x86_64-linux-gnu/ -g -o main main.cpp -I/usr/include/python2.7 -lpython2.7

However, if I compile and run the code in Python 3.6 like this:

g++ -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/ -g -o main main.cpp -I/usr/include/ -lpython3.6

It will show the following errors:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/core/__init__.py", line 35, in <module>
    from . import _internal  # for freeze programs
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/core/_internal.py", line 12, in <module>
    from numpy.compat import basestring
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/compat/__init__.py", line 14, in <module>
    from . import py3k
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/compat/py3k.py", line 14, in <module>
    from pathlib import Path
  File "/usr/lib/python3.6/pathlib.py", line 4, in <module>
    import ntpath
  File "/usr/lib/python3.6/ntpath.py", line 278, in <module>
    from nt import _getvolumepathname
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1129, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1268, in find_spec
  File "<frozen importlib._bootstrap_external>", line 60, in _path_join
  File "<frozen importlib._bootstrap_external>", line 60, in <listcomp>
TypeError: a bytes-like object is required, not 'str'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 57, in apport_excepthook
    from cStringIO import StringIO
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1129, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1268, in find_spec
  File "<frozen importlib._bootstrap_external>", line 60, in _path_join
  File "<frozen importlib._bootstrap_external>", line 60, in <listcomp>
TypeError: a bytes-like object is required, not 'str'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/core/__init__.py", line 35, in <module>
    from . import _internal  # for freeze programs
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/core/_internal.py", line 12, in <module>
    from numpy.compat import basestring
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/compat/__init__.py", line 14, in <module>
    from . import py3k
  File "/home/tfidm/.local/lib/python3.6/site-packages/numpy/compat/py3k.py", line 14, in <module>
    from pathlib import Path
  File "/usr/lib/python3.6/pathlib.py", line 4, in <module>
    import ntpath
  File "/usr/lib/python3.6/ntpath.py", line 278, in <module>
    from nt import _getvolumepathname
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1129, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1268, in find_spec
  File "<frozen importlib._bootstrap_external>", line 60, in _path_join
  File "<frozen importlib._bootstrap_external>", line 60, in <listcomp>
TypeError: a bytes-like object is required, not 'str'

Scenario 2

If I remove the three lines to append the current directory to system path like this:

Py_Initialize();
PyRun_SimpleString("from hello import hello\n");

It will show the following error in both Python 2 and Python 3:

ModuleNotFoundError: No module named 'hello'

From Scenario 2, we can observe that the lines to add the current directory to the system path is required. When adding the path is done, the code in scenario 1 works in Python 2 only. The error logs for Python 3 showed that it expects something like byte object rather than string. However I have no clue how to use the embedding with byte object as I cannot find it in the official python website nor any other places on the web.

Please can someone kindly explain how to use embedding properly in Python 3?

Solution

  • Python 3 sys.path takes Unicode strings, not byte strings. Don't use PyBytes_FromString to create the path element.

    Use PyUnicode_FromString instead, at least when embedding Python 3.

    If the path elements contain just ASCII characters, you can use the same function in Python 2. Non-ASCII character support in sys.path elements in Python 2.x is spotty and depends on the OS.