I am trying to build a C++ static library in Linux (Ubuntu 18.04 in my case) using GCC using a Makefile. I noticed the issue is not with the makefile itself but the way I'm trying to compile and build with GCC. Before I explain a bit more on the GCC side, here is how my current project hierarchy looks like.
The project uses Pybind11 header only library which resides in the External_Libraries directory.
My class definition and implementation, naming Core.h and Core.cpp reside in Internal_libraries along with py_utilities.h and any.hpp which contain some utility functions. (These two files are not used though). and finally test_lib.cpp which uses Core.h resides in the root.
This is how it looks :
📦MainDirectory
┣ 📂External_Libraries
┃ ┗ 📂Pybind11
┃ ┗ 📂Pybind11
┃ ┗ 📂Include
┃
┣ 📂Internal_Libraries
┃ ┣📜Core.h
┃ ┣📜Core.cpp
┃ ┣📜py_utilities.h
┃ ┣📜any.hpp
┃
┗📜test_lib.cpp
Now, the project as you can see, depends on Pybind11 include directories and also <Python.h> plus pythons lib directories. I face the same undefined errors both when I try to build the static library or the normal executable. I include the needed search paths (include and library) to the GCC like this :
Core.o:g++ -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages -c ./Internal_Libraries/Core.cpp -o Core.o
test_lib.o:g++ -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages -c test_lib.cpp -o test_lib.o
and finally :
test_lib executable:g++ -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages test_lib.o Core.o -o test_lib
This is where it fails and I get lots of undefined errors which I guess is related to the linker, however I provided the needed paths, so I'm at a loss what's gone wrong here!
These are the errors I get:
g++ -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages test_lib.o Core.o -o test_lib
test_lib.o: In function `pybind11::cast_error::set_error() const':
test_lib.cpp:(.text._ZNK8pybind1110cast_error9set_errorEv[_ZNK8pybind1110cast_error9set_errorEv]+0x29): undefined reference to `PyExc_RuntimeError'
test_lib.cpp:(.text._ZNK8pybind1110cast_error9set_errorEv[_ZNK8pybind1110cast_error9set_errorEv]+0x34): undefined reference to `PyErr_SetString'
test_lib.o: In function `pybind11::error_scope::error_scope()':
test_lib.cpp:(.text._ZN8pybind1111error_scopeC2Ev[_ZN8pybind1111error_scopeC5Ev]+0x27): undefined reference to `PyErr_Fetch'
test_lib.o: In function `pybind11::error_scope::~error_scope()':
...
Core.o: In function `std::enable_if<((!std::is_floating_point<int>::value)&&std::is_signed<int>::value)&&((sizeof (int))<=(sizeof (long))), pybind11::handle>::type pybind11::detail::type_caster<int, void>::cast<int>(int, pybind11::return_value_policy, pybind11::handle)':
Core.cpp:(.text._ZN8pybind116detail11type_casterIivE4castIiEENSt9enable_ifIXaaaantsrSt17is_floating_pointIT_E5valuesrSt9is_signedIS6_E5valuelestS6_stlENS_6handleEE4typeES6_NS_19return_value_policyESA_[_ZN8pybind116detail11type_casterIivE4castIiEENSt9enable_ifIXaaaantsrSt17is_floating_pointIT_E5valuesrSt9is_signedIS6_E5valuelestS6_stlENS_6handleEE4typeES6_NS_19return_value_policyESA_]+0x2c): undefined reference to `PyLong_FromSsize_t'
Core.o: In function `pybind11::array_t<unsigned char, 16>::ensure(pybind11::handle)':
Core.cpp:(.text._ZN8pybind117array_tIhLi16EE6ensureENS_6handleE[_ZN8pybind117array_tIhLi16EE6ensureENS_6handleE]+0x70): undefined reference to `PyErr_Clear'
Core.o: In function `pybind11::array_t<unsigned char, 16>::raw_array_t(_object*)':
Core.cpp:(.text._ZN8pybind117array_tIhLi16EE11raw_array_tEP7_object[_ZN8pybind117array_tIhLi16EE11raw_array_tEP7_object]+0x26): undefined reference to `PyExc_ValueError'
Core.cpp:(.text._ZN8pybind117array_tIhLi16EE11raw_array_tEP7_object[_ZN8pybind117array_tIhLi16EE11raw_array_tEP7_object]+0x35): undefined reference to `PyErr_SetString'
collect2: error: ld returned 1 exit status
here is the link to the full error log : error log
In case the contents of Core.h is important here it is :
#ifndef CORE_H
#define CORE_H
/* If we are we on Windows, we want a single define for it.*/
#if !defined(_WIN32) && (defined(__WIN32__) || defined(WIN32) || defined(__MINGW32__))
#define _WIN32
#endif /* _WIN32 */
#if defined(_WIN32) && defined(_CORE_BUILD_DLL)
/* We are building CORE as a Win32 DLL */
#define CORE_API __declspec(dllexport)
#elif defined(_WIN32) && defined(CORE_DLL)
/* We are calling CORE as a Win32 DLL */
#define CORE_API __declspec(dllimport)
#elif defined(__GNUC__) && defined(_CORE_BUILD_DLL)
/* We are building CORE as a shared / dynamic library */
#define CORE_API __attribute__((visibility("default")))
#else
/* We are building or calling CORE as a static library */
#define CORE_API
#endif
#include <iostream>
#include <string>
#include <vector>
#include <functional>
#include <memory>
#include <pybind11/pybind11.h>
#include <pybind11/embed.h>
#include <pybind11/numpy.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>
namespace py = pybind11;
using namespace py::literals;
typedef void(*CallbackFn)(bool, std::string, py::array_t<uint8_t>&);
typedef std::function<void(std::string)> LogFunction;
typedef void * HANDLE;
class CORE_API Core
{
private:
py::scoped_interpreter guard{};
py::object serviceUtilsModule;
py::object cls;
py::object obj;
py::object startFunc;
...
public:
Core();
Core(bool showFeed);
Core(LogFunction logInfo);
Core(LogFunction logInfo, LogFunction logWarning);
Core(LogFunction logInfo, LogFunction logWarning, LogFunction logDebug);
Core(LogFunction logInfo, LogFunction logWarning, LogFunction logDebug, std::vector<CallbackFn> callbackList);
...
~Core();
void Start(bool async= false);
py::list GetCallbacks(void);
...
static void DefaultLoger(std::string str);
};
extern "C"
{
//COREAPI wchar_t** GetResults(wchar_t* word, int* length, int threshold = 9);
CORE_API void Start(bool);
CORE_API void Stop(void);
CORE_API void SetCpuAffinity(int mask);
CORE_API void AddCallback(CallbackFn callback);
CORE_API void RemoveCallback(CallbackFn callback);
CORE_API void* GetCallbacks(void);
CORE_API void DefaultLoger(char* str);
CORE_API void* CreateHandle();
CORE_API void* GetCurrentHandle();
CORE_API void DisposeCurrentHandle();
CORE_API void SetCurrentHandle(void* handle);
CORE_API void* GetHandle();
CORE_API void DisposeHandle(void*);
}
#endif // !CORE_H
The project works fine under Windows and Visual C++ v14 (VS2015) and now I'm trying to do the same thing in Linux using GCC. It seems I'm doing something wrong obviously but I can't seem to know where I have gone wrong. What am I missing here?
After following this link on manually compile pybind11, I ended up doing :
g++ -O3 -Wall -std=c++11 -fPIC -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages -c ./Internal_Libraries/Core.cpp -o Core.o
g++ -O3 -Wall -std=c++11 -fPIC -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages -c test_lib.cpp -o test_lib.o
g++ -O3 -Wall -std=c++11 -fPIC -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages test_lib.o Core.o -o test_lib
Please note that, I removed the shared keyword from the commands above so it doesn't create the shared library! Having said that the last command still gives linker errors:
g++ -O3 -Wall -std=c++11 -fPIC -I./External_Libraries/pybind11/pybind11/include -I/home/rika/anaconda3/include/python3.7m -L/home/rika/anaconda3/lib -L/home/rika/anaconda3/lib/python3.7/site-packages test_lib.o -o test_lib
test_lib.o: In function `pybind11_static_get':
test_lib.cpp:(.text.pybind11_static_get[pybind11_static_get]+0x3): undefined reference to `PyProperty_Type'
test_lib.o: In function `pybind11_static_set':
test_lib.cpp:(.text.pybind11_static_set[pybind11_static_set]+0x12): undefined reference to `PyProperty_Type'
test_lib.o: In function `pybind11::cast_error::set_error() const':
test_lib.cpp:(.text._ZNK8pybind1110cast_error9set_errorEv[_ZNK8pybind1110cast_error9set_errorEv]+0x10): undefined reference to `PyExc_RuntimeError'
test_lib.cpp:(.text._ZNK8pybind1110cast_error9set_errorEv[_ZNK8pybind1110cast_error9set_errorEv]+0x1c): undefined reference to `PyErr_SetString'
test_lib.o: In function `pybind11::detail::translate_exception(std::__exception_ptr::exception_ptr)':
...
test_lib.cpp:(.text._ZN8pybind116detail5printENS_5tupleENS_4dictE[_ZN8pybind116detail5printENS_5tupleENS_4dictE]+0x9a): undefined reference to `PyTuple_Size'
test_lib.cpp:(.text._ZN8pybind116detail5printENS_5tupleENS_4dictE[_ZN8pybind116detail5printENS_5tupleENS_4dictE]+0xaa): undefined reference to `PyTuple_GetItem'
test_lib.cpp:(.text._ZN8pybind116detail5printENS_5tupleENS_4dictE[_ZN8pybind116detail5printENS_5tupleENS_4dictE]+0xd3): undefined reference to `PyObject_Str'
test_lib.cpp:(.text._ZN8pybind116detail5printENS_5tupleENS_4dictE[_ZN8pybind116detail5printENS_5tupleENS_4dictE]+0x1cc): undefined reference to `PyDict_Contains'
test_lib.cpp:(.text._ZN8pybind116detail5printENS_5tupleENS_4dictE[_ZN8pybind116detail5printENS_5tupleENS_4dictE]+0x3c1): undefined reference to `PyDict_Contains'
test_lib.cpp:(.text._ZN8pybind116detail5printENS_5tupleENS_4dictE[_ZN8pybind116detail5printENS_5tupleENS_4dictE]+0x518): undefined reference to `PyImport_ImportModule'
test_lib.o: In function `void pybind11::print<(pybind11::return_value_policy)1, pybind11::list>(pybind11::list&&)':
test_lib.cpp:(.text._ZN8pybind115printILNS_19return_value_policyE1EJNS_4listEEEEvDpOT0_[_ZN8pybind115printILNS_19return_value_policyE1EJNS_4listEEEEvDpOT0_]+0x21): undefined reference to `PyDict_New'
test_lib.o: In function `main':
test_lib.cpp:(.text.startup+0x34): undefined reference to `Core::Core(bool)'
test_lib.cpp:(.text.startup+0x41): undefined reference to `Core::SetCpuAffinity(int)'
test_lib.cpp:(.text.startup+0x4b): undefined reference to `Core::Start(bool)'
test_lib.cpp:(.text.startup+0x5b): undefined reference to `Core::GetCallbacks()'
test_lib.cpp:(.text.startup+0x85): undefined reference to `Core::GetCallbacks()'
test_lib.cpp:(.text.startup+0x107): undefined reference to `PyObject_Str'
collect2: error: ld returned 1 exit status
Looking at the errors, it seems to me the issue isnot pybind11 itself, but the Python libraries. PyObject_Str,PyObject_Str, PyImport_ImportModule, PyProperty_Type ,etc are all Python C APIs. However I'm providing the Python include and especially the libs directory, so g++ should be able to find the libs and link them. I'm at a loss here why this doesn't work. What is missing here?
I even made a bash file and tried all paths I've found to no avail:
#!/bin/bash
PYBIND_INCL_DIR=-I/home/rika/Documents/cpp/External_Libraries/pybind11/pybind11/include
PYTHON_INCL_DIR=-I/home/rika/anaconda3/include/python3.7m
PYTHON_INCL_DIR2=-I/home/rika/anaconda3/pkgs/python-3.7.4-h265db76_1/include/python3.7m
PYTHON_LIB_DIRS=-L/home/rika/anaconda3/lib/
PYTHON_LIB_DIRS2=-L/home/rika/anaconda3/lib/python3.7/site-packages/
PYTHON_LIB_DIRS3=-L/home/rika/anaconda3/pkgs/python-3.7.4-h265db76_1/lib/
# echo $PYBIND_INCL_DIR
# echo $PYTHON_INCL_DIR
g++ $PYBIND_INCL_DIR $PYTHON_INCL_DIR $PYTHON_LIB_DIRS $PYTHON_LIB_DIRS2 $PYTHON_LIB_DIRS3 -c Core.cpp -o Core.o
g++ $PYBIND_INCL_DIR $PYTHON_INCL_DIR $PYTHON_LIB_DIRS $PYTHON_LIB_DIRS2 $PYTHON_LIB_DIRS3 test_lib.cpp Core.o -o test_lib
Short answer:
As I said, the linker errors were related to the Python library not being found. In order to have a successful build, thus one, needed to only include that library, so the final build command would be :
PYBIND_INCL_DIR=-I/home/rika/Documents/mahsan_FV/cpp/External_Libraries/pybind11/pybind11/include
PYTHON_INCL_DIR=-I/home/rika/anaconda3/include/python3.7m
PYTHON_LIB_DIRS=-L/home/rika/anaconda3/lib/
g++ $PYBIND_INCL_DIR $PYTHON_INCL_DIR $PYTHON_LIB_DIRS -c Core.cpp -o Core.o
g++ $PYBIND_INCL_DIR $PYTHON_INCL_DIR $PYTHON_LIB_DIRS test_lib.cpp Core.o -lpython3.7m -o test_lib
This simply compiles and links all the object files together and solves the issue.
You can get this either by navigating to the Python lib directory or simply
executing python3-config --ldflags or python3.7-config --ldflags (depending on your version of python if you have several installations) in your terminal.(Thank you @ChrisD)
However, in order for the executable to be run properly, the lib folder needs to be in PATH or one could also use ldconfig to load the needed library. :
I had to also do :
sudo ldconfig /home/rika/anaconda3/lib/
Side note:
I also found this website very helpful on the linking procedure.
I also tried to directly use the library which was:
home/hossein/anaconda3/pkgs/python-3.7.4-h265db76_1/lib/libpython3.7m.a
which resulted in the error :
lto1: fatal error: bytecode stream in file ‘/home/rika/anaconda3/pkgs/python-3.7.4-h265db76_1/lib/libpython3.7m.a’ generated with LTO version 6.0 instead of the expected 6.2
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
proving the only way to get the linking right was to use -lpython3.7m!