Search code examples
pythoncmakecythondistutils

cython - distutils vs cmake: linking against libpython?

I had asked another question that was a bit too complicated for a straight answer so I boiled it down to this basic question...

When I build my aModule.so using a standard cython distutils, it does not seem to be linked against libpython:

$ otool -L aModule.so
aModule.so:
    /usr/local/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/thrift/lib/libthrift-0.9.0.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

But when I build with a cmake setup, it keeps producing a linker command that links libpython into the .so :

$ otool -L aModule.so 
aModule.so:
    /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.1)
    /usr/local/opt/thrift/lib/libthrift-0.9.0.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

The module produced by distutils seems to work fine with any of my python2.7 installation (the system, or a virtualenv for my project). Whereas the cmake crashes with a version mismatch when I try and import it with anything but that linked system python.

Why does the distutils module work fine without being linked? And if thats the case, why do I need to have the cmake build linking libpython, and how could I prevent it if thats the case so that it works with any of my python2.7 interpreters without a crash?

Currently I can direct the cmake at the right python with: CXX=g++ cmake -DPYTHON_LIBRARY=/path/to/another/Python

Solution

  • I realized the source of the issue was related to the cython-cmake-example and how its UseCython.cmake cython_add_module() function would explicitly link the library against libpython.

    What I ended up doing for my own use, since I do not know if this is a completely portable solution, was to add a flag to that function to say DYNAMIC_LOOKUP:

    function( cython_add_module _name _dynamic_lookup )
  set( pyx_module_sources "" )
  set( other_module_sources "" )
  foreach( _file ${ARGN} )
    if( ${_file} MATCHES ".*\\.py[x]?$" )
      list( APPEND pyx_module_sources ${_file} )
    else()
      list( APPEND other_module_sources ${_file} )
    endif()
  endforeach()
  compile_pyx( ${_name} generated_file ${pyx_module_sources} )
  include_directories( ${PYTHON_INCLUDE_DIRS} )
  python_add_module( ${_name} ${generated_file} ${other_module_sources} )
  ### Added here ##
  if( ${_dynamic_lookup} )
    message( STATUS "Not linking target ${_name} against libpython" )
    set_target_properties( ${_name} PROPERTIES LINK_FLAGS "-undefined dynamic_lookup")
  else()
    target_link_libraries( ${_name} ${PYTHON_LIBRARIES} )
  endif()
endfunction()

    Now I can call cython_add_module and it won't link against libpython.