Search code examples
pythondockerubuntucmakeboost

Why does building Boost from source change the Python version found by CMake?

I am trying to update the Boost version within a Docker image used for builds. As the newest versions of Boost are not available via apt for older Ubuntu versions (in my case 18.04 LTS), I am having to build Boost from source. I am encountering some strange behavior in CMake only when building Boost from source as opposed to installing it via apt-get install libboost-all-dev.

When building Boost from source (BUILD_BOOST_FROM_SOURCE=1 in the Dockerfile), CMake finds the 'wrong' python version:

#5 563.5 -- Found Boost: /usr/local/include (found version "1.65.1")  
#5 564.1 -- Found Python: /usr/bin/python2.7 (found version "2.7.17") found components: Interpreter Development Development.Module Development.Embed

When installing Boost via apt-get install libboost-all-dev (BUILD_BOOST_FROM_SOURCE=0 in the Dockerfile), I get my desired configuration:

#5 88.70 -- Found Boost: /usr/include (found version "1.65.1")  
#5 89.16 -- Found Python: /usr/bin/python3.6 (found version "3.6.9") found components: Interpreter Development Development.Module Development.Embed

Here's a stripped down version of my Dockerfile that reproduces the problem and runs CMake in the end to check the result. I have changed the build Boost version to the exact same which is installed through apt, the problem persists for newer versions.

FROM --platform=$BUILDPLATFORM ubuntu:bionic

ENV DEBIAN_FRONTEND noninteractive
ENV TZ America/New_York
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
ENV LANGUAGE en_US.UTF-8

ARG BOOST_VERSION=1.65.1  # adjusted to be equal to the one installed via apt-get
ENV BUILD_BOOST_FROM_SOURCE=1  # build from source or install via apt-get

RUN /bin/bash <<EOF

set -euxo pipefail

# Install package dependencies
apt-get update
apt-get install -y --no-install-recommends \
    build-essential \
    software-properties-common \
    autoconf \
    automake \
    libtool \
    pkg-config \
    ca-certificates \
    libssl-dev \
    wget \
    git \
    curl \
    gnupg \
    language-pack-en \
    locales \
    locales-all \
    vim \
    gdb \
    valgrind

# install python
apt-get install -y --no-install-recommends \
    python \
    python-dev \
    python3 \
    python3-pip \

apt-get clean

# perform the rest in the /tmp dir
cd /tmp

if [ "$BUILD_BOOST_FROM_SOURCE" -eq 1 ]; then \
  # build and install the specific boost version from source given by the BOOST_VERSION ARG
  wget https://boostorg.jfrog.io/artifactory/main/release/${BOOST_VERSION}/source/boost_$(echo ${BOOST_VERSION} | tr . _).tar.bz2 && \
    tar --bzip2 -xf boost_$(echo ${BOOST_VERSION} | tr . _).tar.bz2 && \
    cd boost_$(echo ${BOOST_VERSION} | tr . _) && \
    ./bootstrap.sh --prefix=/usr/local && \
    ./b2 install; \
else \
  # Install boost simply via apt-get
  apt-get install -y --no-install-recommends \
    libboost-all-dev; \
fi

# Use kitware's CMake repository for an up-to-date version and install cmake
curl -sSf https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | apt-key add -
apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'
apt-get install -y --no-install-recommends cmake

# check the boost and python versions found by cmake
echo -e 'cmake_minimum_required(VERSION 3.12)
project(JustCheckingSomeVersions)
find_package(Boost)
find_package(Python COMPONENTS Interpreter Development)
' > CMakeLists.txt

cmake .

EOF

I have tried to change the order of installations in various ways as well as passing the --with-python=/usr/bin/python3 to bootstrap.sh. None of these measures helped and I am out of ideas and do not know enough about the intricacies involved here to make progress. How can I ensure that CMake still finds Python 3.x when building Boost from source within this Docker image?

Edit: Thanks to Qyriads comment I was able to further track this down to CMake not finding the Python.h header for the 3.6 version. The library and executable are found at /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.so and /usr/bin/python3.6 respectively. The output from CMAKE_FIND_DEBUG_MODE shows this:

#5 564.4 CMake Debug Log at /usr/share/cmake-3.25/Modules/FindPython/Support.cmake:2963 (find_path):
#5 564.4   find_path called with the following settings:
#5 564.4 
#5 564.4     VAR: _Python_INCLUDE_DIR
#5 564.4     NAMES: "Python.h"
#5 564.4     Documentation: Path to a file.
#5 564.4     Framework
#5 564.4       Only Search Frameworks: 0
#5 564.4       Search Frameworks Last: 0
#5 564.4       Search Frameworks First: 0
#5 564.4     AppBundle
#5 564.4       Only Search AppBundle: 0
#5 564.4       Search AppBundle Last: 0
#5 564.4       Search AppBundle First: 0
#5 564.4     CMAKE_FIND_USE_CMAKE_PATH: 1
#5 564.4     CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH: 1
#5 564.4     CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH: 0
#5 564.4     CMAKE_FIND_USE_CMAKE_SYSTEM_PATH: 0
#5 564.4     CMAKE_FIND_USE_INSTALL_PREFIX: 1
#5 564.4 
#5 564.4   find_path considered the following locations:
#5 564.4 
#5 564.4     /usr/include/python3.6m/Python.h
#5 564.4     /usr/include/Python.h
#5 564.4     /usr/Python.h
#5 564.4 
#5 564.4   The item was not found.
#5 564.4 
#5 564.4 Call Stack (most recent call first):
#5 564.4   /usr/share/cmake-3.25/Modules/FindPython.cmake:540 (include)
#5 564.4   CMakeLists.txt:5 (find_package)
#5 564.4 
#5 564.4 CMake Debug Log at /usr/share/cmake-3.25/Modules/FindPython/Support.cmake:2975 (find_path):
#5 564.4   find_path called with the following settings:
#5 564.4 
#5 564.4     VAR: _Python_INCLUDE_DIR
#5 564.4     NAMES: "Python.h"
#5 564.4     Documentation: Path to a file.
#5 564.4     Framework
#5 564.4       Only Search Frameworks: 0
#5 564.4       Search Frameworks Last: 0
#5 564.4       Search Frameworks First: 0
#5 564.4     AppBundle
#5 564.4       Only Search AppBundle: 0
#5 564.4       Search AppBundle Last: 0
#5 564.4       Search AppBundle First: 0
#5 564.4     CMAKE_FIND_USE_CMAKE_PATH: 1
#5 564.4     CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH: 1
#5 564.4     CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH: 1
#5 564.4     CMAKE_FIND_USE_CMAKE_SYSTEM_PATH: 1
#5 564.4     CMAKE_FIND_USE_INSTALL_PREFIX: 1
#5 564.4 
#5 564.4   find_path considered the following locations:
#5 564.4 
#5 564.4     /usr/local/sbin/Python.h
#5 564.4     /usr/local/bin/Python.h
#5 564.4     /usr/sbin/Python.h
#5 564.4     /usr/bin/Python.h
#5 564.4     /sbin/Python.h
#5 564.4     /bin/Python.h
#5 564.4     /usr/local/include/x86_64-linux-gnu/Python.h
#5 564.4     /usr/local/include/Python.h
#5 564.4     /usr/local/Python.h
#5 564.4     /usr/include/x86_64-linux-gnu/Python.h
#5 564.4     /usr/include/Python.h
#5 564.4     /usr/Python.h
#5 564.4     /include/x86_64-linux-gnu/Python.h
#5 564.4     /include/Python.h
#5 564.4     /usr/X11R6/include/x86_64-linux-gnu/Python.h
#5 564.4     /usr/X11R6/include/Python.h
#5 564.4     /usr/X11R6/Python.h
#5 564.4     /usr/pkg/include/x86_64-linux-gnu/Python.h
#5 564.4     /usr/pkg/include/Python.h
#5 564.4     /usr/pkg/Python.h
#5 564.4     /opt/include/x86_64-linux-gnu/Python.h
#5 564.4     /opt/include/Python.h
#5 564.4     /opt/Python.h
#5 564.4     /usr/include/X11/Python.h
#5 564.4 
#5 564.4   The item was not found.

When installing Boost via apt-get, this header is found under /usr/include/python3.6m/Python.h.

Solution

  • Ubuntu Bionic is too old to search for directly on https://packages.ubuntu.com and confirm, but I think you need to install libpython3.6-dev, or something similar (e.g., libpython3-dev, libpython-dev, python3.6-dev, python3-dev). My assumption is either that python-dev is Python 2, or that libpython-dev has Python.h but python-dev does not. Whichever it is, libboost-all-dev is likely pulling in the correct package as a dependency, which is why not installing it wouldn't would mean CMake wouldn't find Python.h.