I'm trying to cross-compile ROS2 Humble for Morello arm64. This architecture is near identical to standard arm64 but uses CHERI capability flags, which requires me to build everything from source.
I'm following the ROS2 cross-compile instruction, with the only difference being that I already have a compiled sysroot that I generated using musl-libc, and I have my own Morello arm64 LLVM compilers.
The build command I'm using is colcon build --merge-install --cmake-force-configure --cmake-args -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_TOOLCHAIN_FILE="/workspace/aarch64-unknown-linux-musl_purecap-toolchain.cmake" -DSECURITY=ON.
And the CMake toolchain I have is:
set(CMAKE_SYSTEM_NAME linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(CMAKE_C_COMPILER /build/morello-llvm/bin/clang)
set(CMAKE_CXX_COMPILER /build/morello-llvm/bin/clang++)
set(CMAKE_SYSROOT /build/sysroot)
set(CMAKE_CROSSCOMPILING TRUE)
set(CMAKE_FIND_ROOT_PATH /build/sysroot)
# Don't run the linker on compiler check
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# Search headers and libraries in the target environment
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
set(PYTHON_SOABI cpython-36m-aarch64-unknown-linux-musl_purecap)
# Set compiler flags
set(CMAKE_C_FLAGS "--sysroot=/build/sysroot --target=aarch64-unknown-linux-musl_purecap -march=morello+c64 -mabi=purecap")
set(CMAKE_CXX_FLAGS "--sysroot=/build/sysroot --target=aarch64-unknown-linux-musl_purecap -march=morello+c64 -mabi=purecap -stdlib=libc++ -static")
set(CMAKE_EXE_LINKER_FLAGS "-lunwind -lc++abi -pthread -v")
The issue I'm getting is that the build fails during the building of the iceoryx dependency, likely due to me not linking or including a library properly:
/home/morello-admin/ros2_humble/src/eclipse-iceoryx/iceoryx/iceoryx_hoofs/platform/unix/include/iceoryx_hoofs/platform/mqueue.hpp:23:49: error: unknown type name 'mode_t'; did you mean 'mqd_t'?
/home/morello-admin/ros2_humble/src/eclipse-iceoryx/iceoryx/iceoryx_hoofs/platform/unix/source/mqueue.cpp:26:49: error: unknown type name 'mode_t'; did you mean 'mqd_t'?
This can be fixed trivially by adding #include <sys/stat.h> to the files. But this is just a symptom of a larger issue such as a missing dependency. Besides the issue continues after the "fix":
/home/morello-admin/ros2_humble/src/eclipse-iceoryx/iceoryx/iceoryx_hoofs/source/posix_wrapper/thread.cpp:39:15: error: use of undeclared identifier 'pthread_getname_np'
My assumption is that I'm not linking pthread/dependencies properly or have a missing/erroneous build flag somewhere, but I can't figure out what's wrong specifically.
I've tried adding code to link pthreads in ~/ros2_humble/src/eclipse-iceoryx/iceoryx/iceoryx_hoofs/CMakeLists.txt, hoping that it would allow it to find the dependency:
+ set(THREADS_PREFER_PTHREAD_FLAG ON)
+ find_package(Threads REQUIRED)
target_link_libraries(iceoryx_hoofs
PUBLIC
iceoryx_hoofs::iceoryx_platform
PRIVATE
+ Threads::Threads
${ICEORYX_SANITIZER_FLAGS}
)
I also added -pthread to my CMake toolchain.
However the issue persists.
Is this because pthreads is missing from my sysroot? I can see /usr/lib/x86_64-linux-gnu/libpthread.so on my host environment, but also /build/sysroot/lib/libpthread.a and `/build/sysroot/include/pthread.h in my sysroot.
I'm reasonably new to compiling code to this level, but any help would be appreciated, regarding either my build setup or the process to get ROS2 built from source for this architecture.
pthread_getname_np is a non-portable function (the "np") that is implemented by glibc. In musl libc it was only added relatively recently, and up until (and including) the 1.6.0 release of Arm's Morello musl Linux software stack the version of musl included predated that^1. Since then the 1.6.1 release has included it^2.
As for the lack of mode_t, this again sounds like ROS2 is relying on the libc being glibc, at least when compiling for Linux. Likely something ROS2 includes happens to implicitly end up with mode_t defined, but with musl this doesn't happen, and so you need to add the missing includes to ensure that the declarations you need are pulled in.
All told, I don't think anything there is Morello-specific; rather it's a result of trying to use musl rather than glibc (and an older version thereof).
You may also wish to use Arm's Morello forum and/or our public CHERI-CPU Slack in future to obtain CHERI/Morello-related support rather than Stack Overflow, as they are niche technologies that very few people on Stack Overflow will be able to offer help with (although in this specific case as it's turned out to not be Morello-specific that's less true).
^1: https://git.morello-project.org/morello/musl-libc/-/blob/morello-release-1.6.0/include/pthread.h
^2: https://git.morello-project.org/morello/musl-libc/-/blob/morello-release-1.6.1/include/pthread.h#L224