After doing some research, I found that there isn't a single "tried-and-true" way of cross-compiling ROS for RaspberryPi. The best I could come up with was downloading /usr, /lib, and /opt from the target device onto my development machine (installed at $HOME/Projects/TargetResources), making a CMake Toolchain file, and calling catkin_make_isolated -DCMAKE_TOOLCHAIN_FILE=.... Though the build process starts, I'm met with an immediate error:
$ catkin_make_isolated -DCMAKE_TOOLCHAIN_FILE=${HOME}/Projects/${MY_PROJ}/RPI3+_Melodic_Toolchain.cmake
CMake Error at ${HOME}/Projects/TargetResources/opt/ros/melodic/share/catkin/cmake/assert.cmake:17 (message):
Assertion failed: check for file existence, but filename
(RT_LIBRARY-NOTFOUND) unset. Message: RT Library
Call Stack (most recent call first):
${HOME}/Projects/TargetResources/opt/ros/melodic/share/catkin/cmake/tools/rt.cmake:42 (assert_file_exists)
${HOME}/Projects/TargetResources/opt/ros/melodic/share/catkin/cmake/all.cmake:159 (include)
${HOME}/Projects/TargetResources/opt/ros/melodic/share/catkin/cmake/catkinConfig.cmake:20 (include)
CMakeLists.txt:10 (find_package)
After doing some digging, the target device (i.e., RaspberryPi 3+) does not have librt.so. Interestingly enough, if I just run catkin_make for the project on the device, everything builds successfully. So I don't think attempting to install it is really necessary (or the right solution).
Additionally, I should note that the target has ROS Melodic Base installed whereas my development machine has ROS Melodic Desktop installed. I'm not sure if this would cause the issue I'm experiencing, but I don't want to prematurely rule it out.
So, my question is how should I proceed? Did I overlook something whilst setting up the toolchain or am I assuming something about ROS/Catkin that is false?
Thank you in advance for all of your help and consideration.
Edit / Addendum
As requested, here is the toolchain file:
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_VERSION 1)
SET(CMAKE_C_COMPILER $ENV{HOME}/Projects/RpiDevTools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc)
SET(CMAKE_CXX_COMPILER $ENV{HOME}/Projects/RpiDevTools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin/arm-linux-gnueabihf-g++)
SET(CMAKE_FIND_ROOT_PATH $ENV{HOME}/Projects/RpiDevTools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/arm-linux-gnueabihf/sysroot/)
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
In sysroot, I copied the RaspberryPi's /opt directory as that's where ROS Melodic libraries are installed. I've found that those libraries may have dependencies (for example log4cxx/level.h) which are not found in the RaspberryPi/Tools sysroot/usr/include directory. Likewise, resolving the shared object dependencies are equally cumbersome.
It appears that the issue stems from the cross-compilation tools itself: using arm-linux-gnueabihf-gcc & arm-linux-gnueabihf-g++ from https://github.com/raspberrypi/tools means that the compilers are configured with libc libraries in a pre-known configuration. For example, math.h includes bits/math-vector.h; in the RaspberryPi Tools from GitHub, this is located at .../sysroot/usr/include/bits whereas copied straight from the RaspberryPi 3B+ itself is located at /usr/include/arm-linux-gnueabf/bits. The respective compilers (native and cross-compile) were built with those their respective configurations, but not the others and does not search (well) the generalized paths.