I'm trying to get our cross-toolchain (standard Yocto toolchain) working with Bazel. I followed the instructions on https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain but the linker fails every time I try to build a simple test program. It says
external/toolchain_e6500/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc64-fsl-linux/../../libexec/powerpc64-fsl-linux/gcc/powerpc64-fsl-linux/4.9.2/real-ld: cannot find /lib64/libc.so.6
external/toolchain_e6500/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc64-fsl-linux/../../libexec/powerpc64-fsl-linux/gcc/powerpc64-fsl-linux/4.9.2/real-ld: cannot find /usr/lib64/libc_nonshared.a
external/toolchain_e6500/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc64-fsl-linux/../../libexec/powerpc64-fsl-linux/gcc/powerpc64-fsl-linux/4.9.2/real-ld: cannot find /lib64/ld64.so.1
The sysroot is set properly. When running the linker command outside of the sandbox (e.g. with --spawn_strategy=standalone or just manually) it works always. The strange thing is that when I use --debug_sandbox and run the emitted command from the command line, it works, too.
I've been debugging this issue for two days now, including straceing the Bazel daemon and comparing the real-ld input but I didn't find anything suspicious.
There must be a difference between the execution environments but I'm out of ideas now. Here is the failing command as printed by --debug_sandbox:
(cd /home/sick/.cache/bazel/_bazel_sick/2a7ae5e27644389520091aa03d045c73/execroot/__main__ && \
exec env - \
PATH=/home/sick/bin:/home/sick/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/sick/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/home/sick/.cache/bazel/_bazel_sick/2a7ae5e27644389520091aa03d045c73/execroot/__main__/_bin/linux-sandbox -t 15 -w /home/sick/.cache/bazel/_bazel_sick/2a7ae5e27644389520091aa03d045c73/sandbox/linux-sandbox/2/execroot/__main__ -w /tmp -w /dev/shm -D -- tools/compiler_e6500/e6500_gcc/powerpc64-fsl-linux-gcc -o bazel-out/e6500-fastbuild/bin/test '--sysroot=external/toolchain_e6500/sysroots/ppc64e6500-fsl-linux' -no-canonical-prefixes -pie -Wl,-z,relro,-z,now -Wl,-S -Wl,@bazel-out/e6500-fastbuild/bin/test-2.params)
You can take a look at the workspace here https://github.com/jasal82/bazel-cross-eval
I can provide the toolchain if needed.
UPDATE 2018-09-25
I did some more investigation after reading the answer regarding the linker script below. Section 4.4.2 in this manual says that
In case a sysroot prefix is configured, and the filename starts with the / character, and the script being processed was located inside the sysroot prefix, the filename will be looked for in the sysroot prefix. Otherwise, the linker will try to open the file in the current directory.
Obviously the script is not being considered to be located inside the sysroot prefix by ld and thus used literally (i.e. resolved relative to the current dir, probably the sandbox root). That would explain the observed behavior. However, I still don't understand why it does not happen when I run the command manually, i.e. not through the Bazel daemon.
Could this be related to the relative sysroot path used in the CROSSTOOL file? As far as I understood you cannot specify an absolute path to the sysroot due to the sandboxing. What is the recommended way to handle this in Bazel? I would like to avoid having to patch the toolchain.
Have a look at lib.so in your sysroot. It probably looks something like this:
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc_nonshared.a AS_NEEDED ( /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 )
Change the paths in it to be relative to the directory that libc.so is in. You'll have to perform a similar operation on libm.so and libpthread.so.
The GNU linkers resolve symlinks before evaluating whether a script is within the sysroot. The current Bazel Linux sandbox implementation makes a symlink farm of all an action's inputs. Thus, the libc.so.6 linker script is detected as being in the sysroot outside the sandbox but not within the sandbox.