Search code examples
c++debugginggdbcross-compilingbazel

On embedded device how to natively debug a executable generated by bazel build


Bazel build a simple piece of code hello-world with cross-comipler toolchains.

$ cat examples/hello_world/BUILD
cc_binary(
    name = "hello_world",
    srcs = ["hello_world.cpp"],
)
$ bazel build --cxxopt=-std=c++11 --config=tda4 --copt=-DWITH_TDA4 --subcommands --copt=-DFOOFXXX -c dbg --strip=never //examples/hello_world:hello_world
Starting local Bazel server and connecting to it...

INFO: Analyzed target //examples/hello_world:hello_world (41 packages loaded, 40489 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //examples/hello_world:hello_world [action 'Compiling examples/hello_world/main.cpp', configuration: e23aa3bce872abfb8f06c09952a42b5c1c7714b1c55b5bf022d91a6582852828, execution platform: @local_config_platform//:host]
(cd /home/wss/.cache/bazel/_bazel_wss/4112892ab2eb09dee5980f588ac10e42/execroot/com_minieye_aiplorer && \
  exec env - \
    LD_LIBRARY_PATH=/opt/ros/melodic/lib \
    PATH=/opt/ros/melodic/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  bazel/toolchain/aarch64-none-linux-gnu/wrappers/aarch64-none-linux-gnu-gcc -MD -MF bazel-out/aarch64-dbg/bin/examples/hello_world/_objs/hello_world/main.d '-frandom-seed=bazel-out/aarch64-dbg/bin/examples/hello_world/_objs/hello_world/main.o' '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/aarch64-dbg/bin -iquote external/bazel_tools -iquote bazel-out/aarch64-dbg/bin/external/bazel_tools '--sysroot=external/aarch64-none-linux-gnu-sysroot' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -DWITH_TDA4 -DFOOFXXX '-std=c++11' -c examples/hello_world/main.cpp -o bazel-out/aarch64-dbg/bin/examples/hello_world/_objs/hello_world/main.o)
# Configuration: e53aa3bce872abfb8f06c29952a42b5c1c7714b1c55b5bf022d91a6582852828
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //examples/hello_world:hello_world [action 'Linking examples/hello_world/hello_world', configuration: e23aa3bce872abfb8f06c09952a42b5c1c7714b1c55b5bf022d91a6582852828, execution platform: @local_config_platform//:host]
(cd /home/wss/.cache/bazel/_bazel_wss/4112892ab2eb09dee5980f588ac10e42/execroot/com_minieye_aiplorer && \
  exec env - \
    LD_LIBRARY_PATH=/opt/ros/melodic/lib \
    PATH=/opt/ros/melodic/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  bazel/toolchain/aarch64-none-linux-gnu/wrappers/aarch64-none-linux-gnu-gcc @bazel-out/aarch64-dbg/bin/examples/hello_world/hello_world-2.params)
# Configuration: e23aa3bce872abfb8f06c09952a42b5c1c7714b1c55b5bf022d91a6582852828
# Execution platform: @local_config_platform//:host
Target //examples/hello_world:hello_world up-to-date:
  bazel-bin/examples/hello_world/hello_world
INFO: Elapsed time: 5.316s, Critical Path: 1.37s
INFO: 6 processes: 4 internal, 2 linux-sandbox.
INFO: Build completed successfully, 6 total actions

note: tda4 is the type of soc on target device, and this parameter determines which cross-compilation toolchain to be used, I confirm that the generated executable binary contains .debug_info section, and the size is 000288, and after add the flag --fission=no, the size is the same, because no is the default value of fission.

$ file bazel-bin/examples/hello_world/hello_world
bazel-bin/examples/hello_world/hello_world: ELF 64-bit LSB executable, ARM aarch64,   
version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1,    
for GNU/Linux 3.7.0, with debug_info, not stripped
$ readelf -WS bazel-bin/examples/hello_world/hello_world |grep -E "Address|debug"
[Nr] Name             Type        Address          Off    Size   ES Flg Lk Inf Al
[26] .debug_aranges   PROGBITS    0000000000000000 0010c0 000110 00      0   0 16
[27] .debug_info      PROGBITS    0000000000000000 0011d0 000288 00      0   0  1
[28] .debug_abbrev    PROGBITS    0000000000000000 001458 00019c 00      0   0  1
[29] .debug_line      PROGBITS    0000000000000000 0015f4 0002d3 00      0   0  1
[30] .debug_str       PROGBITS    0000000000000000 0018c7 000338 01  MS  0   0  1
[31] .debug_loc       PROGBITS    0000000000000000 001bff 00019b 00      0   0  1
[32] .debug_ranges    PROGBITS    0000000000000000 001da0 0000c0 00      0   0 16

then scp the executable binary to the target embedded device under the directory /app/bin, and also scp the whole source code directory examples(sub-directory includes hello_world/hello_world.cpp) to the device under /app/bin, now run GDB for natively stepping debug, and set the directories of the souce code at first to make sure the gdb will find the source code, but it'll not display the source code still.

$ gdb hello_world
Reading symbols from hello_world...
(gdb) set directories /app/bin/examples/hello_world
(gdb) shell cat -n /app/bin/examples/hello_world/hello_world.cpp  |tail -10
     5  int fun(int a, int b)
     6  {
     7     return a+b;
     8  }
     9  int main(int argc, char *argv[]) {
    10    std::cout << "WITH LINUX~~~" << std::endl;
    11    int c=fun(1,1);
    12    std::cout << "Hello World!" << std::endl;
    13    std::cout<<c<<std::endl;
    14    return 0;
    15  }
(gdb) b hello_world.cpp:10
No source file named hello_world.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (hello_world.cpp:10) pending.
(gdb) b hello_world.cpp:11
No source file named hello_world.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (hello_world.cpp:11) pending.
(gdb) i b
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   <PENDING>  hello_world.cpp:10
2       breakpoint     keep y   <PENDING>  hello_world.cpp:11
(gdb) r
Starting program: /app/bin/hello_world
WITH LINUX~~~
Hello World!
2
[Inferior 1 (process 17279) exited normally]

We can find the addresses of breakpoints are pending, and they can break the program.


Solution

  • confirm that the generated executable binary contains debug info

    ... with debug_info, not stripped

    This does not actually confirm anything useful -- it just says that there is some debug info present. But that debug info could be coming from crt0.o and not from any of your sources.

    Indeed this line:

    Breakpoint 1, 0x000000000040497c in main ()
    

    tells us that you don't have debug info for main.

    You can use readelf -WS /app/bin/hello_world and look for the size of the .debug_info section. Chances are that size is really small.


    Bazel build a simple piece of code hello-world

    What command line flags did you use? In particular, did you use -c dbg?

    If not, that's likely your problem. Documentation.

    Update:

    bazel build ... -c dbg ...

    The only other thing I can think of is to disable fission: append --fission=no and see if the size of .debug_info changes.

    Update 2:

    In the readelf -wi ... output, we see that there are only 5 DW_TAG_compile_units present, and all of them are from GLIBC startup files; none are from your own application.

    This confirms my suspicion -- you don't actually have any debug info for your own sources.

    But bazel build -c dbg ... should have produced this debug info. The next step is to use bazel build --subcommands --copt=-DFOOFXXX=$RANDOM -c dbg ... to rebuild the app and see what actual compile command is being used.

    Update 3:

    The compilation command:

    bazel/toolchain/aarch64-none-linux-gnu/wrappers/aarch64-none-linux-gnu-gcc \
      -MD -MF \
      bazel-out/aarch64-dbg/bin/examples/hello_world/_objs/hello_world/main.d \
      -frandom-seed=bazel-out/aarch64-dbg/bin/examples/hello_world/_objs/hello_world/main.o \
      -DBAZEL_CURRENT_REPOSITORY="" -iquote . \
      -iquote bazel-out/aarch64-dbg/bin \
      -iquote external/bazel_tools \
      -iquote bazel-out/aarch64-dbg/bin/external/bazel_tools \
      --sysroot=external/aarch64-none-linux-gnu-sysroot \
      -no-canonical-prefixes -fno-canonical-system-headers \
      -Wno-builtin-macro-redefined \
      -D__DATE__="redacted" \
      -D__TIMESTAMP__="redacted" \
      -D__TIME__="redacted" \
      -DWITH_TDA4 -DFOOFXXX -std=c++11 \
      -c examples/hello_world/main.cpp \
      -o bazel-out/aarch64-dbg/bin/examples/hello_world/_objs/hello_world/main.o
    

    lacks the -g flag, so GCC was never told to emit debug info in the first place.

    This means that this toolchain is misconfigured.

    I don't know how this happened, but you should be able to work around this by appending an explicit --copt=-g flag.