Debugging bpf and bpf jit

I wrote some bpf programs. I've enabled echo "2" > /proc/sys/net/core/bpf_jit_enable so it outputs the generated jitted code in logs, but I don't have bpf_jit_disasm inside the qemu environment in which I generated the jitted code. The qemu environment just has busybox and no other tools.

How do I take the jitted output generated in the qemu environment and pass it to bpf_jit_disasm? I've tried -f but it expects a JIT IMAGE. Not sure what that means.

Also what are the other ways I can debug the bpf programs?

Solution

With bpf_jit_enable set to 2, the JIT-ed image of the program should be printed to the kernel logs on a successful program load. You should be able to simply copy-paste it from the kernel logs to a file, and to feed that file to bpf_jit_disasm.

[EDIT] For example:

# echo 2 > /proc/sys/net/core/bpf_jit_enable
# cat /proc/sys/net/core/bpf_jit_enable
2
# bpftool prog load sample_ret0.o /sys/fs/bpf/foo type kprobe
# dmesg
[...]
[5244802.925533] bpf_jit_enable = 2 was set! NEVER use this in production, only for JIT debugging!
[5244815.524017] flen=133 proglen=678 pass=5 image=0000000000000000 from=sshd pid=1398988
[5244815.618778] JIT code: 00000000: 0f 1f 44 00 00 55 48 89 e5 53 41 55 31 c0 45 31
[5244815.708419] JIT code: 00000010: ed 48 89 fb 8b 43 04 be 3e 00 00 c0 48 39 f0 74
[5244815.797957] JIT code: 00000020: 07 31 c0 41 5d 5b c9 c3 8b 43 00 48 83 f8 06 75
...
[5244911.371176] JIT code: 000002a0: c0 e9 7d fd ff ff

Copy-paste...

$ cat sample_ret0.jit.txt
[5244815.524017] flen=133 proglen=678 pass=5 image=0000000000000000 from=sshd pid=1398988
[5244815.618778] JIT code: 00000000: 0f 1f 44 00 00 55 48 89 e5 53 41 55 31 c0 45 31
[5244815.708419] JIT code: 00000010: ed 48 89 fb 8b 43 04 be 3e 00 00 c0 48 39 f0 74
[5244815.797957] JIT code: 00000020: 07 31 c0 41 5d 5b c9 c3 8b 43 00 48 83 f8 06 75
...
[5244911.371176] JIT code: 000002a0: c0 e9 7d fd ff ff

$ wc sample_ret0.jit.txt
  148  2779 12521 sample_ret0.jit.txt

$ bpf_jit_disasm -f sample_ret0.jit.txt
15 bytes emitted from JIT compiler (pass:3, flen:2)
ffffffffc10077f8 + <x>:
   0:   jmpq   0xfffffffffffffed3
   5:   cmp    $0x60,%rax
   9:   jne    0x0000000000000015
   b:   .byte 0xb8
   c:   add    %al,(%rax)
   e:   .byte 0xff

There are a couple of other ways to inspect eBPF programs, although I'm not sure the tools will be available on a busybox:

llvm-objdump (-S|-d) <object file> on the object file (but only for eBPF bytecode, you will not get the JIT-ed image)
bpftool prog dump (jited|xlated) <prog ref> (See man bpftool-prog for details)