Search code examples
rustlinux-kernelebpfbpftraceuprobe

uprobe symbol adress mapping offset


I am trying to set a uprobe in the libart.so android library on an Android x86_64 emulator with Rust (aya[0]). All is well on Android-14 (Kernel 6.1), but not in Android-13 (Kernel 5.15).

As far as I understand it, the perf_event_open[1] sys-call is used to attach an uprobe in both Android versions.

NOTE: in Android "apps" are forked from a "zygote" process, which pre-loads shared libraries such as libart.so, thus libart.so is shared between all apps.

Setup

Android-13:

Symbol art::JNIEnvExt::DeleteLocalRef(_jobject*) - offset: 0x601300 in libart.so

❯ readelf -Ws libart_A13_x86_64.so | c++filt | grep -i DeleteLocalRef
   Num:    Value          Size Type    Bind   Vis      Ndx Name
  1208: 0000000000601300    26 FUNC    GLOBAL PROTECTED   14 art::JNIEnvExt::DeleteLocalRef(_jobject*)

ELF .text section in libart.so

❯ readelf -S --wide libart_A13_x86_64.so
There are 29 section headers, starting at offset 0x89be68:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
...
  [14] .text             PROGBITS        000000000035ff00 15ff00 5df958 00  AX  0   0 128 

(addr -> virtual address. Actual address is base adr + virt adr when loaded into the process.

Runtime:

oia_emu:/ # cat /proc/6160/maps | grep libart.so
7388c1a00000-7388c1b60000 r--p 00000000 fe:21 85                         /apex/com.android.art/lib64/libart.so
7388c1d5f000-7388c2343000 r-xp 0015f000 fe:21 85                         /apex/com.android.art/lib64/libart.so
7388c2542000-7388c2553000 r--p 00742000 fe:21 85                         /apex/com.android.art/lib64/libart.so
7388c2752000-7388c2756000 rw-p 00752000 fe:21 85                         /apex/com.android.art/lib64/libart.so

Android-14:

Symbol: art::JNIEnvExt::DeleteLocalRef(_jobject*) - offset: 0x684d00 in libart.so

❯ readelf -Ws libart14_x86_64.so | c++filt | grep -i DeleteLocalRef
  2138: 0000000000684d00    15 FUNC    GLOBAL DEFAULT   15 art::JNIEnvExt::DeleteLocalRef(_jobject*)

ELF .text section in libart.so

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
...
  [15] .text             PROGBITS        0000000000200000 200000 7adbb6 00  AX  0   0 128

Runtime:

130|emulator_car64_x86_64:/ # cat /proc/26733/maps | grep libart.so
7720d1e00000-7720d1f50000 r--p 00000000 fe:26 53                         /apex/com.android.art/lib64/libart.so
7720d2000000-7720d27b4000 r-xp 00200000 fe:26 53                         /apex/com.android.art/lib64/libart.so
7720d2800000-7720d2822000 r--p 00a00000 fe:26 53                         /apex/com.android.art/lib64/libart.so
7720d2a00000-7720d2a04000 rw-p 00c00000 fe:26 53                         /apex/com.android.art/lib64/libart.so

Runtime behavior

In order to verify which address offset is used to call perf_event_open I used strace

This is the sys-call done by my process on Android-13:

perf_event_open({type=0x7 /* PERF_TYPE_??? */, size=0x88 /* PERF_ATTR_SIZE_??? */, config=0, sample_period=0, sample_type=0, read_format=0, disabled=0, inherit=0, pinned=0, exclusive=0, exclusive_user=0, exclude_kernel=0, exclude_hv=0, exclude_idle=0, mmap=0, comm=0, freq=0, inherit_stat=0, enable_on_exec=0, task=0, watermark=0, precise_ip=0 /* arbitrary skid */, mmap_data=0, sample_id_all=0, exclude_host=0, exclude_guest=0, exclude_callchain_kernel=0, exclude_callchain_user=0, mmap2=0, comm_exec=0, use_clockid=0, context_switch=0, write_backward=0, namespaces=0, wakeup_events=0, config1=0x77f607cdf8d0, config2=0x601300, sample_regs_user=0, sample_regs_intr=0, aux_watermark=0, sample_max_stack=0, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = -1 ENOTSUPP (Unknown error 524)

(offset: config2=0x601300)

On Android-14 it is the same symbol, as specified in the setup topic

perf_event_open({type=0x7 /* PERF_TYPE_??? */, size=0x88 /* PERF_ATTR_SIZE_??? */, config=0,   sample_period=0, sample_type=0, read_format=0, disabled=0, inherit=0, pinned=0, exclusive=0, exclusive_user=0, exclude_kernel=0, exclude_hv=0, exclude_idle=0, mmap=0, comm=0, freq=0, inherit_stat=0, enable_on_exec=0, task=0, watermark=0, precise_ip=0 /* arbitrary skid */, mmap_data=0, sample_id_all=0, exclude_host=0, exclude_guest=0, exclude_callchain_kernel=0, exclude_callchain_user=0, mmap2=0, comm_exec=0, use_clockid=0, context_switch=0, write_backward=0, namespaces=0, wakeup_events=0, config1=0x73a286096250, config2=0x684d00, sample_regs_user=0, sample_regs_intr=0, aux_watermark=0, sample_max_stack=0, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 17

(offset: config2=0x684d00)

On Android-14 I get invocations, but on Android-13 I do not get any. I am sure there should have been calls.

bpftrace:[2] verification

I remembered that bpftrace could be used to verify previously mentioned uprobes.

On Android 14 the symbol offset 0x684d00 is used. It works as expected.

Android14:

# strace -v -f -e trace=perf_event_open -t -r -- ./bpftrace -e 'uprobe:/apex/com.android.art/lib64/libart.so:_ZN3art9JNIEnvExt14DeleteLocalRefEP8_jobject { printf("here\n" ); }'
perf_event_open({type=0x7 /* PERF_TYPE_??? */, size=0x88 /* PERF_ATTR_SIZE_??? */, config=0, sample_period=1, sample_type=0, read_format=0, disabled=0, inherit=0, pinned=0, exclusive=0, exclusive_user=0, exclude_kernel=0, exclude_hv=0, exclude_idle=0, mmap=0, comm=0, freq=0, inherit_stat=0, enable_on_exec=0, task=0, watermark=0, precise_ip=0 /* arbitrary skid */, mmap_data=0, sample_id_all=0, exclude_host=0, exclude_guest=0, exclude_callchain_kernel=0, exclude_callchain_user=0, mmap2=0, comm_exec=0, use_clockid=0, context_switch=0, write_backward=0, namespaces=0, wakeup_events=1, config1=0x593077ec28e0, config2=0x684d00, sample_regs_user=0, sample_regs_intr=0, aux_watermark=0, sample_max_stack=0, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 10

On the other hand bpftrace adjusts the symbol offset by some "magic" value

# strace -v -f -e trace=perf_event_open -t -r -- ./bpftrace -e 'uprobe:/apex/com.android.art/lib64/libart.so:_ZN3art9JNIEnvExt14DeleteLocalRefEP8_jobject { printf("here\n" ); }' -p 17969
perf_event_open({type=0x7 /* PERF_TYPE_??? */, size=0x88 /* PERF_ATTR_SIZE_??? */, config=0, sample_period=1, sample_type=0, read_format=0, disabled=0, inherit=0, pinned=0, exclusive=0, exclusive_user=0, exclude_kernel=0, exclude_hv=0, exclude_idle=0, mmap=0, comm=0, freq=0, inherit_stat=0, enable_on_exec=0, task=0, watermark=0, precise_ip=0 /* arbitrary skid */, mmap_data=0, sample_id_all=0, exclude_host=0, exclude_guest=0, exclude_callchain_kernel=0, exclude_callchain_user=0, mmap2=0, comm_exec=0, use_clockid=0, context_switch=0, write_backward=0, namespaces=0, wakeup_events=1, config1=0x5bc275235b00, config2=0x401300, sample_regs_user=0, sample_regs_intr=0, aux_watermark=0, sample_max_stack=0, ...}, 17969, -1, -1, PERF_FLAG_FD_CLOEXEC) = 10

In this case bpftrace used config2=0x401300 instead of 0x601300. The difference is 0x200000. This value can be calculated by subtracting adr: 0x35ff00 - elf-offset: 0x15ff00 = 0x200000.

If I subtract 0x200000 from my uprobe symbol offsets in my rust program, everything works as expected.

Question

I haven been thinking the .text section from the ELF-File at ELF-File offset 0x15ff00 is mapped into the virt. address-space at addr (0x35ff00). Why do I have to apply a mapping offset?

Any help is highly appreciated.

[0] https://github.com/aya-rs/aya [1] https://man7.org/linux/man-pages/man2/perf_event_open.2.html [2] https://bpftrace.org/


Solution

  • It seems I had a wrong assumption.

    /* Transform symbol's virtual address (absolute for binaries and relative
     * for shared libs) into file offset, which is what kernel is expecting
     * for uprobe/uretprobe attachment.
     * See Documentation/trace/uprobetracer.rst for more details. This is done
     * by looking up symbol's containing section's header and using iter's virtual
     * address (sh_addr) and corresponding file offset (sh_offset) to transform
     * sym.st_value (virtual address) into desired final file offset.
     */
    static unsigned long elf_sym_offset(struct elf_sym *sym)
    {
        return sym->sym.st_value - sym->sh.sh_addr + sym->sh.sh_offset;
    }
    

    Source: https://github.com/libbpf/libbpf/blob/master/src/elf.c#L259C1-L270C2

    According to libbpf the Kernel expects the ELF file offset for a symbol, not the virtual address.

    Applying the formula from libbpf to our example:

    • sh_addr=0x35ff00
    • st_value=0x601300
    • sh_offset=0x15ff00

    st_value The value of the associated symbol. Depending on the context, this can be an absolute value, an address, and so forth. See "Symbol Values".

    sh_addr If the section is to appear in the memory image of a process, this member gives the address at which the section's first byte should reside. Otherwise, the member contains 0.

    sh_offset The byte offset from the beginning of the file to the first byte in the section. Section type SHT_NOBITS occupies no space in the file. Its sh_offset member locates the conceptual placement in the file.

    (source: https://docs.oracle.com/cd/E19683-01/816-1386/chapter6-94076/index.html)

    Applying the "formula" from libbpf:

    0x601300 - 0x35ff00 + 0x15ff00 = 0x401300

    virt_addr_vs_elf_file_offset

    Related