I am using perf profiling golang program running in docker container
I record data with
# perf record -o "perf.data" -p `PID of the container`
and read it with
# perf report -i "perf.data"
Everything is ok at the first time ,I get report like this (my elf binary name is bin) :
Samples: 15 of event 'cpu-clock', Event count (approx.): 3750000
Overhead Command Shared Object Symbol
53.33% bin bin [.] runtime.scanobject
6.67% bin [kernel.kallsyms] [k] __schedule
6.67% bin bin [.] runtime.getStackMap
6.67% bin bin [.] runtime.getempty
6.67% bin bin [.] runtime.gopark
6.67% bin bin [.] runtime.greyobject
6.67% bin bin [.] runtime.scanblock
6.67% bin bin [.] runtime.unlock
But after I stop the container and run the report command again ,all symbols in my program become hex address (kernel symbol still resolved)
Samples: 15 of event 'cpu-clock', Event count (approx.): 3750000
Overhead Command Shared Object Symbol
33.33% bin bin [.] 0x000000000001f7ad
13.33% bin bin [.] 0x000000000001f7b0
6.67% bin [kernel.kallsyms] [k] __schedule
6.67% bin bin [.] 0x000000000000ac1a
6.67% bin bin [.] 0x000000000001f497
6.67% bin bin [.] 0x000000000001f7d6
6.67% bin bin [.] 0x000000000001fc82
6.67% bin bin [.] 0x00000000000242fd
6.67% bin bin [.] 0x0000000000035bf0
6.67% bin bin [.] 0x000000000004d5a9
I try to find the build id of the binary file but don't get any clue:
# perf buildid-list -i perf.data
38b62c386e959108a2ff514c04f7df8f39e443f9 [kernel.kallsyms]
78fa50e860a2bb2b44f03a6a0a6f99735a8b599b [vdso]
as @osgx 's suggestion I run command below
#perf script -D |grep PERF_RECORD_MMAP2|head
Failed to open /bin, continuing without symbols
0 0x2b98 [0x60]: PERF_RECORD_MMAP2 15956/15956: [0x400000(0x852000) @ 0 fc:01 656204 7434654850458070581]: r-xp /bin
0 0x2bf8 [0x60]: PERF_RECORD_MMAP2 15956/15956: [0x7ffca95a8000(0x2000) @ 0 00:00 0 7434654850458070581]: r-xp [vdso]
0 0x2c58 [0x68]: PERF_RECORD_MMAP2 15956/15956: [0xffffffffff600000(0x1000) @ 0 00:00 0 7434654850458070581]: r-xp [vsyscall]
0 0x2ce8 [0x60]: PERF_RECORD_MMAP2 15956/16020: [0x400000(0x852000) @ 0 fc:01 656204 7434654850458070581]: r-xp /bin
0 0x2d48 [0x60]: PERF_RECORD_MMAP2 15956/16020: [0x7ffca95a8000(0x2000) @ 0 00:00 0 7434654850458070581]: r-xp [vdso]
0 0x2da8 [0x68]: PERF_RECORD_MMAP2 15956/16020: [0xffffffffff600000(0x1000) @ 0 00:00 0 7434654850458070581]: r-xp [vsyscall]
0 0x2e38 [0x60]: PERF_RECORD_MMAP2 15956/16021: [0x400000(0x852000) @ 0 fc:01 656204 7434654850458070581]: r-xp /bin
0 0x2e98 [0x60]: PERF_RECORD_MMAP2 15956/16021: [0x7ffca95a8000(0x2000) @ 0 00:00 0 7434654850458070581]: r-xp [vdso]
0 0x2ef8 [0x68]: PERF_RECORD_MMAP2 15956/16021: [0xffffffffff600000(0x1000) @ 0 00:00 0 7434654850458070581]: r-xp [vsyscall]
0 0x2f88 [0x60]: PERF_RECORD_MMAP2 15956/16022: [0x400000(0x852000) @ 0 fc:01 656204 7434654850458070581]: r-xp /bin
Why is that happen? Is there any solution let perf resolve symbol after container stop ?
Here is my environment:
perf version 4.15.18
Ubuntu 18.04 LTS (GNU/Linux 4.15.0-23-generic x86_64)
docker version 18.06.1-ce
My container Dockerfile
FROM scratch
COPY artifact/bin /
ENTRYPOINT ["/bin"]
Why is that happen?
perf tool is doing some kind of searching for binary files. In perf.data there is mmap recorded (seen with perf script -D |grep PERF_RECORD_MMAP2|head command) with path of file mapped with EXEC permission. Main binary file is mapped too, but at moment of mapping the path was relative to the container. In container this file has /bin path as you put it with COPY artifact/bin /. But perf report is started outside the container and tries to resolve /bin path outside of container fs cgroup/namespace. In host system /bin is directory, not file. When you starting perf report while container is still running it may have some heuristics to get executable file with help of /proc/$PID/exe special fs, which probably has success in accessing of files inside container fs namespace.
Is there any solution let perf resolve symbol after container stop ?
I have not too beautiful suggestion (in comments): put the artifact binary into the container at the path which will be equal string to real placement of the binary. For example, if you have the binary as /home/pexie/project1/artifact/bin; in dockerfile create the directory /home/pexie/project1/artifact/ and put the binary into this directory.