Idea is to use argdist to measure latency duration of napi_poll() which returns number of packet processed (called work). Ratio of execution latency of napi_poll() to number of packets processed would give me average amount of time it took to process each packet in form of histogram.
I am using following command
argdist -H 'r:c:napi_poll():u64:$latency/$retval#avg time per packet (ns)'
which end up giving me error Failed to attach BPF to kprobe and in dmesg I get message like Could not insert probe at napi_poll+0: -2
I am just curios why I can not attach kretprobes to napi_poll() when similar trick works with net_rx_action() ?
Most of the time the Failed to attach BPF to kprobe error is caused by an inlined function. As explained in the Kprobes documentation (section Kprobes Features and Limitations), Kprobes will fail to attach if the target function was inlined. Since napi_poll is static, it might have been inlined at compile time.
You can check in kernel symbols if napi_poll was inlined or not:
$ cat /boot/System.map-`uname -r` | grep " napi_poll"
$
$ cat /boot/System.map-`uname -r` | grep " net_rx_action"
ffffffff817d8110 t net_rx_action
On my system, napi_poll is inlined while net_rx_action is not.
There are several workarounds for this problem, depending on your goal.
inline attribute to ensure napi_poll is not inlined.napi_poll that provides the same information. A function called by napi_poll can also work if it provides enough information and is not inlined itself.