How to monitor the IO queue depth

Question

I am benchmarking databases above a NVMe SSD. I want to monitor the number of I/O request in the queue in this figure over time to see if the databases fully take advantage of the queues.

I have tried tools like iostat, but the avgqu-sz field is always zero. I think this may be becase NVMe SSD has a completely new storage stack rather than conventional devices (e.g., SATA SSD).

Answer 1

Solution:

cd /sys/kernel/debug/tracing/events/nvme/nvme_sq
# filter by disk name:
echo 'disk=="nvme0n1"' > filter
# enable the event:
echo 1 > enable
# check results from trace_pipe:
cat /sys/kernel/debug/tracing/trace_pipe

I suggest also enable /sys/kernel/debug/tracing/events/nvme/nvme_setup_cmd, then, you can briefly understand what is the nvme driver doing.

          <idle>-0       [002] d.h.  2558.073405: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=76, tail=76
   systemd-udevd-3805    [002] ....  2558.073454: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=3, cmdid=48, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=104856608, len=7, ctrl=0x8000, dsmgmt=7, reftag=0)
          <idle>-0       [002] d.h.  2558.073664: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=77, tail=77
   systemd-udevd-3805    [002] ....  2558.073704: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=3, cmdid=49, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=104856648, len=7, ctrl=0x8000, dsmgmt=7, reftag=0)
          <idle>-0       [002] d.h.  2558.073899: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=78, tail=78
   systemd-udevd-3805    [002] ....  2558.073938: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=3, cmdid=50, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=104854512, len=7, ctrl=0x8000, dsmgmt=7, reftag=0)
          <idle>-0       [002] d.h.  2558.074134: nvme_sq: nvme0: disk=nvme0n1, qid=3, head=79, tail=79

The explanation of each field in this output can be found here.