Demonstrations of biosnoop, the Linux BPF/bpftrace version.


This traces block I/O, and shows the issuing process (at least, the process
that was on-CPU at the time of queue insert) and the latency of the I/O:

# ./biosnoop.bt
Attaching 4 probes...
TIME(ms)     DISK ID        COMM             PID    LAT(ms)
             MAJOR MINOR
611          259   0        bash             4179        10
611          259   0        cksum            4179         0
627          259   0        cksum            4179        15
641          259   0        cksum            4179        13
644          259   0        cksum            4179         3
658          259   0        cksum            4179        13
673          259   0        cksum            4179        14
686          259   0        cksum            4179        13
701          259   0        cksum            4179        14
710          259   0        cksum            4179         8
717          259   0        cksum            4179         6
728          259   0        cksum            4179        10
735          259   0        cksum            4179         6
751          259   0        cksum            4179        10
758          259   0        cksum            4179        17
783          259   0        cksum            4179        12
796          259   0        cksum            4179        25
802          259   0        cksum            4179        32
[...]

This output shows the cksum process was issuing block I/O, which were
completing with around 12 milliseconds of latency. Each block I/O event is
printed out, with a completion time as the first column, measured from
program start.


An example of some background flushing:

# ./biosnoop.bt
Attaching 4 probes...
TIME(ms)     DISK ID        COMM             PID    LAT(ms)
             MAJOR MINOR
2966         259   0        jbd2/nvme0n1-8   615          0
2967         259   0        jbd2/nvme0n1-8   615          0
[...]


There is another version of this tool in bcc: https://github.com/iovisor/bcc
The bcc version provides more fields.
