You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
210 lines
11 KiB
210 lines
11 KiB
Demonstrations of ext4slower, the Linux eBPF/bcc version.
|
|
|
|
|
|
ext4slower shows ext4 reads, writes, opens, and fsyncs, slower than a threshold.
|
|
For example:
|
|
|
|
# ./ext4slower
|
|
Tracing ext4 operations slower than 10 ms
|
|
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
|
|
06:35:01 cron 16464 R 1249 0 16.05 common-auth
|
|
06:35:01 cron 16463 R 1249 0 16.04 common-auth
|
|
06:35:01 cron 16465 R 1249 0 16.03 common-auth
|
|
06:35:01 cron 16465 R 4096 0 10.62 login.defs
|
|
06:35:01 cron 16464 R 4096 0 10.61 login.defs
|
|
06:35:01 cron 16463 R 4096 0 10.63 login.defs
|
|
06:35:01 cron 16465 R 2972 0 18.52 pam_env.conf
|
|
06:35:01 cron 16464 R 2972 0 18.51 pam_env.conf
|
|
06:35:01 cron 16463 R 2972 0 18.49 pam_env.conf
|
|
06:35:01 dumpsystemstat 16473 R 128 0 12.58 date
|
|
06:35:01 debian-sa1 16474 R 283 0 12.66 sysstat
|
|
06:35:01 debian-sa1 16474 R 128 0 10.39 sa1
|
|
06:35:01 dumpsystemstat 16491 R 128 0 13.22 ifconfig
|
|
06:35:01 DumpThreads 16534 R 128 0 12.78 cut
|
|
06:35:01 cron 16545 R 128 0 14.76 sendmail
|
|
06:35:01 sendmail 16545 R 274 0 10.88 dynamicmaps.cf
|
|
06:35:02 postdrop 16546 R 118 0 32.94 Universal
|
|
06:35:02 pickup 9574 R 118 0 21.02 localtime
|
|
[...]
|
|
|
|
This shows various system tasks reading from ext4. The high latency here is
|
|
due to disk I/O, as I had just evicted the file system cache for this example.
|
|
|
|
This "latency" is measured from when the operation was issued from the VFS
|
|
interface to the file system, to when it completed. This spans everything:
|
|
block device I/O (disk I/O), file system CPU cycles, file system locks, run
|
|
queue latency, etc. This is a better measure of the latency suffered by
|
|
applications reading from the file system than measuring this down at the
|
|
block device interface.
|
|
|
|
Note that this only traces the common file system operations previously
|
|
listed: other file system operations (eg, inode operations including
|
|
getattr()) are not traced.
|
|
|
|
|
|
The threshold can be provided as an argument. Eg, I/O slower than 1 ms:
|
|
|
|
# ./ext4slower 1
|
|
Tracing ext4 operations slower than 1 ms
|
|
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
|
|
06:49:17 bash 3616 R 128 0 7.75 cksum
|
|
06:49:17 cksum 3616 R 39552 0 1.34 [
|
|
06:49:17 cksum 3616 R 96 0 5.36 2to3-2.7
|
|
06:49:17 cksum 3616 R 96 0 14.94 2to3-3.4
|
|
06:49:17 cksum 3616 R 10320 0 6.82 411toppm
|
|
06:49:17 cksum 3616 R 65536 0 4.01 a2p
|
|
06:49:17 cksum 3616 R 55400 0 8.77 ab
|
|
06:49:17 cksum 3616 R 36792 0 16.34 aclocal-1.14
|
|
06:49:17 cksum 3616 R 15008 0 19.31 acpi_listen
|
|
06:49:17 cksum 3616 R 6123 0 17.23 add-apt-repository
|
|
06:49:17 cksum 3616 R 6280 0 18.40 addpart
|
|
06:49:17 cksum 3616 R 27696 0 2.16 addr2line
|
|
06:49:17 cksum 3616 R 58080 0 10.11 ag
|
|
06:49:17 cksum 3616 R 906 0 6.30 ec2-meta-data
|
|
06:49:17 cksum 3616 R 6320 0 10.00 animate.im6
|
|
06:49:17 cksum 3616 R 5680 0 18.69 anytopnm
|
|
06:49:17 cksum 3616 R 2671 0 20.27 apport-bug
|
|
06:49:17 cksum 3616 R 12566 0 16.72 apport-cli
|
|
06:49:17 cksum 3616 R 1622 0 7.95 apport-unpack
|
|
06:49:17 cksum 3616 R 10440 0 2.37 appres
|
|
06:49:17 cksum 3616 R 48112 0 5.42 whatis
|
|
06:49:17 cksum 3616 R 14832 0 6.24 apt
|
|
06:49:17 cksum 3616 R 65536 0 24.74 apt-cache
|
|
06:49:17 cksum 3616 R 27264 0 1.68 apt-cdrom
|
|
06:49:17 cksum 3616 R 23224 0 5.31 apt-extracttemplates
|
|
06:49:17 cksum 3616 R 65536 0 8.08 apt-ftparchive
|
|
06:49:17 cksum 3616 R 65536 128 2.92 apt-ftparchive
|
|
06:49:17 cksum 3616 R 65536 0 9.58 aptitude-curses
|
|
06:49:17 cksum 3616 R 65536 128 44.25 aptitude-curses
|
|
06:49:17 cksum 3616 R 65536 384 1.69 aptitude-curses
|
|
[...]
|
|
|
|
This time a cksum(1) command can be seen reading various files (from /usr/bin).
|
|
|
|
|
|
A threshold of 0 will trace all operations. Warning: the output will be
|
|
verbose, as it will include all file system cache hits.
|
|
|
|
# ./ext4slower 0
|
|
Tracing ext4 operations
|
|
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
|
|
06:58:05 supervise 1884 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1884 W 18 0 0.02 status.new
|
|
06:58:05 supervise 1884 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1884 W 18 0 0.01 status.new
|
|
06:58:05 supervise 15817 O 0 0 0.00 run
|
|
06:58:05 supervise 15817 R 92 0 0.00 run
|
|
06:58:05 supervise 15817 O 0 0 0.00 bash
|
|
06:58:05 supervise 15817 R 128 0 0.00 bash
|
|
06:58:05 supervise 15817 R 504 0 0.00 bash
|
|
06:58:05 supervise 15817 R 28 0 0.00 bash
|
|
06:58:05 supervise 15817 O 0 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15817 R 64 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15817 R 392 0 0.00 ld-2.19.so
|
|
06:58:05 run 15817 O 0 0 0.00 ld.so.cache
|
|
06:58:05 run 15817 O 0 0 0.00 libtinfo.so.5.9
|
|
06:58:05 run 15817 R 832 0 0.00 libtinfo.so.5.9
|
|
06:58:05 run 15817 O 0 0 0.00 libdl-2.19.so
|
|
06:58:05 run 15817 R 832 0 0.00 libdl-2.19.so
|
|
06:58:05 run 15817 O 0 0 0.00 libc-2.19.so
|
|
06:58:05 run 15817 R 832 0 0.00 libc-2.19.so
|
|
06:58:05 supervise 1876 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1876 W 18 0 0.01 status.new
|
|
06:58:05 supervise 1895 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1895 W 18 0 0.02 status.new
|
|
06:58:05 supervise 1876 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1876 W 18 0 0.01 status.new
|
|
06:58:05 supervise 1872 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1872 W 18 0 0.02 status.new
|
|
06:58:05 supervise 1895 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1895 W 18 0 0.01 status.new
|
|
06:58:05 supervise 15818 R 92 0 0.00 run
|
|
06:58:05 supervise 15818 O 0 0 0.00 bash
|
|
06:58:05 supervise 15818 R 128 0 0.00 bash
|
|
06:58:05 supervise 15818 R 504 0 0.00 bash
|
|
06:58:05 supervise 15818 R 28 0 0.00 bash
|
|
06:58:05 supervise 15818 O 0 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15818 R 64 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15818 R 392 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15818 O 0 0 0.00 run
|
|
06:58:05 supervise 1888 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1888 W 18 0 0.01 status.new
|
|
06:58:05 supervise 1888 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1888 W 18 0 0.02 status.new
|
|
06:58:05 supervise 15822 R 119 0 0.00 run
|
|
06:58:05 supervise 15822 O 0 0 0.00 bash
|
|
06:58:05 supervise 15822 R 128 0 0.00 bash
|
|
06:58:05 supervise 15822 R 504 0 0.00 bash
|
|
06:58:05 supervise 15822 R 28 0 0.00 bash
|
|
06:58:05 supervise 15822 O 0 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15822 R 64 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 15822 R 392 0 0.00 ld-2.19.so
|
|
06:58:05 supervise 1892 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1892 W 18 0 0.02 status.new
|
|
06:58:05 supervise 1892 O 0 0 0.00 status.new
|
|
06:58:05 supervise 1892 W 18 0 0.02 status.new
|
|
06:58:05 supervise 15820 O 0 0 0.00 run
|
|
[...]
|
|
|
|
The output now includes open operations ("O"), and writes ("W").
|
|
|
|
|
|
A -j option will print just the fields (parsable output, csv):
|
|
|
|
# ./ext4slower -j 1
|
|
ENDTIME_us,TASK,PID,TYPE,BYTES,OFFSET_b,LATENCY_us,FILE
|
|
127200712278,bash,17225,R,128,0,14329,cksum
|
|
127200722986,cksum,17225,R,3274,0,8368,command-not-found
|
|
127200735581,cksum,17225,R,65536,0,10903,libbfd-2.24-system.so
|
|
127200738482,cksum,17225,R,65536,131072,2419,libbfd-2.24-system.so
|
|
127200749226,cksum,17225,R,65536,655360,8995,libbfd-2.24-system.so
|
|
127200776273,cksum,17225,R,55080,0,25297,libbind9.so.90.0.9
|
|
127200784688,cksum,17225,R,65536,0,7873,libblas.so.3.0
|
|
127200787551,cksum,17225,R,65536,131072,2386,libblas.so.3.0
|
|
127200795524,cksum,17225,R,18624,0,4947,libcpupower.so.3.13.0-49
|
|
127200802073,cksum,17225,R,65536,0,6410,libcwidget.so.3.0.0
|
|
127200808718,cksum,17225,R,65536,131072,6181,libcwidget.so.3.0.0
|
|
127200829518,cksum,17225,R,65536,0,14213,libdns.so.100.2.2
|
|
127200832916,cksum,17225,R,65536,131072,2911,libdns.so.100.2.2
|
|
127200841044,cksum,17225,R,65536,655360,6376,libdns.so.100.2.2
|
|
127200853646,cksum,17225,R,956,0,1022,libdumbnet.la
|
|
127200857814,cksum,17225,R,61096,0,4111,libdumbnet.so.1.0.1
|
|
127200869655,cksum,17225,R,65536,0,11252,libgettextlib-0.18.3.so
|
|
127200872985,cksum,17225,R,65536,131072,2882,libgettextlib-0.18.3.so
|
|
127200883063,cksum,17225,R,65536,0,9661,libgettextsrc-0.18.3.so
|
|
127200884767,cksum,17225,R,65536,131072,1251,libgettextsrc-0.18.3.so
|
|
127200904830,cksum,17225,R,65536,0,19571,libgirepository-1.0.so.1.0.0
|
|
127200906354,cksum,17225,R,65536,131072,1080,libgirepository-1.0.so.1.0.0
|
|
127200936047,cksum,17225,R,65536,0,28674,libGraphicsMagick.a
|
|
127200939091,cksum,17225,R,65536,131072,2576,libGraphicsMagick.a
|
|
127200947295,cksum,17225,R,65536,655360,6463,libGraphicsMagick.a
|
|
127200958793,cksum,17225,R,65536,1966080,7034,libGraphicsMagick.a
|
|
[...]
|
|
|
|
This may be useful for visualizing with another tool, for example, for
|
|
producing a scatter plot of ENDTIME vs LATENCY, to look for time-based
|
|
patterns.
|
|
|
|
|
|
USAGE message:
|
|
|
|
# ./ext4slower -h
|
|
usage: ext4slower [-h] [-j] [-p PID] [min_ms]
|
|
|
|
Trace common ext4 file operations slower than a threshold
|
|
|
|
positional arguments:
|
|
min_ms minimum I/O duration to trace, in ms (default 10)
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-j, --csv just print fields: comma-separated values
|
|
-p PID, --pid PID trace this PID only
|
|
|
|
examples:
|
|
./ext4slower # trace operations slower than 10 ms (default)
|
|
./ext4slower 1 # trace operations slower than 1 ms
|
|
./ext4slower -j 1 # ... 1 ms, parsable output (csv)
|
|
./ext4slower 0 # trace all operations (warning: verbose)
|
|
./ext4slower -p 185 # trace PID 185 only
|