You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
132 lines
4.4 KiB
132 lines
4.4 KiB
4 months ago
|
Demonstrations of tplist.
|
||
|
|
|
||
|
|
|
||
|
|
tplist displays kernel tracepoints and USDT probes, including their
|
||
|
|
format. It can be used to discover probe points for use with the trace
|
||
|
|
and argdist tools. Kernel tracepoints are scattered around the kernel
|
||
|
|
and provide valuable static tracing on block and network I/O, scheduling,
|
||
|
|
power events, and many other subjects. USDT probes are placed in libraries
|
||
|
|
(such as libc) and executables (such as node) and provide static tracing
|
||
|
|
information that can (optionally) be turned on and off at runtime.
|
||
|
|
|
||
|
|
For example, suppose you want to discover which USDT probes a particular
|
||
|
|
executable contains. Just run tplist on that executable (or library):
|
||
|
|
|
||
|
|
$ tplist -l basic_usdt
|
||
|
|
/home/vagrant/basic_usdt basic_usdt:start_main
|
||
|
|
/home/vagrant/basic_usdt basic_usdt:loop_iter
|
||
|
|
/home/vagrant/basic_usdt basic_usdt:end_main
|
||
|
|
|
||
|
|
The loop_iter probe sounds interesting. How many arguments are available?
|
||
|
|
|
||
|
|
$ tplist '*loop_iter' -l basic_usdt -v
|
||
|
|
basic_usdt:loop_iter [sema 0x601036]
|
||
|
|
2 location(s)
|
||
|
|
2 argument(s)
|
||
|
|
|
||
|
|
This output indicates that the loop_iter probe is used in two locations
|
||
|
|
in the basic_usdt executable, and that it has two arguments. Fortunately,
|
||
|
|
the argdist and trace tools understand the probe format and can print out
|
||
|
|
the arguments automatically -- you can refer to them as arg1, arg2, and
|
||
|
|
so on.
|
||
|
|
|
||
|
|
Try to explore with some common libraries on your system and see if they
|
||
|
|
contain UDST probes. Here are two examples you might find interesting:
|
||
|
|
|
||
|
|
$ tplist -l pthread # list probes in libpthread
|
||
|
|
/lib64/libpthread.so.0 libpthread:pthread_start
|
||
|
|
/lib64/libpthread.so.0 libpthread:pthread_create
|
||
|
|
/lib64/libpthread.so.0 libpthread:pthread_join
|
||
|
|
/lib64/libpthread.so.0 libpthread:pthread_join_ret
|
||
|
|
/lib64/libpthread.so.0 libpthread:mutex_init
|
||
|
|
... more output truncated
|
||
|
|
|
||
|
|
$ tplist -l c # list probes in libc
|
||
|
|
/lib64/libc.so.6 libc:setjmp
|
||
|
|
/lib64/libc.so.6 libc:longjmp
|
||
|
|
/lib64/libc.so.6 libc:longjmp_target
|
||
|
|
/lib64/libc.so.6 libc:memory_arena_reuse_free_list
|
||
|
|
/lib64/libc.so.6 libc:memory_heap_new
|
||
|
|
... more output truncated
|
||
|
|
|
||
|
|
tplist also understands kernel tracepoints, and can list their format
|
||
|
|
as well. For example, let's look for all block I/O-related tracepoints:
|
||
|
|
|
||
|
|
# tplist 'block*'
|
||
|
|
block:block_touch_buffer
|
||
|
|
block:block_dirty_buffer
|
||
|
|
block:block_rq_abort
|
||
|
|
block:block_rq_requeue
|
||
|
|
block:block_rq_complete
|
||
|
|
block:block_rq_insert
|
||
|
|
block:block_rq_issue
|
||
|
|
block:block_bio_bounce
|
||
|
|
block:block_bio_complete
|
||
|
|
block:block_bio_backmerge
|
||
|
|
block:block_bio_frontmerge
|
||
|
|
block:block_bio_queue
|
||
|
|
block:block_getrq
|
||
|
|
block:block_sleeprq
|
||
|
|
block:block_plug
|
||
|
|
block:block_unplug
|
||
|
|
block:block_split
|
||
|
|
block:block_bio_remap
|
||
|
|
block:block_rq_remap
|
||
|
|
|
||
|
|
The block:block_rq_complete tracepoints sounds interesting. Let's print
|
||
|
|
its format to see what we can trace with argdist and trace:
|
||
|
|
|
||
|
|
$ tplist -v block:block_rq_complete
|
||
|
|
block:block_rq_complete
|
||
|
|
dev_t dev;
|
||
|
|
sector_t sector;
|
||
|
|
unsigned int nr_sector;
|
||
|
|
int errors;
|
||
|
|
char rwbs[8];
|
||
|
|
|
||
|
|
The dev, sector, nr_sector, etc. variables can now all be used in probes
|
||
|
|
you specify with argdist or trace.
|
||
|
|
|
||
|
|
|
||
|
|
For debugging USDT probes, it is sometimes useful to see the exact locations
|
||
|
|
and arguments of the probes, including the registers or global variables from
|
||
|
|
which their values are coming from. In super-verbose mode, tplist will print
|
||
|
|
this information (note the -vv):
|
||
|
|
|
||
|
|
$ tplist -vv -l c *alloc*
|
||
|
|
libc:memory_malloc_retry [sema 0x0]
|
||
|
|
location #0 /lib64/libc.so.6 0x835c0
|
||
|
|
argument #0 8 unsigned bytes @ bp
|
||
|
|
location #1 /lib64/libc.so.6 0x83778
|
||
|
|
argument #0 8 unsigned bytes @ bp
|
||
|
|
location #2 /lib64/libc.so.6 0x85a50
|
||
|
|
argument #0 8 unsigned bytes @ bp
|
||
|
|
libc:memory_realloc_retry [sema 0x0]
|
||
|
|
location #0 /lib64/libc.so.6 0x84b90
|
||
|
|
argument #0 8 unsigned bytes @ r13
|
||
|
|
argument #1 8 unsigned bytes @ bp
|
||
|
|
location #1 /lib64/libc.so.6 0x85cf0
|
||
|
|
argument #0 8 unsigned bytes @ r13
|
||
|
|
argument #1 8 unsigned bytes @ bp
|
||
|
|
libc:memory_calloc_retry [sema 0x0]
|
||
|
|
location #0 /lib64/libc.so.6 0x850f0
|
||
|
|
argument #0 8 unsigned bytes @ bp
|
||
|
|
|
||
|
|
|
||
|
|
USAGE message:
|
||
|
|
|
||
|
|
$ tplist -h
|
||
|
|
usage: tplist.py [-h] [-p PID] [-l LIB] [-v] [filter]
|
||
|
|
|
||
|
|
Display kernel tracepoints or USDT probes and their formats.
|
||
|
|
|
||
|
|
positional arguments:
|
||
|
|
filter A filter that specifies which probes/tracepoints to print
|
||
|
|
|
||
|
|
optional arguments:
|
||
|
|
-h, --help show this help message and exit
|
||
|
|
-p PID, --pid PID List USDT probes in the specified process
|
||
|
|
-l LIB, --lib LIB List USDT probes in the specified library or executable
|
||
|
|
-v Increase verbosity level (print variables, arguments, etc.)
|
||
|
|
|