You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
126 lines
3.6 KiB
126 lines
3.6 KiB
4 months ago
|
.TH funcslower 8 "2017-03-30" "USER COMMANDS"
|
||
|
|
.SH NAME
|
||
|
|
funcslower \- Trace slow kernel or user function calls.
|
||
|
|
.SH SYNOPSIS
|
||
|
|
.B funcslower [\-hf] [\-p PID] [\-U | \-K] [-m MIN_MS] [-u MIN_US] [-a ARGUMENTS] [-T] [-t] [-v] function [function ...]
|
||
|
|
.SH DESCRIPTION
|
||
|
|
This script traces a kernel or user function's entry and return points, and
|
||
|
|
prints a message when the function's latency exceeded the specified threshold.
|
||
|
|
Multiple functions are supported, and you can mix kernel functions with user
|
||
|
|
functions in different libraries.
|
||
|
|
|
||
|
|
WARNING: See the OVERHEAD section.
|
||
|
|
|
||
|
|
By default, a minimum millisecond threshold of 1 is used. Recursive functions
|
||
|
|
are not supported: only the inner-most recursive invocation will be traced.
|
||
|
|
|
||
|
|
Since this uses BPF, only the root user can use this tool.
|
||
|
|
.SH REQUIREMENTS
|
||
|
|
CONFIG_BPF and bcc.
|
||
|
|
.SH OPTIONS
|
||
|
|
\-p PID
|
||
|
|
Trace this PID only.
|
||
|
|
.TP
|
||
|
|
\-m MIN_NS
|
||
|
|
Minimum duration to trace, in milliseconds. Default is 1 ms.
|
||
|
|
.TP
|
||
|
|
\-u MIN_US
|
||
|
|
Minimum duration to trace, in microseconds.
|
||
|
|
.TP
|
||
|
|
\-a ARGUMENTS
|
||
|
|
Print the function's arguments, up to 6.
|
||
|
|
.TP
|
||
|
|
\-T
|
||
|
|
Print a HH:MM:SS timestamp with each entry.
|
||
|
|
.TP
|
||
|
|
\-t
|
||
|
|
Print a seconds timestamp with each entry, at microsecond resolution.
|
||
|
|
.TP
|
||
|
|
\-f
|
||
|
|
Print output in folded stack format.
|
||
|
|
.TP
|
||
|
|
\-U
|
||
|
|
Show stacks from user space only (no kernel space stacks).
|
||
|
|
.TP
|
||
|
|
\-K
|
||
|
|
Show stacks from kernel space only (no user space stacks).
|
||
|
|
.TP
|
||
|
|
\-v
|
||
|
|
Print the resulting BPF program, for debugging purposes.
|
||
|
|
.TP
|
||
|
|
function
|
||
|
|
The function to trace -- multiple functions are supported. If a plain function
|
||
|
|
name is provided, the function is assumed to be a kernel function. For user
|
||
|
|
functions, provide the library name and the function name, e.g. bash:readline
|
||
|
|
or c:malloc.
|
||
|
|
.SH EXAMPLES
|
||
|
|
.TP
|
||
|
|
Trace vfs_write calls slower than 1ms:
|
||
|
|
#
|
||
|
|
.B funcslower vfs_write
|
||
|
|
.TP
|
||
|
|
Trace open() calls in libc slower than 10us:
|
||
|
|
#
|
||
|
|
.B funcslower \-u 10 c:open
|
||
|
|
.TP
|
||
|
|
Trace both malloc() and free() slower than 10us, in pid 135 only:
|
||
|
|
#
|
||
|
|
.B funcslower \-p 135 \-u 10 c:malloc c:free
|
||
|
|
.TP
|
||
|
|
Trace the write syscall and print its first 4 arguments:
|
||
|
|
#
|
||
|
|
.B funcslower -a 4 SyS_write
|
||
|
|
.TP
|
||
|
|
Trace opens from libc and print the user and kernel stack frames:
|
||
|
|
#
|
||
|
|
.B funcslower -UK c:open
|
||
|
|
.SH FIELDS
|
||
|
|
.TP
|
||
|
|
TIME
|
||
|
|
Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds
|
||
|
|
at microsecond-accuracy from the first event seen.
|
||
|
|
.TP
|
||
|
|
COMM
|
||
|
|
Process name.
|
||
|
|
.TP
|
||
|
|
PID
|
||
|
|
Process ID.
|
||
|
|
.TP
|
||
|
|
LAT
|
||
|
|
Latency of the operation in either microseconds (us) or milliseconds (ms).
|
||
|
|
.TP
|
||
|
|
RVAL
|
||
|
|
The return value from the function. Often useful for diagnosing a relationship
|
||
|
|
between slow and failed function calls.
|
||
|
|
.TP
|
||
|
|
FUNC
|
||
|
|
The function name, followed by its arguments if requested.
|
||
|
|
.SH OVERHEAD
|
||
|
|
Depending on the function(s) being traced, overhead can become severe. For
|
||
|
|
example, tracing a common function like malloc() can slow down a C/C++ program
|
||
|
|
by a factor of 2 or more. On the other hand, tracing a low-frequency event like
|
||
|
|
the SyS_setreuid() function will probably not be as prohibitive, and in fact
|
||
|
|
negligible for functions that are called up to 100-1000 times per second.
|
||
|
|
|
||
|
|
You should first use the funclatency and argdist tools for investigation,
|
||
|
|
because they summarize data in-kernel and have a much lower overhead than this
|
||
|
|
tool. To get a general idea of the number of times a particular function is
|
||
|
|
called (and estimate the overhead), use the funccount tool, e.g.:
|
||
|
|
.PP
|
||
|
|
# funccount c:open
|
||
|
|
.SH SOURCE
|
||
|
|
This is from bcc.
|
||
|
|
.IP
|
||
|
|
https://github.com/iovisor/bcc
|
||
|
|
.PP
|
||
|
|
Also look in the bcc distribution for a companion _examples.txt file containing
|
||
|
|
example usage, output, and commentary for this tool.
|
||
|
|
.SH OS
|
||
|
|
Linux
|
||
|
|
.SH STABILITY
|
||
|
|
Unstable - in development.
|
||
|
|
.SH AUTHOR
|
||
|
|
Sasha Goldshtein
|
||
|
|
.SH SEE ALSO
|
||
|
|
funccount(8), funclatency(8), argdist(8), trace(8)
|