高性能:BPF performance Tools 学习笔记-1

  • 2020 年 2 月 17 日
  • 筆記

本系统列内容来自 性能优化大神Brendan Gregg 出的新书, 目前国内还没有引入。

为了抢先体验到大佬的又一部力作,我直接去oreilly官方网站的看的电子书。

Brendan Gregg 书里提到 Netflix和Facebook都将这些工具作为生产服务器的标配工具,因此我们也可以在经过充分的测试后,考虑将这些工具做到初始化镜像模板里面。

废话不说。直接放图,每个level都有对应的perf分析工具:

每个工具的作用如下:

•argdist:     Display function parameter values as a histogram or frequency count. 

•bashreadline:     Print entered bash commands system wide. 

•biolatency:     Summarize block device I/O latency as a histogram. 

•biotop:     Top for disks:     Summarize block device I/O by process. 

•biosnoop:     Trace block device I/O with PID and latency. 

•bitesize:     Show per process I/O size histogram. 

•bpflist:     Display processes with active BPF programs and maps. 

•btrfsdist:     Summarize btrfs operation latency distribution as a histogram. 

•btrfsslower:     Trace slow btrfs operations. 

•capable:     Trace security capability checks. 

•cachestat:     Trace page cache hit/miss ratio. 

•cachetop:     Trace page cache hit/miss ratio by processes. 

•compactsnoop:     Trace compact zone events with PID and latency. 

•cpudist:     Summarize on- and off-CPU time per task as a histogram. Examples

•cpuunclaimed:     Sample CPU run queues and calculate unclaimed idle CPU. Examples

•criticalstat:     Trace and report long atomic critical sections in the kernel. Examples

•dbslower:     Trace MySQL/PostgreSQL queries slower than a threshold. 

•dbstat:     Summarize MySQL/PostgreSQL query latency as a histogram. 

•dcsnoop:     Trace directory entry cache (dcache) lookups. 

•dcstat:     Directory entry cache (dcache) stats. 

•deadlock:     Detect potential deadlocks on a running process. 

•drsnoop:     Trace direct reclaim events with PID and latency. 

•execsnoop:     Trace new processes via exec() syscalls. 

•exitsnoop:     Trace process termination (exit and fatal signals). 

•ext4dist:     Summarize ext4 operation latency distribution as a histogram. 

•ext4slower:     Trace slow ext4 operations. 

•filelife:     Trace the lifespan of short-lived files. 

•fileslower:     Trace slow synchronous file reads and writes. 

•filetop:     File reads and writes by filename and process. Top for files. 

•funccount:     Count kernel function calls. 

•funclatency:     Time functions and show their latency distribution. 

•funcslower:     Trace slow kernel or user function calls. 

•gethostlatency:     Show latency for getaddrinfo/gethostbyname[2] calls. 

•hardirqs:     Measure hard IRQ (hard interrupt) event time. 

•inject:     Targeted error injection with call chain and predicates 

•killsnoop:     Trace signals issued by the kill() syscall. 

•klockstat:     Traces kernel mutex lock events and display locks statistics. 

•llcstat:     Summarize CPU cache references and misses by process. 

•mdflush:     Trace md flush events. 

•memleak:     Display outstanding memory allocations to find memory leaks. 

•mountsnoop:     Trace mount and umount syscalls system-wide. 

•mysqld_qslower:     Trace MySQL server queries slower than a threshold. 

•nfsslower:     Trace slow NFS operations. 

•nfsdist:     Summarize NFS operation latency distribution as a histogram. 

•offcputime:     Summarize off-CPU time by kernel stack trace. 

•offwaketime:     Summarize blocked time by kernel off-CPU stack and waker stack. 

•oomkill:     Trace the out-of-memory (OOM) killer. 

•opensnoop:     Trace open() syscalls. 

•pidpersec:     Count new processes (via fork). 

•profile:     Profile CPU usage by sampling stack traces at a timed interval. 

•reset-trace:     Reset the state of tracing. Maintenance tool only. 

•runqlat:     Run queue (scheduler) latency as a histogram. 

•runqlen:     Run queue length as a histogram. 

•runqslower:     Trace long process scheduling delays. 

•shmsnoop:     Trace System V shared memory syscalls. 

•sofdsnoop:     Trace FDs passed through unix sockets. 

•slabratetop:     Kernel SLAB/SLUB memory cache allocation rate top. 

•softirqs:     Measure soft IRQ (soft interrupt) event time. 

•solisten:     Trace TCP socket listen. 

•sslsniff:     Sniff OpenSSL written and readed data. 

•stackcount:     Count kernel function calls and their stack traces. 

•syncsnoop:     Trace sync() syscall. 

•syscount:     Summarize syscall counts and latencies. 

•tcpaccept:     Trace TCP passive connections (accept()). 

•tcpconnect:     Trace TCP active connections (connect()). 

•tcpconnlat:     Trace TCP active connection latency (connect()). 

•tcpdrop:     Trace kernel-based TCP packet drops with details. 

•tcplife:     Trace TCP sessions and summarize lifespan. 

•tcpretrans:     Trace TCP retransmits and TLPs. 

•tcpstates:     Trace TCP session state changes with durations. 

•tcpsubnet:     Summarize and aggregate TCP send by subnet. 

•tcptop:     Summarize TCP send/recv throughput by host. Top for TCP. 

•tcptracer:     Trace TCP established connections (connect(), accept(), close()). 

•tplist:     Display kernel tracepoints or USDT probes and their formats. 

•trace:     Trace arbitrary functions, with filters. 

•ttysnoop:     Watch live output from a tty or pts device. 

•ucalls:     Summarize method calls or Linux syscalls in high-level languages. 

•uflow:     Print a method flow graph in high-level languages. 

•ugc:     Trace garbage collection events in high-level languages. 

•uobjnew:     Summarize object allocation events by object type and number of bytes allocated. 

•ustat:     Collect events such as GCs, thread creations, object allocations, exceptions and more in high-level languages. 

•uthreads:     Trace thread creation events in Java and raw pthreads. 

•vfscount vfscount.c:     Count VFS calls. 

•vfsstat vfsstat.c:     Count some VFS calls, with column output. 

•wakeuptime:     Summarize sleep to wakeup time by waker kernel stack. 

•xfsdist:     Summarize XFS operation latency distribution as a histogram. 

•xfsslower:     Trace slow XFS operations. 

•zfsdist:     Summarize ZFS operation latency distribution as a histogram. 

•zfsslower:     Trace slow ZFS operations. 

上面提到的工具非常多,我们可以稍微精简些。如下:

不同level推荐的分析工具:

主题

推荐使用的工具

Debugging / multi-purpose

trace,   argdist, funccount, stackcount, opensnoop

CPUs

execsnoop,   runqlat, runqlen, cpudist, profile, offcputime, syscount, softirq, hardirq

Memory

memleak

File systems

opensnoop,   filelife, vfsstatt, fileslower, cachestat, writeback, dcstat, xfsslower,   xfsdist, ext4dist

Disk I/O

biolatency,   biosnoop, biotop, bitesize

Networking

tcpconnect,   tcpaccept, tcplife, tcpretrans

Security

capable

Languages

javastat,   javacalls, javathreads, javaflow, javagc

Applications

mysqld_qslower,   signals, killsnoop

Kernel

wakeuptime,   offwaketime