Brendan Gregg, profile picture
Uploaded byBrendan Gregg
PDF, PPTX5,137 views

OSSNA 2017 Performance Analysis Superpowers with Linux BPF

The document presents an overview of the performance analysis capabilities of Linux's eBPF (enhanced Berkeley Packet Filter) introduced by Brendan Gregg in September 2017. It covers various aspects such as the history of eBPF and tracing tools, including BCC (BPF Compiler Collection) and its applications in performance monitoring, latency analysis, and dynamic tracing of network and file system operations. Key takeaways include the understanding of Linux tracing, usage of BPF tools, and insights into future development areas.

Embed presentation

Download as PDF, PPTX
Performance Analysis Superpowers with Linux BPF Brendan Gregg Sep 2017
bcc/BPF tools
DEMO
Agenda 1. eBPF & bcc 2. bcc/BPF CLI Tools 3. bcc/BPF Visualizations
Take aways 1.  Understand Linux tracing and enhanced BPF 2.  How to use BPF tools 3.  Areas of future development
Who at Ne/lix will use BPF?
BPF Introducing enhanced BPF for tracing: kernel-level software
Ye Olde BPF Berkeley Packet Filter # tcpdump host 127.0.0.1 and port 22 -d (000) ldh [12] (001) jeq #0x800 jt 2 jf 18 (002) ld [26] (003) jeq #0x7f000001 jt 6 jf 4 (004) ld [30] (005) jeq #0x7f000001 jt 6 jf 18 (006) ldb [23] (007) jeq #0x84 jt 10 jf 8 (008) jeq #0x6 jt 10 jf 9 (009) jeq #0x11 jt 10 jf 18 (010) ldh [20] (011) jset #0x1fff jt 18 jf 12 (012) ldxb 4*([14]&0xf) (013) ldh [x + 14] [...] User-defined bytecode executed by an in-kernel sandboxed virtual machine Steven McCanne and Van Jacobson, 1993 2 x 32-bit registers & scratch memory Optimizes packet filter performance
Enhanced BPF aka eBPF or just "BPF" Alexei Starovoitov, 2014+ 10 x 64-bit registers maps (hashes) actions
BPF for Tracing, Internals BPF bytecode Observability Program Kernel tracepoints kprobes uprobes BPF maps per-event data statistics verifier output static tracing dynamic tracing async copy perf_events sampling, PMCs BPF program event config attach load Enhanced BPF is also now used for SDNs, DDOS mitigation, intrusion detection, container security, …
Dynamic Tracing
1999: Kerninst http://www.paradyn.org/html/kerninst.html
Event Tracing Efficiency send receive tcpdump Kernel buffer file system 1. read 2. dump Analyzer 1. read 2. process 3. print disks Old way: packet capture New way: dynamic tracing Tracer 1. configure 2. read tcp_retransmit_skb() E.g., tracing TCP retransmits
Linux Events & BPF Support Linux 4.3 Linux 4.7 Linux 4.9 Linux 4.9 Linux 4.1 BPF stacks Linux 4.6 BPF output Linux 4.4 (version BPF support arrived)
A Linux Tracing Timeline -  1990’s: Static tracers, prototype dynamic tracers -  2000: LTT + DProbes (dynamic tracing; not integrated) -  2004: kprobes (2.6.9) -  2005: DTrace (not Linux), SystemTap (out-of-tree) -  2008: ftrace (2.6.27) -  2009: perf_events (2.6.31) -  2009: tracepoints (2.6.32) -  2010-2017: ftrace & perf_events enhancements -  2012: uprobes (3.5) -  2014-2017: enhanced BPF patches: supporting tracing events -  2016-2017: ftrace hist triggers also: LTTng, ktap, sysdig, ...
BCC Introducing BPF Complier Collection: user-level front-end
bcc •  BPF Compiler Collection –  https://github.com/iovisor/bcc –  Lead developer: Brenden Blanco •  Includes tracing tools •  Provides BPF front-ends: –  Python –  Lua –  C++ –  C helper libraries –  golang (gobpf) BPF Python Events Kernel lua bcc front-ends bcc tool bcc tool … … user kernel Tracing layers:
Raw BPF samples/bpf/sock_example.c 87 lines truncated
C/BPF samples/bpf/tracex1_kern.c 58 lines truncated
bcc/BPF (C & Python) bcc examples/tracing/bitehist.py enBre program
bpftrace hHps://github.com/ajor/bpJrace enBre program
The Tracing Landscape, Sep 2017 Scope & Capability Easeofuse sysdig perf ftrace C/BPF ktap stap Stage of Development (my opinion) dtrace4L. (brutal)(lessbrutal) (alpha) (mature) bcc/BPF ply/BPF Raw BPF LTTng (hist triggers)recent changes (many) bpftrace
BCC/BPF CLI Tools Performance Analysis
Pre-BPF: Linux Perf Analysis in 60s 1.  uptime 2.  dmesg -T | tail 3.  vmstat 1 4.  mpstat -P ALL 1 5.  pidstat 1 6.  iostat -xz 1 7.  free -m 8.  sar -n DEV 1 9.  sar -n TCP,ETCP 1 10.  top hHp://techblog.ne/lix.com/2015/11/linux-performance-analysis-in-60s.html
bcc Installation •  https://github.com/iovisor/bcc/blob/master/INSTALL.md •  eg, Ubuntu Xenial: –  Also available as an Ubuntu snap –  Ubuntu 16.04 is good, 16.10 better: more tools work •  Installs many tools –  In /usr/share/bcc/tools, and …/tools/old for older kernels # echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" |
 sudo tee /etc/apt/sources.list.d/iovisor.list # sudo apt-get update # sudo apt-get install bcc-tools
bcc General Performance Checklist 1.  execsnoop 2.  opensnoop 3.  ext4slower (…) 4.  biolatency 5.  biosnoop 6.  cachestat 7.  tcpconnect 8.  tcpaccept 9.  tcpretrans 10.  gethostlatency 11.  runqlat 12.  profile
Discover short-lived process issues using execsnoop # execsnoop -t TIME(s) PCOMM PID PPID RET ARGS 0.031 dirname 23832 23808 0 /usr/bin/dirname /apps/tomcat/bin/catalina.sh 0.888 run 23833 2344 0 ./run 0.889 run 23833 2344 -2 /command/bash 0.889 run 23833 2344 -2 /usr/local/bin/bash 0.889 run 23833 2344 -2 /usr/local/sbin/bash 0.889 bash 23833 2344 0 /bin/bash 0.894 svstat 23835 23834 0 /command/svstat /service/nflx-httpd 0.894 perl 23836 23834 0 /usr/bin/perl -e $l=<>;$l=~/(d+) sec/;print $1||0; 0.899 ps 23838 23837 0 /bin/ps --ppid 1 -o pid,cmd,args 0.900 grep 23839 23837 0 /bin/grep org.apache.catalina 0.900 sed 23840 23837 0 /bin/sed s/^ *//; 0.900 cut 23841 23837 0 /usr/bin/cut -d -f 1 0.901 xargs 23842 23837 0 /usr/bin/xargs 0.912 xargs 23843 23842 -2 /command/echo 0.912 xargs 23843 23842 -2 /usr/local/bin/echo 0.912 xargs 23843 23842 -2 /usr/local/sbin/echo 0.912 echo 23843 23842 0 /bin/echo [...] Efficient: only traces exec()
Discover short-lived process issues using execsnoop # execsnoop -t TIME(s) PCOMM PID PPID RET ARGS 0.031 dirname 23832 23808 0 /usr/bin/dirname /apps/tomcat/bin/catalina.sh 0.888 run 23833 2344 0 ./run 0.889 run 23833 2344 -2 /command/bash 0.889 run 23833 2344 -2 /usr/local/bin/bash 0.889 run 23833 2344 -2 /usr/local/sbin/bash 0.889 bash 23833 2344 0 /bin/bash 0.894 svstat 23835 23834 0 /command/svstat /service/nflx-httpd 0.894 perl 23836 23834 0 /usr/bin/perl -e $l=<>;$l=~/(d+) sec/;print $1||0; 0.899 ps 23838 23837 0 /bin/ps --ppid 1 -o pid,cmd,args 0.900 grep 23839 23837 0 /bin/grep org.apache.catalina 0.900 sed 23840 23837 0 /bin/sed s/^ *//; 0.900 cut 23841 23837 0 /usr/bin/cut -d -f 1 0.901 xargs 23842 23837 0 /usr/bin/xargs 0.912 xargs 23843 23842 -2 /command/echo 0.912 xargs 23843 23842 -2 /usr/local/bin/echo 0.912 xargs 23843 23842 -2 /usr/local/sbin/echo 0.912 echo 23843 23842 0 /bin/echo [...] Efficient: only traces exec()
Exonerate or confirm storage latency outliers with ext4slower # /usr/share/bcc/tools/ext4slower 1 Tracing ext4 operations slower than 1 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 17:31:42 postdrop 15523 S 0 0 2.32 5630D406E4 17:31:42 cleanup 15524 S 0 0 1.89 57BB7406EC 17:32:09 titus-log-ship 19735 S 0 0 1.94 slurper_checkpoint.db 17:35:37 dhclient 1061 S 0 0 3.32 dhclient.eth0.leases 17:35:39 systemd-journa 504 S 0 0 26.62 system.journal 17:35:39 systemd-journa 504 S 0 0 1.56 system.journal 17:35:39 systemd-journa 504 S 0 0 1.73 system.journal 17:35:45 postdrop 16187 S 0 0 2.41 C0369406E4 17:35:45 cleanup 16188 S 0 0 6.52 C1B90406EC […] Tracing at the file system is a more reliable and complete indicator than measuring disk I/O latency Also: btrfsslower, xfsslower, zfsslower
Exonerate or confirm storage latency outliers with ext4slower # /usr/share/bcc/tools/ext4slower 1 Tracing ext4 operations slower than 1 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 17:31:42 postdrop 15523 S 0 0 2.32 5630D406E4 17:31:42 cleanup 15524 S 0 0 1.89 57BB7406EC 17:32:09 titus-log-ship 19735 S 0 0 1.94 slurper_checkpoint.db 17:35:37 dhclient 1061 S 0 0 3.32 dhclient.eth0.leases 17:35:39 systemd-journa 504 S 0 0 26.62 system.journal 17:35:39 systemd-journa 504 S 0 0 1.56 system.journal 17:35:39 systemd-journa 504 S 0 0 1.73 system.journal 17:35:45 postdrop 16187 S 0 0 2.41 C0369406E4 17:35:45 cleanup 16188 S 0 0 6.52 C1B90406EC […] Tracing at the file system is a more reliable and complete indicator than measuring disk I/O latency Also: btrfsslower, xfsslower, zfsslower
Identify multimodal disk I/O latency and outliers with biolatency # biolatency -mT 10 Tracing block device I/O... Hit Ctrl-C to end. 19:19:04 msecs : count distribution 0 -> 1 : 238 |********* | 2 -> 3 : 424 |***************** | 4 -> 7 : 834 |********************************* | 8 -> 15 : 506 |******************** | 16 -> 31 : 986 |****************************************| 32 -> 63 : 97 |*** | 64 -> 127 : 7 | | 128 -> 255 : 27 |* | 19:19:14 msecs : count distribution 0 -> 1 : 427 |******************* | 2 -> 3 : 424 |****************** | […] Average latency (iostat/sar) may not be represen[[ve with mul[ple modes or outliers The "count" column is summarized in-kernel
Identify multimodal disk I/O latency and outliers with biolatency # biolatency -mT 10 Tracing block device I/O... Hit Ctrl-C to end. 19:19:04 msecs : count distribution 0 -> 1 : 238 |********* | 2 -> 3 : 424 |***************** | 4 -> 7 : 834 |********************************* | 8 -> 15 : 506 |******************** | 16 -> 31 : 986 |****************************************| 32 -> 63 : 97 |*** | 64 -> 127 : 7 | | 128 -> 255 : 27 |* | 19:19:14 msecs : count distribution 0 -> 1 : 427 |******************* | 2 -> 3 : 424 |****************** | […] Average latency (iostat/sar) may not be represen[[ve with mul[ple modes or outliers The "count" column is summarized in-kernel
Efficiently trace TCP sessions with PID and bytes using tcplife # /usr/share/bcc/tools/tcplife PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 2509 java 100.82.34.63 8078 100.82.130.159 12410 0 0 5.44 2509 java 100.82.34.63 8078 100.82.78.215 55564 0 0 135.32 2509 java 100.82.34.63 60778 100.82.207.252 7001 0 13 15126.87 2509 java 100.82.34.63 38884 100.82.208.178 7001 0 0 15568.25 2509 java 127.0.0.1 4243 127.0.0.1 42166 0 0 0.61 2509 java 127.0.0.1 42166 127.0.0.1 4243 0 0 0.67 12030 upload-mes 127.0.0.1 34020 127.0.0.1 8078 11 0 3.38 2509 java 127.0.0.1 8078 127.0.0.1 34020 0 11 3.41 12030 upload-mes 127.0.0.1 21196 127.0.0.1 7101 0 0 12.61 3964 mesos-slav 127.0.0.1 7101 127.0.0.1 21196 0 0 12.64 12021 upload-sys 127.0.0.1 34022 127.0.0.1 8078 372 0 15.28 2509 java 127.0.0.1 8078 127.0.0.1 34022 0 372 15.31 2235 dockerd 100.82.34.63 13730 100.82.136.233 7002 0 4 18.50 2235 dockerd 100.82.34.63 34314 100.82.64.53 7002 0 8 56.73 [...] Dynamic tracing of TCP set state only; does not trace send/receive Also see: tcpconnect, tcpaccept, tcpretrans
Efficiently trace TCP sessions with PID and bytes using tcplife # /usr/share/bcc/tools/tcplife PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 2509 java 100.82.34.63 8078 100.82.130.159 12410 0 0 5.44 2509 java 100.82.34.63 8078 100.82.78.215 55564 0 0 135.32 2509 java 100.82.34.63 60778 100.82.207.252 7001 0 13 15126.87 2509 java 100.82.34.63 38884 100.82.208.178 7001 0 0 15568.25 2509 java 127.0.0.1 4243 127.0.0.1 42166 0 0 0.61 2509 java 127.0.0.1 42166 127.0.0.1 4243 0 0 0.67 12030 upload-mes 127.0.0.1 34020 127.0.0.1 8078 11 0 3.38 2509 java 127.0.0.1 8078 127.0.0.1 34020 0 11 3.41 12030 upload-mes 127.0.0.1 21196 127.0.0.1 7101 0 0 12.61 3964 mesos-slav 127.0.0.1 7101 127.0.0.1 21196 0 0 12.64 12021 upload-sys 127.0.0.1 34022 127.0.0.1 8078 372 0 15.28 2509 java 127.0.0.1 8078 127.0.0.1 34022 0 372 15.31 2235 dockerd 100.82.34.63 13730 100.82.136.233 7002 0 4 18.50 2235 dockerd 100.82.34.63 34314 100.82.64.53 7002 0 8 56.73 [...] Dynamic tracing of TCP set state only; does not trace send/receive Also see: tcpconnect, tcpaccept, tcpretrans
Identify DNS latency issues system wide with gethostlatency # /usr/share/bcc/tools/gethostlatency TIME PID COMM LATms HOST 18:56:36 5055 mesos-slave 0.01 100.82.166.217 18:56:40 5590 java 3.53 ec2-…-79.compute-1.amazonaws.com 18:56:51 5055 mesos-slave 0.01 100.82.166.217 18:56:53 30166 ncat 0.21 localhost 18:56:56 6661 java 2.19 atlas-alert-….prod.netflix.net 18:56:59 5589 java 1.50 ec2-…-207.compute-1.amazonaws.com 18:57:03 5370 java 0.04 localhost 18:57:03 30259 sudo 0.07 titusagent-mainvpc-m…3465 18:57:06 5055 mesos-slave 0.01 100.82.166.217 18:57:10 5590 java 3.10 ec2-…-79.compute-1.amazonaws.com 18:57:21 5055 mesos-slave 0.01 100.82.166.217 18:57:29 5589 java 52.36 ec2-…-207.compute-1.amazonaws.com 18:57:36 5055 mesos-slave 0.01 100.82.166.217 18:57:40 5590 java 1.83 ec2-…-79.compute-1.amazonaws.com 18:57:51 5055 mesos-slave 0.01 100.82.166.217 […] Instruments using user-level dynamic tracing of getaddrinfo(), gethostbyname(), etc.
Identify DNS latency issues system wide with gethostlatency # /usr/share/bcc/tools/gethostlatency TIME PID COMM LATms HOST 18:56:36 5055 mesos-slave 0.01 100.82.166.217 18:56:40 5590 java 3.53 ec2-…-79.compute-1.amazonaws.com 18:56:51 5055 mesos-slave 0.01 100.82.166.217 18:56:53 30166 ncat 0.21 localhost 18:56:56 6661 java 2.19 atlas-alert-….prod.netflix.net 18:56:59 5589 java 1.50 ec2-…-207.compute-1.amazonaws.com 18:57:03 5370 java 0.04 localhost 18:57:03 30259 sudo 0.07 titusagent-mainvpc-m…3465 18:57:06 5055 mesos-slave 0.01 100.82.166.217 18:57:10 5590 java 3.10 ec2-…-79.compute-1.amazonaws.com 18:57:21 5055 mesos-slave 0.01 100.82.166.217 18:57:29 5589 java 52.36 ec2-…-207.compute-1.amazonaws.com 18:57:36 5055 mesos-slave 0.01 100.82.166.217 18:57:40 5590 java 1.83 ec2-…-79.compute-1.amazonaws.com 18:57:51 5055 mesos-slave 0.01 100.82.166.217 […] Instruments using user-level dynamic tracing of getaddrinfo(), gethostbyname(), etc.
Examine CPU scheduler latency as a histogram with runqlat # /usr/share/bcc/tools/runqlat 10 Tracing run queue latency... Hit Ctrl-C to end. usecs : count distribution 0 -> 1 : 2810 |* | 2 -> 3 : 5248 |** | 4 -> 7 : 12369 |****** | 8 -> 15 : 71312 |****************************************| 16 -> 31 : 55705 |******************************* | 32 -> 63 : 11775 |****** | 64 -> 127 : 6230 |*** | 128 -> 255 : 2758 |* | 256 -> 511 : 549 | | 512 -> 1023 : 46 | | 1024 -> 2047 : 11 | | 2048 -> 4095 : 4 | | 4096 -> 8191 : 5 | | […] As efficient as possible: scheduler calls can become frequent
Examine CPU scheduler latency as a histogram with runqlat # /usr/share/bcc/tools/runqlat 10 Tracing run queue latency... Hit Ctrl-C to end. usecs : count distribution 0 -> 1 : 2810 |* | 2 -> 3 : 5248 |** | 4 -> 7 : 12369 |****** | 8 -> 15 : 71312 |****************************************| 16 -> 31 : 55705 |******************************* | 32 -> 63 : 11775 |****** | 64 -> 127 : 6230 |*** | 128 -> 255 : 2758 |* | 256 -> 511 : 549 | | 512 -> 1023 : 46 | | 1024 -> 2047 : 11 | | 2048 -> 4095 : 4 | | 4096 -> 8191 : 5 | | […] As efficient as possible: scheduler calls can become frequent
Construct programmatic one-liners with trace # trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' TIME PID COMM FUNC - 05:18:23 4490 dd sys_read read 1048576 bytes 05:18:23 4490 dd sys_read read 1048576 bytes 05:18:23 4490 dd sys_read read 1048576 bytes ^C argdist by Sasha Goldshtein # trace -h [...] trace –K blk_account_io_start Trace this kernel function, and print info with a kernel stack trace trace 'do_sys_open "%s", arg2' Trace the open syscall and print the filename being opened trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' Trace the read syscall and print a message for reads >20000 bytes trace r::do_sys_return Trace the return from the open syscall trace 'c:open (arg2 == 42) "%s %d", arg1, arg2' Trace the open() call from libc only if the flags (arg2) argument is 42 [...] e.g. reads over 20000 bytes:
Create in-kernel summaries with argdist # argdist -H 'p::tcp_cleanup_rbuf(struct sock *sk, int copied):int:copied' [15:34:45] copied : count distribution 0 -> 1 : 15088 |********************************** | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 4786 |*********** | 128 -> 255 : 1 | | 256 -> 511 : 1 | | 512 -> 1023 : 4 | | 1024 -> 2047 : 11 | | 2048 -> 4095 : 5 | | 4096 -> 8191 : 27 | | 8192 -> 16383 : 105 | | 16384 -> 32767 : 0 | | argdist by Sasha Goldshtein e.g. histogram of tcp_cleanup_rbuf() copied:
BCC/BPF Visualizations Coming to a GUI near you
BPF metrics and analysis can be automated in GUIs Flame Graphs Heat Maps Tracing Reports … Eg, Netflix Vector (self-service UI): Should be open sourced; you may also build/buy your own
Latency heatmaps show histograms over time
Optimize CPU flame graphs with BPF: count stacks in-kernel
What about Off-CPU? Generic thread state digram
Efficient Off-CPU flame graphs via scheduler tracing and BPF CPU Off-CPU Solve everything?
Off-CPU Time (zoomed): gzip(1) Off-CPU doesn't always make sense: what is gzip blocked on?
Wakeup time flame graphs show waker thread stacks
Wakeup Time (zoomed): gzip(1) gzip(1) is blocked on tar(1)! tar cf - * | gzip > out.tar.gz Can't we associate off-CPU with wakeup stacks?
Off-wake flame graphs: BPF can merge blocking plus waker stacks in-kernel Waker task Waker stack Blocked stack Blocked task Stack Direc[on Wokeup
Another example
Chain graphs: merge all wakeup stacks
Future Work BPF
BCC Improvements •  Challenges –  Initialize all variables –  BPF_PERF_OUTPUT() –  Verifier errors –  Still explicit bpf_probe_read()s. It's getting better (thanks): •  High-Level Languages –  One-liners and scripts –  Can use libbcc tcpconnlat.py
ply •  A new BPF-based language and tracer for Linux –  Created by Tobias Waldekranz –  https://github.com/iovisor/ply https://wkz.github.io/ply/ –  Promising, was in development # ply -c 'kprobe:do_sys_open { printf("opened: %sn", mem(arg(1), "128s")); }' 1 probe active opened: /sys/kernel/debug/tracing/events/enable opened: /etc/ld.so.cache opened: /lib/x86_64-linux-gnu/libselinux.so.1 opened: /lib/x86_64-linux-gnu/libc.so.6 opened: /proc/filesystems opened: /usr/lib/locale/locale-archive opened: . [...]
ply programs are concise, such as measuring read latency # ply -A -c 'kprobe:SyS_read { @start[tid()] = nsecs(); } kretprobe:SyS_read /@start[tid()]/ { @ns.quantize(nsecs() - @start[tid()]); @start[tid()] = nil; }' 2 probes active ^Cde-activating probes [...] @ns: [ 512, 1k) 3 |######## | [ 1k, 2k) 7 |################### | [ 2k, 4k) 12 |################################| [ 4k, 8k) 3 |######## | [ 8k, 16k) 2 |##### | [ 16k, 32k) 0 | | [ 32k, 64k) 0 | | [ 64k, 128k) 3 |######## | [128k, 256k) 1 |### | [256k, 512k) 1 |### | [512k, 1M) 2 |##### | [...]
bpftrace •  Another new BPF-based language and tracer for Linux –  Created by Alastair Robertson –  https://github.com/ajor/bpftrace –  In active development # bpftrace -e 'kprobe:sys_open { printf("opened: %sn", str(arg0)); }' Attaching 1 probe... opened: /sys/devices/system/cpu/online opened: /proc/1956/stat opened: /proc/1241/stat opened: /proc/net/dev opened: /proc/net/if_inet6 opened: /sys/class/net/eth0/device/vendor opened: /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms [...]
bpftrace programs are concise, such as measuring read latency # bpftrace -e 'kprobe:SyS_read { @start[tid] = nsecs; } kretprobe:SyS_read /@start[tid]/ { @ns = quantize(nsecs - @start[tid]); @start[tid] = delete(); }' Attaching 2 probes... ^C @ns: [0, 1] 0 | | [2, 4) 0 | | [4, 8) 0 | | [8, 16) 0 | | [16, 32) 0 | | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 0 | | [256, 512) 0 | | [512, 1k) 0 | | [1k, 2k) 6 |@@@@@ | [2k, 4k) 20 |@@@@@@@@@@@@@@@@@@@ | [4k, 8k) 4 |@@@ | [8k, 16k) 14 |@@@@@@@@@@@@@ | [16k, 32k) 53 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32k, 64k) 2 |@ |
New Tooling/Metrics
New Visualizations
Case Studies •  Use it •  Solve something •  Write about it •  Talk about it •  Recent posts: –  https://blogs.dropbox.com/tech/2017/09/optimizing-web-servers-for-high-throughput- and-low-latency/ –  https://josefbacik.github.io/kernel/scheduler/bcc/bpf/2017/08/03/sched-time.html
Advanced Analysis •  Find/draw a functional diagram •  Apply performance methods –  http://www.brendangregg.com/methodology.html –  Workload Characterization –  USE Method –  Latency Analysis –  Start with the Q's, then find the A's •  Use multi-tools: –  funccount, trace, argdist, stackcount e.g., storage I/O subsystem
Take aways 1.  Understand Linux tracing and enhanced BPF 2.  How to use eBPF tools 3.  Areas of future development BPF Tracing in Linux •  3.19: sockets •  3.19: maps •  4.1: kprobes •  4.3: uprobes •  4.4: BPF output •  4.6: stacks •  4.7: tracepoints •  4.9: profiling •  4.9: PMCs Please contribute: -  hHps://github.com/ iovisor/bcc -  hHps://github.com/ iovisor/ply Upgrade to Linux 4.9+!
Links & References iovisor bcc: -  https://github.com/iovisor/bcc https://github.com/iovisor/bcc/tree/master/docs -  http://www.brendangregg.com/blog/ (search for "bcc") -  http://www.brendangregg.com/ebpf.html#bcc -  http://blogs.microsoft.co.il/sasha/2016/02/14/two-new-ebpf-tools-memleak-and-argdist/ -  On designing tracing tools: https://www.youtube.com/watch?v=uibLwoVKjec bcc tutorial: -  https://github.com/iovisor/bcc/blob/master/INSTALL.md -  …/docs/tutorial.md …/docs/tutorial_bcc_python_developer.md …/docs/reference_guide.md -  .../CONTRIBUTING-SCRIPTS.md ply: https://github.com/iovisor/ply bpftrace: https://github.com/ajor/bpftrace BPF: -  https://www.kernel.org/doc/Documentation/networking/filter.txt -  https://github.com/iovisor/bpf-docs -  https://suchakra.wordpress.com/tag/bpf/ Flame Graphs: -  http://www.brendangregg.com/flamegraphs.html -  http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html -  http://www.brendangregg.com/blog/2016-02-01/linux-wakeup-offwake-profiling.html Netflix Tech Blog on Vector: -  http://techblog.netflix.com/2015/04/introducing-vector-netflixs-on-host.html Linux Performance: http://www.brendangregg.com/linuxperf.html
BPF @ Open Source Summit •  Making the Kernel's Networking Data Path Programmable with BPF and XDP –  Daniel Borkmann, Tuesday, 11:55am @ Georgia I/II •  Performance Analysis Superpowers with Linux BPF –  Brendan Gregg, this talk •  Cilium - Container Security and Networking using BPF and XDP –  Thomas Graf, Wednesday, 2:50pm @ Diamond Ballroom 6
Thank You –  Questions? –  iovisor bcc: https://github.com/iovisor/bcc –  http://www.brendangregg.com –  http://slideshare.net/brendangregg –  bgregg@netflix.com –  @brendangregg Thanks to Alexei Starovoitov (Facebook), Brenden Blanco (PLUMgrid/VMware), Sasha Goldshtein (Sela), Teng Qin (Facebook), Yonghong Song (Facebook), Daniel Borkmann (Cisco/Covalent), Wang Nan (Huawei), Vicent Martí (GitHub), Paul Chaignon (Orange), and other BPF and bcc contributors!

More Related Content

PDF
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
byAnne Nicolas
 
PDF
Linux Performance Profiling and Monitoring
byGeorg Schönberger
 
PDF
Linux Performance Tools
byBrendan Gregg
 
PDF
Blazing Performance with Flame Graphs
byBrendan Gregg
 
PDF
Systems@Scale 2021 BPF Performance Getting Started
byBrendan Gregg
 
PDF
YOW2020 Linux Systems Performance
byBrendan Gregg
 
PDF
EuroBSDcon 2017 System Performance Analysis Methodologies
byBrendan Gregg
 
PPTX
QCon 2015 Broken Performance Tools
byBrendan Gregg
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
byAnne Nicolas
 
Linux Performance Profiling and Monitoring
byGeorg Schönberger
 
Linux Performance Tools
byBrendan Gregg
 
Blazing Performance with Flame Graphs
byBrendan Gregg
 
Systems@Scale 2021 BPF Performance Getting Started
byBrendan Gregg
 
YOW2020 Linux Systems Performance
byBrendan Gregg
 
EuroBSDcon 2017 System Performance Analysis Methodologies
byBrendan Gregg
 
QCon 2015 Broken Performance Tools
byBrendan Gregg
 

What's hot

PDF
Kernel Recipes 2017: Performance Analysis with BPF
byBrendan Gregg
 
PDF
Container Performance Analysis
byBrendan Gregg
 
PDF
LSFMM 2019 BPF Observability
byBrendan Gregg
 
PDF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
byBrendan Gregg
 
PDF
Velocity 2015 linux perf tools
byBrendan Gregg
 
PDF
BPF: Tracing and more
byBrendan Gregg
 
PDF
Linux Systems Performance 2016
byBrendan Gregg
 
PDF
Performance Wins with eBPF: Getting Started (2021)
byBrendan Gregg
 
PDF
USENIX ATC 2017: Visualizing Performance with Flame Graphs
byBrendan Gregg
 
PDF
Linux Performance Analysis: New Tools and Old Secrets
byBrendan Gregg
 
PDF
Analyzing OS X Systems Performance with the USE Method
byBrendan Gregg
 
PDF
The New Systems Performance
byBrendan Gregg
 
PDF
BPF Tools 2017
byBrendan Gregg
 
PDF
bcc/BPF tools - Strategy, current tools, future challenges
byIO Visor Project
 
PDF
Performance Wins with BPF: Getting Started
byBrendan Gregg
 
PDF
FreeBSD 2014 Flame Graphs
byBrendan Gregg
 
PDF
Introduction to Perf
byWang Hsiangkai
 
PDF
Designing Tracing Tools
byBrendan Gregg
 
PDF
MeetBSD2014 Performance Analysis
byBrendan Gregg
 
PDF
Intro to linux performance analysis
byChris McEniry
 
Kernel Recipes 2017: Performance Analysis with BPF
byBrendan Gregg
 
Container Performance Analysis
byBrendan Gregg
 
LSFMM 2019 BPF Observability
byBrendan Gregg
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
byBrendan Gregg
 
Velocity 2015 linux perf tools
byBrendan Gregg
 
BPF: Tracing and more
byBrendan Gregg
 
Linux Systems Performance 2016
byBrendan Gregg
 
Performance Wins with eBPF: Getting Started (2021)
byBrendan Gregg
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
byBrendan Gregg
 
Linux Performance Analysis: New Tools and Old Secrets
byBrendan Gregg
 
Analyzing OS X Systems Performance with the USE Method
byBrendan Gregg
 
The New Systems Performance
byBrendan Gregg
 
BPF Tools 2017
byBrendan Gregg
 
bcc/BPF tools - Strategy, current tools, future challenges
byIO Visor Project
 
Performance Wins with BPF: Getting Started
byBrendan Gregg
 
FreeBSD 2014 Flame Graphs
byBrendan Gregg
 
Introduction to Perf
byWang Hsiangkai
 
Designing Tracing Tools
byBrendan Gregg
 
MeetBSD2014 Performance Analysis
byBrendan Gregg
 
Intro to linux performance analysis
byChris McEniry
 

Similar to OSSNA 2017 Performance Analysis Superpowers with Linux BPF

PDF
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
byAnne Nicolas
 
PDF
eBPF Perf Tools 2019
byBrendan Gregg
 
PDF
Velocity 2017 Performance analysis superpowers with Linux eBPF
byBrendan Gregg
 
PDF
re:Invent 2019 BPF Performance Analysis at Netflix
byBrendan Gregg
 
PDF
LISA2019 Linux Systems Performance
byBrendan Gregg
 
PDF
ATO Linux Performance 2018
byBrendan Gregg
 
PDF
test
byWentingLiu4
 
PDF
Linux Tracing Superpowers by Eugene Pirogov
byPivorak MeetUp
 
PDF
UM2019 Extended BPF: A New Type of Software
byBrendan Gregg
 
PDF
Using eBPF Off-CPU Sampling to See What Your DBs are Really Waiting For by Ta...
byScyllaDB
 
PDF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
byBrendan Gregg
 
PDF
NetConf 2018 BPF Observability
byBrendan Gregg
 
PDF
Linux 4.x Tracing Tools: Using BPF Superpowers
byBrendan Gregg
 
PDF
Linux BPF Superpowers
byBrendan Gregg
 
PDF
Linux Perf Tools
byRaj Pandey
 
PDF
Kernel bug hunting
byAndrea Righi
 
PPTX
Designing Tracing Tools
bySysdig
 
PDF
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021
byValeriy Kravchuk
 
PDF
Efficient System Monitoring in Cloud Native Environments
byGergely Szabó
 
PDF
Linux Server Deep Dives (DrupalCon Amsterdam)
byAmin Astaneh
 
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
byAnne Nicolas
 
eBPF Perf Tools 2019
byBrendan Gregg
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
byBrendan Gregg
 
re:Invent 2019 BPF Performance Analysis at Netflix
byBrendan Gregg
 
LISA2019 Linux Systems Performance
byBrendan Gregg
 
ATO Linux Performance 2018
byBrendan Gregg
 
test
byWentingLiu4
 
Linux Tracing Superpowers by Eugene Pirogov
byPivorak MeetUp
 
UM2019 Extended BPF: A New Type of Software
byBrendan Gregg
 
Using eBPF Off-CPU Sampling to See What Your DBs are Really Waiting For by Ta...
byScyllaDB
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
byBrendan Gregg
 
NetConf 2018 BPF Observability
byBrendan Gregg
 
Linux 4.x Tracing Tools: Using BPF Superpowers
byBrendan Gregg
 
Linux BPF Superpowers
byBrendan Gregg
 
Linux Perf Tools
byRaj Pandey
 
Kernel bug hunting
byAndrea Righi
 
Designing Tracing Tools
bySysdig
 
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021
byValeriy Kravchuk
 
Efficient System Monitoring in Cloud Native Environments
byGergely Szabó
 
Linux Server Deep Dives (DrupalCon Amsterdam)
byAmin Astaneh
 

More from Brendan Gregg

PDF
YOW2018 Cloud Performance Root Cause Analysis at Netflix
byBrendan Gregg
 
PDF
Computing Performance: On the Horizon (2021)
byBrendan Gregg
 
PDF
Kernel Recipes 2017: Using Linux perf at Netflix
byBrendan Gregg
 
PDF
YOW2021 Computing Performance
byBrendan Gregg
 
PDF
LPC2019 BPF Tracing Tools
byBrendan Gregg
 
PDF
IntelON 2021 Processor Benchmarking
byBrendan Gregg
 
PDF
BPF Internals (eBPF)
byBrendan Gregg
 
PDF
LISA17 Container Performance Analysis
byBrendan Gregg
 
PDF
Linux Performance 2018 (PerconaLive keynote)
byBrendan Gregg
 
PDF
How Netflix Tunes EC2 Instances for Performance
byBrendan Gregg
 
PDF
FlameScope 2018
byBrendan Gregg
 
PDF
YOW2018 CTO Summit: Working at netflix
byBrendan Gregg
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
byBrendan Gregg
 
Computing Performance: On the Horizon (2021)
byBrendan Gregg
 
Kernel Recipes 2017: Using Linux perf at Netflix
byBrendan Gregg
 
YOW2021 Computing Performance
byBrendan Gregg
 
LPC2019 BPF Tracing Tools
byBrendan Gregg
 
IntelON 2021 Processor Benchmarking
byBrendan Gregg
 
BPF Internals (eBPF)
byBrendan Gregg
 
LISA17 Container Performance Analysis
byBrendan Gregg
 
Linux Performance 2018 (PerconaLive keynote)
byBrendan Gregg
 
How Netflix Tunes EC2 Instances for Performance
byBrendan Gregg
 
FlameScope 2018
byBrendan Gregg
 
YOW2018 CTO Summit: Working at netflix
byBrendan Gregg
 

Recently uploaded

PDF
November 2025 OpenMetadata Community Spotlight - Astronomer & Airflow 3
byOpenMetadata
 
PDF
Supervised Machine Learning Approaches for Log-Based Anomaly Detection: A Cas...
byMohammed BEKKOUCHE
 
PDF
Building a Multi-Tenant OpenShift Platform with HyperShift
byTosin Akinosho
 
PDF
IDSECCONF2025 - Rama Tri Nanda - Hunting Stingray GSM on Budget.pdf
byidsecconf
 
PDF
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
PDF
LVM Management and Disaster Recovery.pdf
byLinuxCert Guru
 
PDF
Beyond Basics: How to Build Scalable, Intelligent Imagery Pipelines
bySafe Software
 
PDF
Linux Foundation Certified System Administrator (LFCS) Exam.pdf
byLinuxCert Guru
 
PPTX
oop pillor polymorphism Code Presentation
byhifzamahmood8
 
PPTX
AI: Beyond Generative AI and LLM | Harrie de Groot (harrie.dev)
byHarrie de Groot
 
PDF
IDSECCONF2025 - Budi Rahardjo - Cyber Security and AI.pdf
byidsecconf
 
PPTX
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
PDF
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
PPTX
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
PPTX
Streamlining Business Processes with UiPath Apps
byTracy Dixon
 
PDF
PMI PMBOK8 Released on November 13, 2025
byScott M. Graffius
 
PPTX
The power of Slack and MuleSoft | Bangalore MuleSoft Meetup #60
byshyamraj55
 
PDF
AI in Food Production (Foodwell) - AI Solutions Supporting Operations
bybyteLAKE
 
PDF
Building Powerful Web Apps to Improve Productivity and Engagement - Esri UK W...
byEsri UK
 
PDF
How to Spot a Fraudulent Shopping Website
byAshwini Singh
 
November 2025 OpenMetadata Community Spotlight - Astronomer & Airflow 3
byOpenMetadata
 
Supervised Machine Learning Approaches for Log-Based Anomaly Detection: A Cas...
byMohammed BEKKOUCHE
 
Building a Multi-Tenant OpenShift Platform with HyperShift
byTosin Akinosho
 
IDSECCONF2025 - Rama Tri Nanda - Hunting Stingray GSM on Budget.pdf
byidsecconf
 
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
LVM Management and Disaster Recovery.pdf
byLinuxCert Guru
 
Beyond Basics: How to Build Scalable, Intelligent Imagery Pipelines
bySafe Software
 
Linux Foundation Certified System Administrator (LFCS) Exam.pdf
byLinuxCert Guru
 
oop pillor polymorphism Code Presentation
byhifzamahmood8
 
AI: Beyond Generative AI and LLM | Harrie de Groot (harrie.dev)
byHarrie de Groot
 
IDSECCONF2025 - Budi Rahardjo - Cyber Security and AI.pdf
byidsecconf
 
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
Streamlining Business Processes with UiPath Apps
byTracy Dixon
 
PMI PMBOK8 Released on November 13, 2025
byScott M. Graffius
 
The power of Slack and MuleSoft | Bangalore MuleSoft Meetup #60
byshyamraj55
 
AI in Food Production (Foodwell) - AI Solutions Supporting Operations
bybyteLAKE
 
Building Powerful Web Apps to Improve Productivity and Engagement - Esri UK W...
byEsri UK
 
How to Spot a Fraudulent Shopping Website
byAshwini Singh
 
In this document
Powered by AI

Discussion on Linux BPF and its potential. Agenda covers eBPF, bcc tools, and future development.

Introduction to Enhanced BPF and its capabilities for tracing, including dynamic tracing, event configurations.

Discussion of dynamic tracing evolution, support in Linux versions, and key tools for Linux event tracing.

BPF Compiler Collection tools introduction, installation guides, and general performance analysis checklist.

Tools like execsnoop and ext4slower used to analyze process latency and storage latency.

Tools like biolatency and tcplife used to identify disk I/O latency issues and TCP session tracking.

System-wide DNS latency issues examined with gethostlatency tool functionality.

Utilizing tools such as runqlat and argdist for CPU scheduler latency and argument distribution analyses.

Future developments in visualizations for BPF metrics and analysis including graphical interfaces and heatmaps.

Wrap-up on BPF's applications, challenges, community contributions, and presentation of acknowledgments.

OSSNA 2017 Performance Analysis Superpowers with Linux BPF

  • 1.
    Performance Analysis Superpowers withLinux BPF Brendan Gregg Sep 2017
  • 3.
  • 4.
  • 5.
    Agenda 1. eBPF &bcc 2. bcc/BPF CLI Tools 3. bcc/BPF Visualizations
  • 6.
    Take aways 1.  UnderstandLinux tracing and enhanced BPF 2.  How to use BPF tools 3.  Areas of future development
  • 8.
  • 9.
    BPF Introducing enhanced BPFfor tracing: kernel-level software
  • 10.
    Ye Olde BPF Berkeley Packet Filter #tcpdump host 127.0.0.1 and port 22 -d (000) ldh [12] (001) jeq #0x800 jt 2 jf 18 (002) ld [26] (003) jeq #0x7f000001 jt 6 jf 4 (004) ld [30] (005) jeq #0x7f000001 jt 6 jf 18 (006) ldb [23] (007) jeq #0x84 jt 10 jf 8 (008) jeq #0x6 jt 10 jf 9 (009) jeq #0x11 jt 10 jf 18 (010) ldh [20] (011) jset #0x1fff jt 18 jf 12 (012) ldxb 4*([14]&0xf) (013) ldh [x + 14] [...] User-defined bytecode executed by an in-kernel sandboxed virtual machine Steven McCanne and Van Jacobson, 1993 2 x 32-bit registers & scratch memory Optimizes packet filter performance
  • 11.
    Enhanced BPF aka eBPF or just "BPF" Alexei Starovoitov,2014+ 10 x 64-bit registers maps (hashes) actions
  • 12.
    BPF for Tracing,Internals BPF bytecode Observability Program Kernel tracepoints kprobes uprobes BPF maps per-event data statistics verifier output static tracing dynamic tracing async copy perf_events sampling, PMCs BPF program event config attach load Enhanced BPF is also now used for SDNs, DDOS mitigation, intrusion detection, container security, …
  • 13.
  • 14.
  • 15.
    Event Tracing Efficiency send receive tcpdump Kernel buffer filesystem 1. read 2. dump Analyzer 1. read 2. process 3. print disks Old way: packet capture New way: dynamic tracing Tracer 1. configure 2. read tcp_retransmit_skb() E.g., tracing TCP retransmits
  • 16.
    Linux Events &BPF Support Linux 4.3 Linux 4.7 Linux 4.9 Linux 4.9 Linux 4.1 BPF stacks Linux 4.6 BPF output Linux 4.4 (version BPF support arrived)
  • 17.
    A Linux TracingTimeline -  1990’s: Static tracers, prototype dynamic tracers -  2000: LTT + DProbes (dynamic tracing; not integrated) -  2004: kprobes (2.6.9) -  2005: DTrace (not Linux), SystemTap (out-of-tree) -  2008: ftrace (2.6.27) -  2009: perf_events (2.6.31) -  2009: tracepoints (2.6.32) -  2010-2017: ftrace & perf_events enhancements -  2012: uprobes (3.5) -  2014-2017: enhanced BPF patches: supporting tracing events -  2016-2017: ftrace hist triggers also: LTTng, ktap, sysdig, ...
  • 18.
    BCC Introducing BPF ComplierCollection: user-level front-end
  • 19.
    bcc •  BPF CompilerCollection –  https://github.com/iovisor/bcc –  Lead developer: Brenden Blanco •  Includes tracing tools •  Provides BPF front-ends: –  Python –  Lua –  C++ –  C helper libraries –  golang (gobpf) BPF Python Events Kernel lua bcc front-ends bcc tool bcc tool … … user kernel Tracing layers:
  • 20.
  • 21.
  • 22.
    bcc/BPF (C &Python) bcc examples/tracing/bitehist.py enBre program
  • 23.
  • 24.
    The Tracing Landscape,Sep 2017 Scope & Capability Easeofuse sysdig perf ftrace C/BPF ktap stap Stage of Development (my opinion) dtrace4L. (brutal)(lessbrutal) (alpha) (mature) bcc/BPF ply/BPF Raw BPF LTTng (hist triggers)recent changes (many) bpftrace
  • 25.
  • 26.
    Pre-BPF: Linux PerfAnalysis in 60s 1.  uptime 2.  dmesg -T | tail 3.  vmstat 1 4.  mpstat -P ALL 1 5.  pidstat 1 6.  iostat -xz 1 7.  free -m 8.  sar -n DEV 1 9.  sar -n TCP,ETCP 1 10.  top hHp://techblog.ne/lix.com/2015/11/linux-performance-analysis-in-60s.html
  • 27.
    bcc Installation •  https://github.com/iovisor/bcc/blob/master/INSTALL.md • eg, Ubuntu Xenial: –  Also available as an Ubuntu snap –  Ubuntu 16.04 is good, 16.10 better: more tools work •  Installs many tools –  In /usr/share/bcc/tools, and …/tools/old for older kernels # echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" |
 sudo tee /etc/apt/sources.list.d/iovisor.list # sudo apt-get update # sudo apt-get install bcc-tools
  • 28.
    bcc General PerformanceChecklist 1.  execsnoop 2.  opensnoop 3.  ext4slower (…) 4.  biolatency 5.  biosnoop 6.  cachestat 7.  tcpconnect 8.  tcpaccept 9.  tcpretrans 10.  gethostlatency 11.  runqlat 12.  profile
  • 29.
    Discover short-lived processissues using execsnoop # execsnoop -t TIME(s) PCOMM PID PPID RET ARGS 0.031 dirname 23832 23808 0 /usr/bin/dirname /apps/tomcat/bin/catalina.sh 0.888 run 23833 2344 0 ./run 0.889 run 23833 2344 -2 /command/bash 0.889 run 23833 2344 -2 /usr/local/bin/bash 0.889 run 23833 2344 -2 /usr/local/sbin/bash 0.889 bash 23833 2344 0 /bin/bash 0.894 svstat 23835 23834 0 /command/svstat /service/nflx-httpd 0.894 perl 23836 23834 0 /usr/bin/perl -e $l=<>;$l=~/(d+) sec/;print $1||0; 0.899 ps 23838 23837 0 /bin/ps --ppid 1 -o pid,cmd,args 0.900 grep 23839 23837 0 /bin/grep org.apache.catalina 0.900 sed 23840 23837 0 /bin/sed s/^ *//; 0.900 cut 23841 23837 0 /usr/bin/cut -d -f 1 0.901 xargs 23842 23837 0 /usr/bin/xargs 0.912 xargs 23843 23842 -2 /command/echo 0.912 xargs 23843 23842 -2 /usr/local/bin/echo 0.912 xargs 23843 23842 -2 /usr/local/sbin/echo 0.912 echo 23843 23842 0 /bin/echo [...] Efficient: only traces exec()
  • 30.
    Discover short-lived processissues using execsnoop # execsnoop -t TIME(s) PCOMM PID PPID RET ARGS 0.031 dirname 23832 23808 0 /usr/bin/dirname /apps/tomcat/bin/catalina.sh 0.888 run 23833 2344 0 ./run 0.889 run 23833 2344 -2 /command/bash 0.889 run 23833 2344 -2 /usr/local/bin/bash 0.889 run 23833 2344 -2 /usr/local/sbin/bash 0.889 bash 23833 2344 0 /bin/bash 0.894 svstat 23835 23834 0 /command/svstat /service/nflx-httpd 0.894 perl 23836 23834 0 /usr/bin/perl -e $l=<>;$l=~/(d+) sec/;print $1||0; 0.899 ps 23838 23837 0 /bin/ps --ppid 1 -o pid,cmd,args 0.900 grep 23839 23837 0 /bin/grep org.apache.catalina 0.900 sed 23840 23837 0 /bin/sed s/^ *//; 0.900 cut 23841 23837 0 /usr/bin/cut -d -f 1 0.901 xargs 23842 23837 0 /usr/bin/xargs 0.912 xargs 23843 23842 -2 /command/echo 0.912 xargs 23843 23842 -2 /usr/local/bin/echo 0.912 xargs 23843 23842 -2 /usr/local/sbin/echo 0.912 echo 23843 23842 0 /bin/echo [...] Efficient: only traces exec()
  • 31.
    Exonerate or confirmstorage latency outliers with ext4slower # /usr/share/bcc/tools/ext4slower 1 Tracing ext4 operations slower than 1 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 17:31:42 postdrop 15523 S 0 0 2.32 5630D406E4 17:31:42 cleanup 15524 S 0 0 1.89 57BB7406EC 17:32:09 titus-log-ship 19735 S 0 0 1.94 slurper_checkpoint.db 17:35:37 dhclient 1061 S 0 0 3.32 dhclient.eth0.leases 17:35:39 systemd-journa 504 S 0 0 26.62 system.journal 17:35:39 systemd-journa 504 S 0 0 1.56 system.journal 17:35:39 systemd-journa 504 S 0 0 1.73 system.journal 17:35:45 postdrop 16187 S 0 0 2.41 C0369406E4 17:35:45 cleanup 16188 S 0 0 6.52 C1B90406EC […] Tracing at the file system is a more reliable and complete indicator than measuring disk I/O latency Also: btrfsslower, xfsslower, zfsslower
  • 32.
    Exonerate or confirmstorage latency outliers with ext4slower # /usr/share/bcc/tools/ext4slower 1 Tracing ext4 operations slower than 1 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 17:31:42 postdrop 15523 S 0 0 2.32 5630D406E4 17:31:42 cleanup 15524 S 0 0 1.89 57BB7406EC 17:32:09 titus-log-ship 19735 S 0 0 1.94 slurper_checkpoint.db 17:35:37 dhclient 1061 S 0 0 3.32 dhclient.eth0.leases 17:35:39 systemd-journa 504 S 0 0 26.62 system.journal 17:35:39 systemd-journa 504 S 0 0 1.56 system.journal 17:35:39 systemd-journa 504 S 0 0 1.73 system.journal 17:35:45 postdrop 16187 S 0 0 2.41 C0369406E4 17:35:45 cleanup 16188 S 0 0 6.52 C1B90406EC […] Tracing at the file system is a more reliable and complete indicator than measuring disk I/O latency Also: btrfsslower, xfsslower, zfsslower
  • 33.
    Identify multimodal diskI/O latency and outliers with biolatency # biolatency -mT 10 Tracing block device I/O... Hit Ctrl-C to end. 19:19:04 msecs : count distribution 0 -> 1 : 238 |********* | 2 -> 3 : 424 |***************** | 4 -> 7 : 834 |********************************* | 8 -> 15 : 506 |******************** | 16 -> 31 : 986 |****************************************| 32 -> 63 : 97 |*** | 64 -> 127 : 7 | | 128 -> 255 : 27 |* | 19:19:14 msecs : count distribution 0 -> 1 : 427 |******************* | 2 -> 3 : 424 |****************** | […] Average latency (iostat/sar) may not be represen[[ve with mul[ple modes or outliers The "count" column is summarized in-kernel
  • 34.
    Identify multimodal diskI/O latency and outliers with biolatency # biolatency -mT 10 Tracing block device I/O... Hit Ctrl-C to end. 19:19:04 msecs : count distribution 0 -> 1 : 238 |********* | 2 -> 3 : 424 |***************** | 4 -> 7 : 834 |********************************* | 8 -> 15 : 506 |******************** | 16 -> 31 : 986 |****************************************| 32 -> 63 : 97 |*** | 64 -> 127 : 7 | | 128 -> 255 : 27 |* | 19:19:14 msecs : count distribution 0 -> 1 : 427 |******************* | 2 -> 3 : 424 |****************** | […] Average latency (iostat/sar) may not be represen[[ve with mul[ple modes or outliers The "count" column is summarized in-kernel
  • 35.
    Efficiently trace TCPsessions with PID and bytes using tcplife # /usr/share/bcc/tools/tcplife PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 2509 java 100.82.34.63 8078 100.82.130.159 12410 0 0 5.44 2509 java 100.82.34.63 8078 100.82.78.215 55564 0 0 135.32 2509 java 100.82.34.63 60778 100.82.207.252 7001 0 13 15126.87 2509 java 100.82.34.63 38884 100.82.208.178 7001 0 0 15568.25 2509 java 127.0.0.1 4243 127.0.0.1 42166 0 0 0.61 2509 java 127.0.0.1 42166 127.0.0.1 4243 0 0 0.67 12030 upload-mes 127.0.0.1 34020 127.0.0.1 8078 11 0 3.38 2509 java 127.0.0.1 8078 127.0.0.1 34020 0 11 3.41 12030 upload-mes 127.0.0.1 21196 127.0.0.1 7101 0 0 12.61 3964 mesos-slav 127.0.0.1 7101 127.0.0.1 21196 0 0 12.64 12021 upload-sys 127.0.0.1 34022 127.0.0.1 8078 372 0 15.28 2509 java 127.0.0.1 8078 127.0.0.1 34022 0 372 15.31 2235 dockerd 100.82.34.63 13730 100.82.136.233 7002 0 4 18.50 2235 dockerd 100.82.34.63 34314 100.82.64.53 7002 0 8 56.73 [...] Dynamic tracing of TCP set state only; does not trace send/receive Also see: tcpconnect, tcpaccept, tcpretrans
  • 36.
    Efficiently trace TCPsessions with PID and bytes using tcplife # /usr/share/bcc/tools/tcplife PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 2509 java 100.82.34.63 8078 100.82.130.159 12410 0 0 5.44 2509 java 100.82.34.63 8078 100.82.78.215 55564 0 0 135.32 2509 java 100.82.34.63 60778 100.82.207.252 7001 0 13 15126.87 2509 java 100.82.34.63 38884 100.82.208.178 7001 0 0 15568.25 2509 java 127.0.0.1 4243 127.0.0.1 42166 0 0 0.61 2509 java 127.0.0.1 42166 127.0.0.1 4243 0 0 0.67 12030 upload-mes 127.0.0.1 34020 127.0.0.1 8078 11 0 3.38 2509 java 127.0.0.1 8078 127.0.0.1 34020 0 11 3.41 12030 upload-mes 127.0.0.1 21196 127.0.0.1 7101 0 0 12.61 3964 mesos-slav 127.0.0.1 7101 127.0.0.1 21196 0 0 12.64 12021 upload-sys 127.0.0.1 34022 127.0.0.1 8078 372 0 15.28 2509 java 127.0.0.1 8078 127.0.0.1 34022 0 372 15.31 2235 dockerd 100.82.34.63 13730 100.82.136.233 7002 0 4 18.50 2235 dockerd 100.82.34.63 34314 100.82.64.53 7002 0 8 56.73 [...] Dynamic tracing of TCP set state only; does not trace send/receive Also see: tcpconnect, tcpaccept, tcpretrans
  • 37.
    Identify DNS latencyissues system wide with gethostlatency # /usr/share/bcc/tools/gethostlatency TIME PID COMM LATms HOST 18:56:36 5055 mesos-slave 0.01 100.82.166.217 18:56:40 5590 java 3.53 ec2-…-79.compute-1.amazonaws.com 18:56:51 5055 mesos-slave 0.01 100.82.166.217 18:56:53 30166 ncat 0.21 localhost 18:56:56 6661 java 2.19 atlas-alert-….prod.netflix.net 18:56:59 5589 java 1.50 ec2-…-207.compute-1.amazonaws.com 18:57:03 5370 java 0.04 localhost 18:57:03 30259 sudo 0.07 titusagent-mainvpc-m…3465 18:57:06 5055 mesos-slave 0.01 100.82.166.217 18:57:10 5590 java 3.10 ec2-…-79.compute-1.amazonaws.com 18:57:21 5055 mesos-slave 0.01 100.82.166.217 18:57:29 5589 java 52.36 ec2-…-207.compute-1.amazonaws.com 18:57:36 5055 mesos-slave 0.01 100.82.166.217 18:57:40 5590 java 1.83 ec2-…-79.compute-1.amazonaws.com 18:57:51 5055 mesos-slave 0.01 100.82.166.217 […] Instruments using user-level dynamic tracing of getaddrinfo(), gethostbyname(), etc.
  • 38.
    Identify DNS latencyissues system wide with gethostlatency # /usr/share/bcc/tools/gethostlatency TIME PID COMM LATms HOST 18:56:36 5055 mesos-slave 0.01 100.82.166.217 18:56:40 5590 java 3.53 ec2-…-79.compute-1.amazonaws.com 18:56:51 5055 mesos-slave 0.01 100.82.166.217 18:56:53 30166 ncat 0.21 localhost 18:56:56 6661 java 2.19 atlas-alert-….prod.netflix.net 18:56:59 5589 java 1.50 ec2-…-207.compute-1.amazonaws.com 18:57:03 5370 java 0.04 localhost 18:57:03 30259 sudo 0.07 titusagent-mainvpc-m…3465 18:57:06 5055 mesos-slave 0.01 100.82.166.217 18:57:10 5590 java 3.10 ec2-…-79.compute-1.amazonaws.com 18:57:21 5055 mesos-slave 0.01 100.82.166.217 18:57:29 5589 java 52.36 ec2-…-207.compute-1.amazonaws.com 18:57:36 5055 mesos-slave 0.01 100.82.166.217 18:57:40 5590 java 1.83 ec2-…-79.compute-1.amazonaws.com 18:57:51 5055 mesos-slave 0.01 100.82.166.217 […] Instruments using user-level dynamic tracing of getaddrinfo(), gethostbyname(), etc.
  • 39.
    Examine CPU schedulerlatency as a histogram with runqlat # /usr/share/bcc/tools/runqlat 10 Tracing run queue latency... Hit Ctrl-C to end. usecs : count distribution 0 -> 1 : 2810 |* | 2 -> 3 : 5248 |** | 4 -> 7 : 12369 |****** | 8 -> 15 : 71312 |****************************************| 16 -> 31 : 55705 |******************************* | 32 -> 63 : 11775 |****** | 64 -> 127 : 6230 |*** | 128 -> 255 : 2758 |* | 256 -> 511 : 549 | | 512 -> 1023 : 46 | | 1024 -> 2047 : 11 | | 2048 -> 4095 : 4 | | 4096 -> 8191 : 5 | | […] As efficient as possible: scheduler calls can become frequent
  • 40.
    Examine CPU schedulerlatency as a histogram with runqlat # /usr/share/bcc/tools/runqlat 10 Tracing run queue latency... Hit Ctrl-C to end. usecs : count distribution 0 -> 1 : 2810 |* | 2 -> 3 : 5248 |** | 4 -> 7 : 12369 |****** | 8 -> 15 : 71312 |****************************************| 16 -> 31 : 55705 |******************************* | 32 -> 63 : 11775 |****** | 64 -> 127 : 6230 |*** | 128 -> 255 : 2758 |* | 256 -> 511 : 549 | | 512 -> 1023 : 46 | | 1024 -> 2047 : 11 | | 2048 -> 4095 : 4 | | 4096 -> 8191 : 5 | | […] As efficient as possible: scheduler calls can become frequent
  • 41.
    Construct programmatic one-linerswith trace # trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' TIME PID COMM FUNC - 05:18:23 4490 dd sys_read read 1048576 bytes 05:18:23 4490 dd sys_read read 1048576 bytes 05:18:23 4490 dd sys_read read 1048576 bytes ^C argdist by Sasha Goldshtein # trace -h [...] trace –K blk_account_io_start Trace this kernel function, and print info with a kernel stack trace trace 'do_sys_open "%s", arg2' Trace the open syscall and print the filename being opened trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' Trace the read syscall and print a message for reads >20000 bytes trace r::do_sys_return Trace the return from the open syscall trace 'c:open (arg2 == 42) "%s %d", arg1, arg2' Trace the open() call from libc only if the flags (arg2) argument is 42 [...] e.g. reads over 20000 bytes:
  • 42.
    Create in-kernel summarieswith argdist # argdist -H 'p::tcp_cleanup_rbuf(struct sock *sk, int copied):int:copied' [15:34:45] copied : count distribution 0 -> 1 : 15088 |********************************** | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 4786 |*********** | 128 -> 255 : 1 | | 256 -> 511 : 1 | | 512 -> 1023 : 4 | | 1024 -> 2047 : 11 | | 2048 -> 4095 : 5 | | 4096 -> 8191 : 27 | | 8192 -> 16383 : 105 | | 16384 -> 32767 : 0 | | argdist by Sasha Goldshtein e.g. histogram of tcp_cleanup_rbuf() copied:
  • 43.
  • 44.
    BPF metrics andanalysis can be automated in GUIs Flame Graphs Heat Maps Tracing Reports … Eg, Netflix Vector (self-service UI): Should be open sourced; you may also build/buy your own
  • 45.
    Latency heatmaps showhistograms over time
  • 46.
    Optimize CPU flamegraphs with BPF: count stacks in-kernel
  • 47.
    What about Off-CPU? Genericthread state digram
  • 48.
    Efficient Off-CPU flamegraphs via scheduler tracing and BPF CPU Off-CPU Solve everything?
  • 49.
    Off-CPU Time (zoomed):gzip(1) Off-CPU doesn't always make sense: what is gzip blocked on?
  • 50.
    Wakeup time flamegraphs show waker thread stacks
  • 51.
    Wakeup Time (zoomed):gzip(1) gzip(1) is blocked on tar(1)! tar cf - * | gzip > out.tar.gz Can't we associate off-CPU with wakeup stacks?
  • 52.
    Off-wake flame graphs:BPF can merge blocking plus waker stacks in-kernel Waker task Waker stack Blocked stack Blocked task Stack Direc[on Wokeup
  • 53.
  • 54.
    Chain graphs: mergeall wakeup stacks
  • 55.
  • 56.
    BCC Improvements •  Challenges – Initialize all variables –  BPF_PERF_OUTPUT() –  Verifier errors –  Still explicit bpf_probe_read()s. It's getting better (thanks): •  High-Level Languages –  One-liners and scripts –  Can use libbcc tcpconnlat.py
  • 57.
    ply •  A newBPF-based language and tracer for Linux –  Created by Tobias Waldekranz –  https://github.com/iovisor/ply https://wkz.github.io/ply/ –  Promising, was in development # ply -c 'kprobe:do_sys_open { printf("opened: %sn", mem(arg(1), "128s")); }' 1 probe active opened: /sys/kernel/debug/tracing/events/enable opened: /etc/ld.so.cache opened: /lib/x86_64-linux-gnu/libselinux.so.1 opened: /lib/x86_64-linux-gnu/libc.so.6 opened: /proc/filesystems opened: /usr/lib/locale/locale-archive opened: . [...]
  • 58.
    ply programs areconcise, such as measuring read latency # ply -A -c 'kprobe:SyS_read { @start[tid()] = nsecs(); } kretprobe:SyS_read /@start[tid()]/ { @ns.quantize(nsecs() - @start[tid()]); @start[tid()] = nil; }' 2 probes active ^Cde-activating probes [...] @ns: [ 512, 1k) 3 |######## | [ 1k, 2k) 7 |################### | [ 2k, 4k) 12 |################################| [ 4k, 8k) 3 |######## | [ 8k, 16k) 2 |##### | [ 16k, 32k) 0 | | [ 32k, 64k) 0 | | [ 64k, 128k) 3 |######## | [128k, 256k) 1 |### | [256k, 512k) 1 |### | [512k, 1M) 2 |##### | [...]
  • 59.
    bpftrace •  Another newBPF-based language and tracer for Linux –  Created by Alastair Robertson –  https://github.com/ajor/bpftrace –  In active development # bpftrace -e 'kprobe:sys_open { printf("opened: %sn", str(arg0)); }' Attaching 1 probe... opened: /sys/devices/system/cpu/online opened: /proc/1956/stat opened: /proc/1241/stat opened: /proc/net/dev opened: /proc/net/if_inet6 opened: /sys/class/net/eth0/device/vendor opened: /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms [...]
  • 60.
    bpftrace programs areconcise, such as measuring read latency # bpftrace -e 'kprobe:SyS_read { @start[tid] = nsecs; } kretprobe:SyS_read /@start[tid]/ { @ns = quantize(nsecs - @start[tid]); @start[tid] = delete(); }' Attaching 2 probes... ^C @ns: [0, 1] 0 | | [2, 4) 0 | | [4, 8) 0 | | [8, 16) 0 | | [16, 32) 0 | | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 0 | | [256, 512) 0 | | [512, 1k) 0 | | [1k, 2k) 6 |@@@@@ | [2k, 4k) 20 |@@@@@@@@@@@@@@@@@@@ | [4k, 8k) 4 |@@@ | [8k, 16k) 14 |@@@@@@@@@@@@@ | [16k, 32k) 53 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [32k, 64k) 2 |@ |
  • 61.
  • 62.
  • 63.
    Case Studies •  Useit •  Solve something •  Write about it •  Talk about it •  Recent posts: –  https://blogs.dropbox.com/tech/2017/09/optimizing-web-servers-for-high-throughput- and-low-latency/ –  https://josefbacik.github.io/kernel/scheduler/bcc/bpf/2017/08/03/sched-time.html
  • 64.
    Advanced Analysis •  Find/drawa functional diagram •  Apply performance methods –  http://www.brendangregg.com/methodology.html –  Workload Characterization –  USE Method –  Latency Analysis –  Start with the Q's, then find the A's •  Use multi-tools: –  funccount, trace, argdist, stackcount e.g., storage I/O subsystem
  • 65.
    Take aways 1.  UnderstandLinux tracing and enhanced BPF 2.  How to use eBPF tools 3.  Areas of future development BPF Tracing in Linux •  3.19: sockets •  3.19: maps •  4.1: kprobes •  4.3: uprobes •  4.4: BPF output •  4.6: stacks •  4.7: tracepoints •  4.9: profiling •  4.9: PMCs Please contribute: -  hHps://github.com/ iovisor/bcc -  hHps://github.com/ iovisor/ply Upgrade to Linux 4.9+!
  • 66.
    Links & References iovisorbcc: -  https://github.com/iovisor/bcc https://github.com/iovisor/bcc/tree/master/docs -  http://www.brendangregg.com/blog/ (search for "bcc") -  http://www.brendangregg.com/ebpf.html#bcc -  http://blogs.microsoft.co.il/sasha/2016/02/14/two-new-ebpf-tools-memleak-and-argdist/ -  On designing tracing tools: https://www.youtube.com/watch?v=uibLwoVKjec bcc tutorial: -  https://github.com/iovisor/bcc/blob/master/INSTALL.md -  …/docs/tutorial.md …/docs/tutorial_bcc_python_developer.md …/docs/reference_guide.md -  .../CONTRIBUTING-SCRIPTS.md ply: https://github.com/iovisor/ply bpftrace: https://github.com/ajor/bpftrace BPF: -  https://www.kernel.org/doc/Documentation/networking/filter.txt -  https://github.com/iovisor/bpf-docs -  https://suchakra.wordpress.com/tag/bpf/ Flame Graphs: -  http://www.brendangregg.com/flamegraphs.html -  http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html -  http://www.brendangregg.com/blog/2016-02-01/linux-wakeup-offwake-profiling.html Netflix Tech Blog on Vector: -  http://techblog.netflix.com/2015/04/introducing-vector-netflixs-on-host.html Linux Performance: http://www.brendangregg.com/linuxperf.html
  • 67.
    BPF @ OpenSource Summit •  Making the Kernel's Networking Data Path Programmable with BPF and XDP –  Daniel Borkmann, Tuesday, 11:55am @ Georgia I/II •  Performance Analysis Superpowers with Linux BPF –  Brendan Gregg, this talk •  Cilium - Container Security and Networking using BPF and XDP –  Thomas Graf, Wednesday, 2:50pm @ Diamond Ballroom 6
  • 68.
    Thank You –  Questions? – iovisor bcc: https://github.com/iovisor/bcc –  http://www.brendangregg.com –  http://slideshare.net/brendangregg –  bgregg@netflix.com –  @brendangregg Thanks to Alexei Starovoitov (Facebook), Brenden Blanco (PLUMgrid/VMware), Sasha Goldshtein (Sela), Teng Qin (Facebook), Yonghong Song (Facebook), Daniel Borkmann (Cisco/Covalent), Wang Nan (Huawei), Vicent Martí (GitHub), Paul Chaignon (Orange), and other BPF and bcc contributors!