Linux Performance Tuning: perf, bpftrace & BCC

The USE Method guides production debugging: first measure Utilization, Saturation, and Errors for each resource, then deep-dive with specific tools. This covers today's three most important tools.

perf_events: CPU-level Hotspot Analysis

perf uses CPU PMU hardware counters with tiny overhead (~2%). Sample at 99Hz to avoid lock-step with scheduler periodicity for accurate distributions. perf stat captures macro metrics; perf record + FlameGraph builds interactive visualizations.

#!/usr/bin/env bash
set -euo pipefail
LOG=perf-report-$(date +%Y%m%d-%H%M).log
exec > >(tee -a "$LOG") 2>&1

echo "===== [1] CPU TOP functions 30s ====="
perf record -F 99 -a -g -- sleep 30
perf report --stdio -n --sort comm,dso,symbol | head -60
# perf script | FlameGraph/stackcollapse-perf.pl | flamegraph.pl > cpu-flame.svg

echo "\n===== [2] perf stat system-wide ====="
perf stat -a -d -- sleep 10
# 关注 context-switches, cache-misses 比率, stalled-cycles-frontend

echo "\n===== [3] bpftrace: block IO size distribution by comm ====="
bpftrace -e '
tracepoint:block:block_rq_issue { @[comm, args->rwbs[0]] = hist(args->bytes); }
interval:s:15 { printf("\n=== 15s IO hist by process ===\n"); print(@); clear(@); exit(); }
' 2>&1 | head -40 || true

echo "\n===== [4] bpftrace: TCP active connect trace ====="
bpftrace -e '
kprobe:tcp_connect { printf("PID %d (%s) -> %s\n", pid, comm, ntop(args->family, args->daddr)); }
' -c 'curl -s https://example.com >/dev/null' 2>&1 | head -20 || true

echo "\n===== [5] BCC biolatency + tcplife ====="
/usr/share/bcc/tools/biolatency -mF 1 5 || true
# /usr/share/bcc/tools/offcputime -df 5 1 > offcpu.stacks
timeout 15 /usr/share/bcc/tools/tcplife -L -T 2>/dev/null | head -40 || true

echo "\n===== [6] snapshot vmstat slabtop sar ====="
vmstat 1 5
echo "--- slab top ---"
slabtop -o -s c | head -25 || true
echo "--- THP check ---"
grep -E "AnonHugePages|HugePages_Total" /proc/meminfo || true
echo "\nSaved to: $LOG"

bpftrace and BCC: Kernel Event Tracing

eBPF attaches sandboxed C programs to kernel functions, tracepoints, USDT probes. bpftrace excels at one-liner rapid investigations. BCC ships 200+ prepackaged tools; biolatency, offcputime, and tcplife are your triad for IO/block/network latency triage.

Best Practices

Macro before micro: start with sar/vmstat to classify which resource type is the issue, then pick right BCC tools, finally drop into custom perf/bpftrace. eBPF is privileged; validate scripts in staging first and limit production runs to few minutes at a time.