Writing an eBPF probe with bpftrace

bpftrace is the shortest path from a question about a running kernel to an answer. No module to build, no reboot, no printk. The program is compiled to BPF bytecode, verified, and attached to a tracepoint while the machine keeps serving traffic.

Start by finding out what you are allowed to ask:

sudo bpftrace -l 'tracepoint:syscalls:sys_enter_open*'
tracepoint:syscalls:sys_enter_open
tracepoint:syscalls:sys_enter_openat
tracepoint:syscalls:sys_enter_openat2

1. A histogram in one line

The question that comes up most often is “how long does this syscall actually take”. A pair of probes and a built-in map answers it:

sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_openat { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_openat /@start[tid]/ {
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

The output is a log2 histogram per bucket. What you are looking for is not the average — it is the tail. A p99 three orders of magnitude above the median usually means a filesystem waiting on something else.

2. Filter before you aggregate

Predicates run in the kernel, so filtering there costs nothing on the userspace side. Attach to a single process rather than the whole machine:

sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_write /comm == "nginx"/ {
    @bytes = sum(args->count);
  }'

3. Know what it costs

A probe on a hot path is not free. Attaching to sys_enter_write across a busy web server adds a measurable percentage to system time. Scope by process, run the probe for thirty seconds, and detach. Leaving instrumentation attached indefinitely is how tracing becomes the problem it was meant to diagnose.