Benchmarking bpf_loop

2 min read

The eBPF verifier enforces strict limits on program complexity. If a loop iterates too many times or contains too many branches, the verifier will reject the program as it is unable to guarantee the loop will terminate within the 1-million instruction limit.

Kernel 5.17 introduced the bpf_loop helper to bypass this exact constraint. Because this helper executes the loop body via a callback, the verifier only needs to analyze the callback function once. The downside is that it forces an indirect function call on every iteration.

I wanted to measure exactly what that indirect call costs.

The benchmark setup

I wrote a small Rust harness using the Aya framework to compare a native for loop against bpf_loop. Both programs were compiled as kprobes attached to __arm64_sys_getpid on a Linux (v6.9) VM and measured the execution time using bpf_ktime_get_ns().

Benchmarking empty loops in Rust is problematic because LLVM will optimize them away entirely. In order to force the compiler to emit the loop instructions and prevent unrolling, I used volatile memory operations on an accumulator variable. (there might be better ways to do this)

The native loop:

let start = unsafe { bpf_ktime_get_ns() };
let mut sum: u64 = 0;

for i in 0..10_000 {
    unsafe {
        write_volatile(&mut sum, read_volatile(&sum) + i as u64);
    }
}

let end = unsafe { bpf_ktime_get_ns() };

The bpf_loop implementation requires an extern "C" callback:

unsafe extern "C" fn callback(i: u32, ctx: *mut c_void) -> c_int {
    let sum = ctx as *mut u64;

    unsafe {
        write_volatile(sum, read_volatile(sum) + i as u64);
    }

    0
}

// inside kprobe:
let start = unsafe { bpf_ktime_get_ns() };
let mut sum: u64 = 0;

unsafe {
    bpf_loop(10_000, callback as *mut c_void, &mut sum as *mut _ as *mut c_void, 0);
}

let end = unsafe { bpf_ktime_get_ns() };

Cost of indirect calls

I ran both probes and recorded the elapsed time for 10k iterations.

  • Native loop: 20,194 ns
  • bpf_loop: 50,005 ns

The native loop compiles to standard bpf branch and arithmetic instructions. When JIT compiled by the kernel, these become basic machine code jumps. The loop executes at roughly 2.0 ns per iteration.

bpf_loop took almost 2.5 times longer, averaging at around 5.0 ns per iteration. That extra ~ 3.0 ns per iteration is the direct cost required to invoke the callback pointer from within the kernel helper.

The benchmarking code can be found here: https://github.com/zignis/bpf-loop-bench