Skip to content

Strip PERF_CONTEXT_* markers from bpf_get_stack callchains - #128

Open
SSE4 wants to merge 1 commit into
yandex:mainfrom
SSE4:unwinder-strip-perf-context-markers
Open

SSE4 wants to merge 1 commit into
yandex:mainfrom
SSE4:unwinder-strip-perf-context-markers

Conversation

@SSE4

@SSE4 SSE4 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Every kernel stack recorded by Perforator ends in a fake leaf frame: the last symbol of the last loaded
kernel module, e.g. cleanup_module@[tun], xt_nat_init@[xt_nat] or init_module@[nft_reject_inet].
Which module it is depends on the host.

The agent opens perf events with PERF_SAMPLE_CALLCHAIN, so bpf_get_stack() copies the precomputed
callchain. Its kernel part starts with the PERF_CONTEXT_KERNEL marker ((u64)-128), not with the leaf.
try_get_stack() kept the marker as frame 0:

  • kallsyms resolves 0xffffffffffffff80 to the last symbol in /proc/kallsyms, a module symbol;
  • the real leaf moved to index 1 and was decremented by one as if it were a return address.

Fix

  • try_get_stack() drops a leading PERF_CONTEXT_* marker before the return address adjustment.
    The helper also collects frame pointer user stacks, whose callchain would start with
    PERF_CONTEXT_USER, so the check is generic.
  • The sample consumer skips marker values in kernel and user stacks as a second line of defense.

Testing

  • TestKernelStackSkipsPerfContextMarkers (agent/collector/pkg/profiler): a kernel stack with
    markers resolves to the real frames only. Without the Go check it fails with
    cleanup_module@[tun] frames.
  • Repro below, perforator record -a -F 999 -d 15s, before and after the change.

Repro

// repro-a.c: gcc -O2 -g -fno-omit-frame-pointer -o repro-a repro-a.c
// A getrandom(2) loop: time is spent in the kernel (chacha), issued inline so the user leaf is the loop.

Full source: repro-a.c.

Before After
before after

Before: every getrandom stack ends in init_module@[nft_reject_inet]; it is the leaf of 99.9% of the
samples, so it shows up as the hottest function. After: the real leaves are chacha_permute, copy_to_iter and
chacha_block_generic.

before after

On a perf event with PERF_SAMPLE_CALLCHAIN, bpf_get_stack() copies the
precomputed callchain, and the kernel part starts with the
PERF_CONTEXT_KERNEL marker. try_get_stack() treated the marker as the
leaf frame:

- kallsyms resolved PERF_CONTEXT_KERNEL (0xffffffffffffff80) to the last
  symbol of the last loaded module, so every kernel stack got a fake leaf
  such as cleanup_module@[tun] or init_module@[nft_reject_inet];
- the real leaf was decremented as if it were a return address.

Drop a leading marker in try_get_stack() before the return address
adjustment. The same helper collects frame pointer user stacks, where
the callchain would start with PERF_CONTEXT_USER, so the check is not
specific to kernel stacks. Skip marker values in the sample consumer as
a second line of defense.
@robot-magpie

robot-magpie Bot commented Nov 13, 2026

Copy link
Copy Markdown

✅ All contributors are covered under a CLA with Yandex

See CONTRIBUTING.md for more info about Yandex Contributor License Agreement.


The following contributors were found:
✅ f9dda63 Author: @SSE4 (to*****de@gmail.com)

(Only the first commit for a unique contributor is listed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant