Repository navigation
Conversation
Replay Shopify PR #48 on upstream main after Ruby JIT support merged. Preserve the full LuaJIT interpreter and eBPF implementation while adapting it to current interpreter configuration, process mapping, stack-delta, tracer, metrics, and coredump APIs. Original-Commit: 81b1e8f Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Regenerate Go ABI types and the eBPF error header after adding LuaJIT process data, constants, metrics, and error catalog entries. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Rebuild both embedded tracer objects with clang 17.0.6 in the pinned profiler development image. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
The Loader only matched libluajit-5.1.so*/luajit*/nginx/openresty. tarantool (shopkv) statically links LuaJIT into the main 'tarantool' executable, so no separate libluajit mapping exists and the unwinder never engaged. Add 'tarantool' to the allowlist so the LuaJIT offset extraction runs against the tarantool binary. NOTE: offset extraction was derived from openresty/luajit2; tarantool ships its own LuaJIT fork, so extraction may still need tarantool-specific offsets — this makes the unwinder attempt tarantool and surfaces concrete extraction signal. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…antool) Generalize the executable-detection allowlist: instead of hardcoding which binaries statically link LuaJIT, add a luajit.Config.Executables list so embedders (tarantool, custom static binaries) opt in via the per-interpreter interpreter.Config. embedsLuaJIT() = built-in set (libluajit-5.1.so*/luajit*/ nginx/openresty) + configured Executables. Supersedes the hardcoded 'tarantool' match; this is the patch proposed upstream on open-telemetry#1236. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…resent
extractInterpreterBounds scans stack deltas for the first large gap matching the
VM unwind pattern. On x86 tarantool that matched an unrelated function (0x164469)
instead of the real lj_vm_asm_begin (0x261ca0), so extractOffsets failed its
start-address sanity check ('unexpected start address') and LuaJIT unwinding was
skipped (only native lj_* frames + ERROR_4012 from unmapped JIT mcode).
When the binary is unstripped (tarantool ships lj_vm_asm_begin), look the symbol
up via ef.LookupSymbol and return the delta gap that starts exactly there;
fall back to the heuristic for stripped binaries (asmBegin==0). Tests pass 0.
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…m_asm_begin The VM asm FDE starts at lj_vm_asm_begin but the preceding function's unwind info extends to a few bytes below it, so the delta interval starts just under the symbol (exact-address match missed it). Match the large delta interval that contains the symbol and start the range exactly at lj_vm_asm_begin. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
….LookupSymbol ef.LookupSymbol only resolves dynamic symbols; lj_vm_asm_begin is a hidden .symtab symbol, so the lookup returned ErrSymbolNotFound and asmBegin stayed 0, skipping the symbol-anchored interp-range logic entirely. Use scanSymbols (reads .symtab via VisitSymbols), matching how extractOffsets resolves the symbol. Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
Retain focused failure-path diagnostics for invalid or unreadable GCproto objects without emitting the temporary per-frame normal-frame trace logging. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…egression - Hermetic TestExtractInterpreterBoundsAnchor: proves the anchor picks the VM asm over a decoy SP/param gap (x86 bug) and is byte-identical to the heuristic on the arm64 frame-pointer layout; covers start-below-symbol + fallback. - Strengthen the OpenResty integration test to run the anchor on real shared-lib builds (amd64+arm64) and assert it yields identical offsets to the heuristic. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…ARF) Tarantool's LuaJIT inserts a mem_L field in global_State between cur_L and jit_base, so the eBPF assumption that jit_base is adjacent to cur_L read mem_L instead -> garbage frame base for JIT-trace samples -> unresolved <lua> frames. Extract jit_base's offset (g2jitbase) from lj_vm_exit_handler, which clears G->jit_base via the DISPATCH register on trace exit (g2jitbase = g2dispatch + disp). Cheap targeted disassembly, no DWARF. Falls back to cur_L+8 on stripped binaries, correct for OpenResty/luajit2 (no mem_L). arm64 extraction is a TODO (falls back; no regression to the arm64 interpreter path). - support/ebpf: LuaJITProcInfo.g2jitbase; read jit_base from G+g2jitbase. - offsets.go: findG2JitBaseOffset (+ lj_vm_exit_handler symbol); extractor iface. - extractor_x86.go: findG2JitBaseFromExitHandler (movq $0,disp(%r14) -> g2dispatch+disp). - offsets_test.go: assert openresty g2jitbase == cur_L+8 (regression guard, both arches). Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
…t unsigned) x86asm.Decode reports a negative disp32 as its unsigned 32-bit value in the int64 Disp field (0xFFFFF1E0, not -3616), so the g2dispatch+disp candidate overflowed the validation range and the extractor fell back to cur_L+8 (mem_L). Sign-extend the low 32 bits. Verified against the tarantool binary: g2jitbase now extracts 0x1f0 (jit_base) instead of 0x1e8 (mem_L). Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
arm64 tarantool also has the mem_L field, so jit_base is at cur_L+0x10 not +8. On arm64 the VM holds G directly in a register (x22), so the trace-exit clear 'str xzr, [x22, #0x1f0]' encodes jit_base's offset from G directly (g2dispatch unused, unlike x86). Mirrors the setgcrefnull(g->cur_L) matcher in lua_close. Verified against the real arm64 binary: g2jitbase=0x1f0. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…R_4012/4005) After Lua frames resolve, the unwinder hands back to native by stepping over the VM gate's C frame using a hardcoded LUAJIT_CFRAME_SPACE (x86 80, arm64 208). Both are wrong for tarantool: the interp region's own stack delta says x86 CFA=sp+96 and arm64 CFA=fp+16 (frame-pointer based). The mismatch lands the resume PC in an unmapped region -> every tarantool stack carried ERROR_4012 (x86, no_pid_page_mapping) / ERROR_4005 (arm64, stack_delta_invalid), losing the native frames below the VM. Extract the interp CFA offset + base register from the interpreter region's stack delta (info.Deltas covering luaInterp.Start) and use them in unwind_native_frame: SP-based (sp+=param) on x86, FP-based (sp=fp+param) on arm64. Falls back to LUAJIT_CFRAME_SPACE when unavailable. New LuaJITProcInfo.cframe_size_interp/interp_fp. Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
… ERROR_4012) At the bottom of the Lua stack (diff<=2) the code only stepped over the C frame (unwind_native_frame) for is_jit; for the interpreter it set next_unwinder=NATIVE with state->pc still in the luajit-mapped interp region, so the native unwinder re-entered luajit/failed -> ERROR_4012 and the C stack below the VM was dropped. arm64 happened to take the FRAME_CP path (which pokes); amd64 (tarantool, where the bottom Lua frame meets C here) took this path. Now poke for both. Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
…r handback) The JIT native handback resumed 16 bytes short -> into the fiber stack/heap instead of lua_pcall's return -> ERROR_4012, no native C stack. Root cause: cframeSizeJIT = cframeSize(80) + 16, but 80 is OpenResty's VM gate cframe; tarantool's lj_vm_pcall gate cframe is 96 (4 pushes + sub 56 + ret), which we already extract as cframeSizeInterp. Use that base on SP-based (x86) builds: cframeSizeJIT = interp + 16 = 112. Verified against the live fiber stack: lua_pcall's return address sits at new_sp+8. arm64 (FP-based) keeps the arch constant; its JIT path isn't the hot path there. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…d LuaJIT hosts) Lets coredump test cases probe binaries that statically link LuaJIT (e.g. tarantool) during extraction, and persists the config into the test case so the harness probes the same hosts on replay. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Core's transitional 4.0.5-pshopify3 build reports RUBY_VERSION=4.0.5, but its exported ruby_description identifies source revision 21a2595676. That revision is 31 commits after master_box change 99aac00 and is itself the rb_thread_sched change, so it unambiguously carries both 4.0.6 layout shifts. Treat only Ruby 4.0.5 at that exact description revision as using the new layout. Stock 4.0.0-4.0.5 retain their original offsets, while 4.0.6+ continues using the upstream open-telemetry#1634 version gate. If the description is missing or a pshopify rebuild changes revision, detection fails safe. Assisted-By: devx/38cc1a8a-dd3e-41a2-8b65-28604cba42e1 Assisted-By: Claude <noreply@anthropic.com>
Adapt native-handback cframe discovery to the current block-based IntervalData representation while preserving absolute-address interval matching. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Add the final jit_base and interpreter cframe fields to LuaJITProcInfo after replaying the Tarantool and native-handback fixes. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Rebuild both embedded tracer objects with clang 17.0.6 in the pinned profiler development image after the final native-handback changes. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Keep the upstream extractor test API stable while adding the symbol-anchored loader path, reconcile generated LuaJIT constants, refresh dependencies, and satisfy the current linter. Assisted-By: devx/d42532af-a9f8-4598-b82f-d768e8e865f4
Rebuild both embedded tracer objects with clang 17.0.6 in the pinned profiler development image after rebasing onto the merged LuaJIT offset extractor. Assisted-By: devx/d42532af-a9f8-4598-b82f-d768e8e865f4
Assisted-By: devx/d42532af-a9f8-4598-b82f-d768e8e865f4
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork-local scope
Important
This integration snapshot intentionally targets Shopify's
mainbranch. It is not an upstream pull request and must not be retargeted toopen-telemetry/opentelemetry-ebpf-profiler.Relationship to #78
PR #78 remains unchanged at
7211a319because that exact branch is deployed broadly. This PR is a new successor branch; it does not overwrite or repurpose the deployed branch.What changed
mainto upstreammainatee129ee2.Validation
make generateCGO_ENABLED=1 go test -run '^$' ./...tools/luajitoffsetsmatrix across OpenResty amd64/arm64 images0 issues)Final blob SHA-256:
f79a4a810cf84aaf08c60d391a27f79da1b1ce648e6714b3dd1d5a790f8013669d29737e7831614d42211960a0eb714548e0def36511857a6eb285c573368e36