Repository navigation
Conversation
This was referenced Jul 21, 2026
dalehamel
force-pushed
the
dale/luajit-omnibus
branch
from
July 21, 2026 16:51
6ac1d24 to
f3e2115
Compare
dalehamel
added a commit
that referenced
this pull request
Jul 21, 2026
Bring in the green Ruby JIT open-telemetry#1102 follow-up through Shopify PR #59 while keeping PR #50 pinned to the production LuaJIT snapshot.
This was referenced Jul 24, 2026
dalehamel
added a commit
that referenced
this pull request
Jul 24, 2026
Bring PR #50 forward through upstream open-telemetry#1605, including dynamic trace origins (open-telemetry#1607) and an isolated kprobe unwind record (open-telemetry#1605), which the ProbeContext API requires. Preserve the current PR #50 Ruby 4.0.6 layout detection and coredump fixtures; regenerate support types and both BPF blobs. Assisted-By: devx/2d48a84f-4aab-48eb-9009-5dd68fa593f3
Replay Shopify PR #48 on upstream main after Ruby JIT support merged. Preserve the full LuaJIT interpreter and eBPF implementation while adapting it to current interpreter configuration, process mapping, stack-delta, tracer, metrics, and coredump APIs. Original-Commit: 81b1e8f Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Regenerate Go ABI types and the eBPF error header after adding LuaJIT process data, constants, metrics, and error catalog entries. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Rebuild both embedded tracer objects with clang 17.0.6 in the pinned profiler development image. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
The Loader only matched libluajit-5.1.so*/luajit*/nginx/openresty. tarantool (shopkv) statically links LuaJIT into the main 'tarantool' executable, so no separate libluajit mapping exists and the unwinder never engaged. Add 'tarantool' to the allowlist so the LuaJIT offset extraction runs against the tarantool binary. NOTE: offset extraction was derived from openresty/luajit2; tarantool ships its own LuaJIT fork, so extraction may still need tarantool-specific offsets — this makes the unwinder attempt tarantool and surfaces concrete extraction signal. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…antool) Generalize the executable-detection allowlist: instead of hardcoding which binaries statically link LuaJIT, add a luajit.Config.Executables list so embedders (tarantool, custom static binaries) opt in via the per-interpreter interpreter.Config. embedsLuaJIT() = built-in set (libluajit-5.1.so*/luajit*/ nginx/openresty) + configured Executables. Supersedes the hardcoded 'tarantool' match; this is the patch proposed upstream on open-telemetry#1236. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
… from non-GC64 lua_close) extractor_x86.findOffsetsFromLuaClose derives glref/curL by matching the register that holds global_State across the lua_close disassembly. sameReg only mapped EAX/ECX/EDX/EBX/ESI to their 64-bit forms, omitting EBP/ESP/EDI. Tarantool's x86 (non-GC64) LuaJIT loads G with a 32-bit `movl 0x8(%rdi),%ebp` (greg=EBP) and then stores cur_L via `movl $0x0,0x188(%rbp)` (base RBP). sameReg(RBP,EBP) returned false, so the cur_L store never matched and extraction failed with 'find offsets from lua_close failed' — no Lua unwinding on x86 tarantool (arm64 worked). Complete the 32->64-bit GPR mapping via reg64(). Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…resent
extractInterpreterBounds scans stack deltas for the first large gap matching the
VM unwind pattern. On x86 tarantool that matched an unrelated function (0x164469)
instead of the real lj_vm_asm_begin (0x261ca0), so extractOffsets failed its
start-address sanity check ('unexpected start address') and LuaJIT unwinding was
skipped (only native lj_* frames + ERROR_4012 from unmapped JIT mcode).
When the binary is unstripped (tarantool ships lj_vm_asm_begin), look the symbol
up via ef.LookupSymbol and return the delta gap that starts exactly there;
fall back to the heuristic for stripped binaries (asmBegin==0). Tests pass 0.
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…m_asm_begin The VM asm FDE starts at lj_vm_asm_begin but the preceding function's unwind info extends to a few bytes below it, so the delta interval starts just under the symbol (exact-address match missed it). Match the large delta interval that contains the symbol and start the range exactly at lj_vm_asm_begin. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
….LookupSymbol ef.LookupSymbol only resolves dynamic symbols; lj_vm_asm_begin is a hidden .symtab symbol, so the lookup returned ErrSymbolNotFound and asmBegin stayed 0, skipping the symbol-anchored interp-range logic entirely. Use scanSymbols (reads .symtab via VisitSymbols), matching how extractOffsets resolves the symbol. Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
Retain focused failure-path diagnostics for invalid or unreadable GCproto objects without emitting the temporary per-frame normal-frame trace logging. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…egression - Hermetic TestExtractInterpreterBoundsAnchor: proves the anchor picks the VM asm over a decoy SP/param gap (x86 bug) and is byte-identical to the heuristic on the arm64 frame-pointer layout; covers start-below-symbol + fallback. - Strengthen the OpenResty integration test to run the anchor on real shared-lib builds (amd64+arm64) and assert it yields identical offsets to the heuristic. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…ARF) Tarantool's LuaJIT inserts a mem_L field in global_State between cur_L and jit_base, so the eBPF assumption that jit_base is adjacent to cur_L read mem_L instead -> garbage frame base for JIT-trace samples -> unresolved <lua> frames. Extract jit_base's offset (g2jitbase) from lj_vm_exit_handler, which clears G->jit_base via the DISPATCH register on trace exit (g2jitbase = g2dispatch + disp). Cheap targeted disassembly, no DWARF. Falls back to cur_L+8 on stripped binaries, correct for OpenResty/luajit2 (no mem_L). arm64 extraction is a TODO (falls back; no regression to the arm64 interpreter path). - support/ebpf: LuaJITProcInfo.g2jitbase; read jit_base from G+g2jitbase. - offsets.go: findG2JitBaseOffset (+ lj_vm_exit_handler symbol); extractor iface. - extractor_x86.go: findG2JitBaseFromExitHandler (movq $0,disp(%r14) -> g2dispatch+disp). - offsets_test.go: assert openresty g2jitbase == cur_L+8 (regression guard, both arches). Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
…t unsigned) x86asm.Decode reports a negative disp32 as its unsigned 32-bit value in the int64 Disp field (0xFFFFF1E0, not -3616), so the g2dispatch+disp candidate overflowed the validation range and the extractor fell back to cur_L+8 (mem_L). Sign-extend the low 32 bits. Verified against the tarantool binary: g2jitbase now extracts 0x1f0 (jit_base) instead of 0x1e8 (mem_L). Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
arm64 tarantool also has the mem_L field, so jit_base is at cur_L+0x10 not +8. On arm64 the VM holds G directly in a register (x22), so the trace-exit clear 'str xzr, [x22, #0x1f0]' encodes jit_base's offset from G directly (g2dispatch unused, unlike x86). Mirrors the setgcrefnull(g->cur_L) matcher in lua_close. Verified against the real arm64 binary: g2jitbase=0x1f0. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…R_4012/4005) After Lua frames resolve, the unwinder hands back to native by stepping over the VM gate's C frame using a hardcoded LUAJIT_CFRAME_SPACE (x86 80, arm64 208). Both are wrong for tarantool: the interp region's own stack delta says x86 CFA=sp+96 and arm64 CFA=fp+16 (frame-pointer based). The mismatch lands the resume PC in an unmapped region -> every tarantool stack carried ERROR_4012 (x86, no_pid_page_mapping) / ERROR_4005 (arm64, stack_delta_invalid), losing the native frames below the VM. Extract the interp CFA offset + base register from the interpreter region's stack delta (info.Deltas covering luaInterp.Start) and use them in unwind_native_frame: SP-based (sp+=param) on x86, FP-based (sp=fp+param) on arm64. Falls back to LUAJIT_CFRAME_SPACE when unavailable. New LuaJITProcInfo.cframe_size_interp/interp_fp. Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
… ERROR_4012) At the bottom of the Lua stack (diff<=2) the code only stepped over the C frame (unwind_native_frame) for is_jit; for the interpreter it set next_unwinder=NATIVE with state->pc still in the luajit-mapped interp region, so the native unwinder re-entered luajit/failed -> ERROR_4012 and the C stack below the VM was dropped. arm64 happened to take the FRAME_CP path (which pokes); amd64 (tarantool, where the bottom Lua frame meets C here) took this path. Now poke for both. Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255 Assisted-By: Claude <noreply@anthropic.com>
…r handback) The JIT native handback resumed 16 bytes short -> into the fiber stack/heap instead of lua_pcall's return -> ERROR_4012, no native C stack. Root cause: cframeSizeJIT = cframeSize(80) + 16, but 80 is OpenResty's VM gate cframe; tarantool's lj_vm_pcall gate cframe is 96 (4 pushes + sub 56 + ret), which we already extract as cframeSizeInterp. Use that base on SP-based (x86) builds: cframeSizeJIT = interp + 16 = 112. Verified against the live fiber stack: lua_pcall's return address sits at new_sp+8. arm64 (FP-based) keeps the arch constant; its JIT path isn't the hot path there. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
…d LuaJIT hosts) Lets coredump test cases probe binaries that statically link LuaJIT (e.g. tarantool) during extraction, and persists the config into the test case so the harness probes the same hosts on replay. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Core's transitional 4.0.5-pshopify3 build reports RUBY_VERSION=4.0.5, but its exported ruby_description identifies source revision 21a2595676. That revision is 31 commits after master_box change 99aac00 and is itself the rb_thread_sched change, so it unambiguously carries both 4.0.6 layout shifts. Treat only Ruby 4.0.5 at that exact description revision as using the new layout. Stock 4.0.0-4.0.5 retain their original offsets, while 4.0.6+ continues using the upstream open-telemetry#1634 version gate. If the description is missing or a pshopify rebuild changes revision, detection fails safe. Assisted-By: devx/38cc1a8a-dd3e-41a2-8b65-28604cba42e1 Assisted-By: Claude <noreply@anthropic.com>
Adapt native-handback cframe discovery to the current block-based IntervalData representation while preserving absolute-address interval matching. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Add the final jit_base and interpreter cframe fields to LuaJITProcInfo after replaying the Tarantool and native-handback fixes. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
Rebuild both embedded tracer objects with clang 17.0.6 in the pinned profiler development image after the final native-handback changes. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: devx/18d54ca0-685e-40de-8b30-f7ba6a0b0255
dalehamel
force-pushed
the
dale/luajit-omnibus
branch
from
August 13, 2026 22:41
7c16777 to
19aafba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LuaJIT unwinding — production omnibus snapshot
This PR preserves the complete LuaJIT-unwinding work exactly at the revision used by Shopify production.
Pinned production head
dale/luajit-omnibusf3e2115amain.6ac1d24fwas removed from this PR; that synchronization belongs on the intermediate Hack Days base above LuaJIT unwinding — omnibus snapshot (tarantool + openresty, amd64/arm64) #50.Keeping #50 production-exact lets existing consumers continue to reference the deployed LuaJIT snapshot without inheriting unrelated upstream changes.
Dependent I/O/network stack
The Hack Days stack uses a separate intermediate base for current-main synchronization and the latest Ruby-JIT open-telemetry#1102 fixes:
f3e2115a)0a00112e)ProbeContext/ custom-probes API (a4f279c1)9c56b6ec)f87ae55a)1630cf2d)d046cc66)Ancestry:
#50 ← #59 ← #58 ← #53 ← #54 ← #55 ← #56.#59 is fully green (43 passed, 1 skipped, 0 failed). GitHub may report #50 itself as conflicting with current
main; that is expected for this intentionally pinned production snapshot, and #59 is the green integration surface for dependent work.The dependent stack is confined to Shopify/opentelemetry-ebpf-profiler; no upstream branch or PR was created for it.
Goal
Enable profiling of LuaJIT business logic for Shopify's profiling stack (prophiler / profiles-processor): resolved Lua frames interleaved with the native C stack, on tarantool (shopkv) and OpenResty/nginx (ingress), amd64 + arm64.
Historical LuaJIT stack (ancestry:
#46 → #48 → #49)dale/ruby-jit-wide-anon-exec-fallback— Ruby-JIT base, owned separately.dale/luajit-unwinder-rebase— merges upstream Parca LuaJIT interpreter work (Add LuaJIT interpreter open-telemetry/opentelemetry-ebpf-profiler#1236), with interface adaptations.dale/luajit-tarantool-fixes— tarantool-specific fixes: statically-linked GC64 detection, x86 offset extraction,mem_L/jit_base,.eh_frame-derived cframe sizes, end-of-stack handback, and anonymous-mapping support.f3e2115a, includingcoredump new -luajit-executablessupport.Shipped
parseTracers+--luajit-executables+ on-demand UI tracer picker). LuaJIT is off by default and opt-in per schedule.5733d05b(profiles-processor main).LuaJIT validation
fib/churnLua frames resolved with the full native stack (coro_init → fiber_loop → … → lua_pcall → hot_loop → fib), error frames below 0.1%.box:space:ffs_node_lifecycle:get,slot_counts_for_target,node_lifecycle_counts_for_placement).Ongoing follow-up
Coredump regression coverage for the tarantool LuaJIT unwinder continues separately. Tooling is present (
coredump new -luajit-executables); core data backup and OCI upload remain tracked outside this PR.