Skip to content

Commit b858f7e

Browse files
committed
Update crash details.
1 parent 8546962 commit b858f7e

1 file changed

Lines changed: 52 additions & 10 deletions

File tree

‎test/PyExtTest/CRASH-INVESTIGATION.md‎

Lines changed: 52 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -69,16 +69,58 @@ Both `dbghelp.dll` and `dbgeng.dll` are the same SDK version
6969
(10.0.26100.8249, copied into the output dir by the PostBuild step) — not a
7070
version mismatch.
7171

72-
## Not yet pinned
73-
74-
The exact corrupting write is not yet identified. Plain TTD clips the
75-
`__fastfail` (`STATUS_HEAP_CORRUPTION`) a couple of instructions past the
76-
recorded trace end, so the fault itself is not in the trace and there is no
77-
corrupted-block address to reverse-trace from. Pinning it needs full PageHeap
78-
(to fault as an immediate AV at the corrupting access) under TTD with a
79-
per-iteration watchdog. Leading hypothesis: a bug in, or PyExt's use of,
80-
`dbghelp`/`dbgeng` symbol resolution during/around a `Reload`. This is the
81-
`engextcpp`/dbgeng layer the Option-3 initiative is removing.
72+
## Not yet pinned — and why every capture method fought us
73+
74+
The exact corrupting write is not identified. The bug is a
75+
**TTD-perturbation-sensitive heisenbug**, and the capture techniques are
76+
mutually exclusive with reproducing it:
77+
78+
- **Reproduces only under TTD** (~1 in 7 recorded runs). Bare runs (0/60),
79+
live under cdb (0/30+), and page-heap-live (0/15+) do **not** reproduce —
80+
the bug needs TTD's specific timing.
81+
- **Plain TTD clips the fault.** `STATUS_HEAP_CORRUPTION` fires via
82+
`__fastfail`, which bypasses normal exception dispatch; the trace ends ~2
83+
instructions before it (last recorded frame: `ntdll!RtlFlsSetValue`, a benign
84+
instruction). No corrupted-block address to reverse-trace from.
85+
- **`!heap -triage` on the trace is a false positive.** It flags one block, but
86+
reverse-execution shows that block is dbghelp's per-thread **TLS** data,
87+
written legitimately during thread startup (constant from ~10% into the
88+
trace). No genuine adjacent-block overrun is visible in the trace.
89+
- **Full PageHeap under TTD hangs** (guards every dbghelp alloc -> symbol
90+
enumeration crawls past the watchdog). **Standard PageHeap under TTD also
91+
hangs.** **PageHeap live doesn't reproduce** (changes timing away from the
92+
TTD window).
93+
- **Four standalone DbgEng repros do not reproduce** (see `fh4-repro/`):
94+
`IDebugControl::Evaluate`; the `IDebugAdvanced2::Request(EXT_TDOP_SET_FROM_EXPR)`
95+
typed-data path `ExtRemoteTyped::Set` uses; that plus a nested
96+
`ExtCaptureOutput`-equivalent; and a minimal **extension DLL** loaded in cdb
97+
that recreates the `dbgeng -> extension -> dbgeng` reentrancy. All clean
98+
10-30 iters under TTD. So the trigger is **not** the bare engine symbol path;
99+
it needs the full `PyExtTest` process context.
100+
101+
### Source audit (the actual crash path, before any object reads)
102+
103+
The crash is in `PythonDumpFile` ctor -> `pysymfix` -> `ensureSymbolsLoaded`,
104+
which runs before any object-type reads. Audited and **clean**:
105+
`getSymbolPath` buffer sizing, `GetModuleNames` (`MAX_PATH`/`sizeof`),
106+
`ExtCaptureOutput::Output` (overflow-guarded realloc/memcpy, LIFO nesting),
107+
`makeAutoInterpreterState`, `RemoteType`, and the object-read buffers
108+
(`readArray`, unicode, bytearray). No heap overrun found in PyExt code.
109+
110+
> Unrelated real bug spotted: `pysymfix.cpp getSymbolPath()` does
111+
> `throw runtime_error("..." + hr)` — pointer arithmetic on a string literal
112+
> (UB), on the `GetSymbolPath` failure path only. Not the crash cause; fix
113+
> separately (use `std::to_string(hr)`).
114+
115+
### Current best assessment
116+
117+
Not the compiler. Not an obvious PyExt logic bug (audited). A timing-sensitive
118+
heap corruption on the symbol-(re)load path that only manifests under TTD and
119+
resists navigable capture. Most likely a latent corruption inside, or in
120+
PyExt's interaction with, the `dbghelp`/`dbgeng` symbol engine (the
121+
`engextcpp`/dbgeng layer Option-3 removes), exposed only by TTD's timing.
122+
Artifacts kept under `fh4-repro/`: the clipped 700 MB TTD trace
123+
(`ttd/crash/iter7_heap/PyExtTest01.run`) and the four standalone repros.
82124

83125
## Reproduce
84126

0 commit comments

Comments
 (0)