refactor(gfql): route flow productized: chain_specializations layout, routes-off replay, shape registry + route harness, CI ledger

[lmeyerov]

The route-centric test flow from the #2054 review, folded into one PR at the owner's request (formerly #2054 + #2061 + #2057). Stacked on #2056.

• Layout: graphistry/compute/chain_specializations/{admission,hotpaths}.py (pandas/cuDF) and gfql/lazy/engine/polars/chain_specializations/{admission,hotpaths}.py; each lane next to the admission predicate its dispatcher calls; chain.py and the polars chain only dispatch; chain_fast_paths.py keeps the shared seed/index helpers. Behavior-neutral. Tests mirror the paths.
• Routes-off replay: GFQL_ROUTES_OFF=<route,...> (tests/conftest.py) makes named hot paths decline so every existing test replays through the other routes; @pytest.mark.route_engaged(...) marks the tests that assert a route serves (skipped in that mode); bin/test-routes-off.sh writes the per-route ledger; the gfql-routes-off CI matrix publishes it non-blocking. Ledger table from the 7-mode replay: every remaining id was an engagement pin or a filed class (gfql: the general chain path upcasts integer node/property columns to float where every fast lane keeps int (route-dependent dtypes) #2058 → fix(gfql): fixes found by the route sweeps: source dtypes on every route, polars prune decline, polars-gpu fused lane (#2058, #2053, #2064) #2062, gfql polars: the general (lowered) path fails on Cypher shapes that only the fast lanes serve #2059, gfql: rows(table=nodes, source=alias) multiplies rows for duplicate node ids and joins null ids to each other #2034).
• Registry + harness: tests/compute/gfql/routes/registry.py (80 shapes registered by the specialization tables themselves) and test_route_harness.py (per admitted cell: served-by, same-engine general-path value parity, pandas general-path set oracle; declines recorded as the attenuation ledger → gfql polars: the seeded lane declines alias-collision shapes its predicate admits (29 ms full chain vs 1.9 ms pandas) #2063).
• CI path filter (formerly ci: run the gfql lanes on chain-engine changes #2057): the gfql filter now includes the chain engine files, so tck-gfql and the gfql lanes run on chain changes.
• tck-gfql's per-route ledger is ledger: per-route conformance replay (GFQL_ROUTES_OFF) with a baseline diff tck-gfql#196.

Receipts on this head below; earlier receipts on the closed #2061 and #2057.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA

[lmeyerov]

Slice 2 added on the same PR (462202f): polars_seeded_lane_admits is the structural gate _try_seeded_chain_polars calls first (frame conditions stay in the body). Pins over the shared corpus: the admitted set (8 shapes), the lane never serves a shape it does not admit under the real dispatch, and called directly on an indexed fixture it serves every admitted non-colliding shape and declines the three alias-collision ones. Local: 115 passed / 1 xfailed on the two admission files; guards OK; mypy identical to master; ruff clean.

[lmeyerov]

maybe we take this as a chance to cleanup the code a bit, like

• polars/chain_specializations/{admission,hotpaths}.py: keep in same files to aid DRY, understanding, etc
• unit tests can then label (& verify) inputs that can tried against multiple hotpaths instead of just 1, increasing tested shapes

note: this is a disruptive refactor, so should rebase & redo once things settle (or prioritize for before more?) to avoid issues like incomplete/drifted refactors

[lmeyerov]

Agreed on both points, and on the sequencing.

Structure. Redo as graphistry/compute/gfql/lazy/engine/polars/chain_specializations/{admission,hotpaths}.py for polars and the eager twin graphistry/compute/chain_specializations/{admission,hotpaths}.py (the seeded typed-hop lane, single-node lane, RETURN-destination lane and their predicates move out of chain_fast_paths.py / chain.py together), so each specialization's gate and body sit in the same module and the dispatchers only import. Tests mirror those paths.

Corpus. The shape corpus becomes a registry, not a hand-written list: each specialization's own test module registers its shape table (the fast/bypass tables in test_chain.py, the seeded-resolution and lookup shapes, the collision matrix, the residual-polars shapes, …) tagged with the defect classes it exercises; the route harness then runs every registered shape against every route whose predicate admits it, asserting served-by and full-path parity. That is what lets one input be tried against several hot paths instead of one.

Sequencing. This PR stays as-is (behavior-neutral, green) but I am converting it to draft: the restructure is disruptive to chain.py and the polars chain, which #2055 and #2056 also touch, so those land first and the restructure is redone on top of the settled tree in one pass rather than carried through rebases.

[lmeyerov]

CI on 462202f: 77/77 check-runs success (kept as draft per the restructure plan above).

[lmeyerov]

CI receipt at ae307f6: 84 check-runs, all success (python-lint-types incl. the comment/type guards and the relative-import rule; test-polars 3.12 coverage audit with floors for the three new polars modules; tck-gfql; gfql-benchmarks; test-gfql-core; test-pandas-compat-gfql). Local at this head: cuDF lane 1390 pass / 2 xfail; broad CPU 12592 pass (the 2 failures were the polars-lane completeness pin for the harness file, which #2061 registers); routes-off replay per mode posted in the stacked PR. The dgx cuDF + polars-gpu lane runs on the stack head (#2062, which contains this PR) once the current dgx batch finishes.

[lmeyerov]

Routes-off replay ledger at the harness head (bin/test-routes-off.sh, local, cuDF on; suites test_chain/test_hop/test_gfql + tests/compute/gfql, ~13.5k tests per mode):

mode	failing ids after skipping marked pins	what they were
native-fast	25	7 engagement functions (hits==1 / served spies, now marked in #2061), the #2058 dtype class (fixed in #2062), the #2039 alias-collision shape's general-path dtypes (same class)
polars-seeded	12	5 engagement functions (index served / explain), marked
polars-plain	1	the harness serve-ratio pin, marked
index-hop	14	#2058 dtype parity ×3 functions (#2062), 2 GPU edge-match engagement tests (marked)
indexed-kernel	27	served-flag pins (marked), #2058 two-alias dtype parity (#2062)
cypher-fast	46	14 engagement functions (fused/col-stats/degree-fact consult, served lanes; marked), #2059's typed NotImplementedError on the polars rows op
all-off	57	the union above plus 4 combined-route engagement pins (marked) and #2034 (duplicate node ids multiply on the full path; the lanes answer once)

Zero result divergences outside the filed classes (#2058 → #2062, #2059, #2034). Ledger files: reviews/2054/ on #2061's branch. tck-gfql per-route ledger (graphistry/tck-gfql#196): every mode 4143 passed / 689 xfailed, 0 masked, 0 route-only.

[lmeyerov]

GPU receipt (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars, cudf 26.02.01 / cupy 13.6.0 / polars 1.35.2, safe_run, TEST_CUDF=1; suites test_chain.py, test_hop.py, test_chain_alias_column_collision.py, tests/compute/gfql/**) on the stack head 676232a (#2062, which contains #2055, #2056, #2054 and #2061): 13383 passed / 93 failed / 55 skipped / 44 xfailed / 4 xpassed. The 93 failures are exactly master 86de0f5's set in the same image (13227 passed / 93 failed: the polars-gpu fused grouped-aggregate lane and strictness tests, #2064): set difference empty, zero branch-specific failures.

[lmeyerov]

GPU receipt (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars, cudf 26.02.01 / polars 1.35.2, safe_run, TEST_CUDF=1; suites test_chain.py, test_hop.py, test_chain_alias_column_collision.py, tests/compute/gfql/**) on the restacked stack head a54ac4b (#2065, containing #2055 rework, #2056, #2054, #2061, #2062): 13386 passed / 95 failed / 38 xfailed / 4 xpassed. 93 of the failures are master 86de0f5's polars-gpu fused-lane set (#2064, fixed separately in #2066). The other 2 are strict XPASS of the #2043 prune_to_endpoints markers on cuDF 26.02: the prune shapes now agree fast-vs-full on this stack (the #2062 closure fix removed the divergence), so #2062 drops that marker in a follow-up commit (tree otherwise identical; #2065 restacked).

[lmeyerov]

CI receipt at 70ddc36 (restacked on the #2055 rework): 84 check-runs, all success. GPU receipt on the stack head above.

[lmeyerov]

GPU receipt (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars, cudf 26.02.01 / polars 1.35.2, safe_run, TEST_CUDF=1; suites test_chain.py, test_hop.py, test_chain_alias_column_collision.py, tests/compute/gfql/**) on the folded stack top 500265e (#2062, containing #2056 and #2054): 13532 passed / 7 failed / 37 xfailed / 4 xpassed, versus master 86de0f5's 13227 / 93 in the same image. Every one of the 7 is inside master's set (5 polars-gpu strictness cases and the routing-contract / indexed-bypass pins tracked on #2064 and #1803); zero branch-specific failures, and the fused-lane fix folded from #2066 clears the other 86.

[lmeyerov]

CI receipt at b19ac3c (folded: layout + routes-off replay + registry/harness + CI filter; includes the fold repair of the polars plain lane): 84 check-runs, all success. Local on the stack top: broad CPU + cuDF 12939 / 0; tck-gfql 4143 / 689 xfail; GPU receipt above.

[lmeyerov]

Rebased onto the #2056 head 5775f76 → b5ba132. The one conflict was the chain.py block this PR moves out plus #2056's new helper in the same region; resolved by keeping only the helper. The PR's own product delta is line-for-line identical before and after the rebase (sorted diff: 0 lines). Gates on the new head: routes harness, both specialization packages, collision matrix, chain, scoping (609 pass / 14 xfail incl. cuDF), lane completeness 12/12, lint/mypy unchanged vs master. CI 84/84.

[lmeyerov]

Post-#2056 audit of the moved code (owner ask: did the files this PR moves change in #2056?)

Checked mechanically at head 9616f11 (rebased onto master f45469c):

• Every function fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056 changed or added (13) is body-identical somewhere on this head, old or new location, except two this PR rewrites on purpose. In those two, fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056's edits are carried: the polars dispatch's only fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056 line (unique(subset=[ncol]), one row per node id) lives in the extracted _plain_single_hop_polars lane body; the seeded typed-hop Cypher fast path keeps fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056's typed tail_ops copies, and this PR's delta there is the import lines only.
• The one function on master absent from both locations is the inline _fp_node helper, replaced by polars_plain_single_hop_admits by design.
• Body diffs of the other relocated functions are this PR's own edits only (comment trims for the density guard, predicate extraction, absolute imports, annotations).
• The PR's own product delta is line-for-line identical before and after each rebase (0 drift lines, twice).
• fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056's own pins (collision matrix, internal columns, scoping, polars 2039 pins) plus the route harness on this head: 376 pass / 13 xfail incl. cuDF.

CI on 9616f11 in progress; posted when it lands.

[lmeyerov]

Read-through of the moved code vs master f45469c (owner: "don't trust mechanical; read them")

Read side by side, function by function, at head 9616f11:

• Eager gate native_fast_path_admits: master's inline conditions one for one (engine pandas/cuDF, no start_nodes; 1 op → unqueried node; 3 ops → unqueried nodes, simple single hop with no node matches/queries, no edge query, no zero-hop seed, no prune; distinct aliases; undirected declines with names or filters).
• _try_chain_fast_path: single-node branch statement-identical (index-or-filter rows, alias column placement, empty edges). Three-op body statement-identical from the alias tuple to the final drop_duplicates and _tag_fast_path_aliases, including both binding-collision declines (node alias == node id; edge alias == from-side binding), the unconstrained endpoint validation, the seeded typed-hop call, and the filtered branch.
• _seeded_typed_hop_pandas_cudf, _seeded_typed_return_dst_pandas_cudf, _single_node_rows_via_index_or_filter: bodies identical; only comment trims and the absolute import.
• Polars gate polars_plain_single_hop_admits / _plain_node / _plain_edge: master's _fp_node/_plain_edge and the two inline conditions (seed filter + no destination filter + directed → index consult; unconstrained or directed → skip-combine; filtered undirected → full chain).
• _plain_single_hop_polars: master's inline branch statement for statement (endpoint alignment, null-endpoint drop, per-side semi joins, the closure check for sides not closed by a filter, and fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056's unique(subset=[ncol], maintain_order=True)).
• _plain_seeded_index_hop_polars: master's inline block, same maybe_index_hop call and policy/registry gate (negated for the early return).
• _try_seeded_chain_polars, _seeded_typed_return_dst_polars: identical from the frame checks through the alias flags and _record_indexed_traversal.
• Dispatch order in the polars chain matches master: index consult → seeded lane → plain lane. _combine_edges keeps fix(gfql): chain result contract: binding-column aliases decline, duplicate ids answer once, no internal columns (#2050, #2051, #2067) #2056's shadow-restore scoping verbatim. chain.py keeps reject_alias_named_like_binding and calls the relocated fast path at the same point (policy-free only).

Nothing from #2056 is missing and nothing moved changed meaning. CI on 9616f11: 70 green, 15 running.

[lmeyerov]

CI on 9616f11: 86/86 green. Head is final; ready to merge.

[lmeyerov]

Corpus breadth (owner: "corpus.py seems myopic") and review ledgers (head f5cee3b)

The registry the harness runs is wider than corpus.py (80 shapes from four test modules, 161 route cases), but every frame was a 4–5 node graph with int64 ids and nothing structural to trip on. This head adds a frame dimension: the same 22 shapes now also run over string ids, nullable ids with nulls, duplicate ids, self-loops plus a cycle, an empty edge table, and no edge-id binding, on pandas, cuDF and polars (359 harness cases). Findings:

• One real divergence, already known: gfql: rows(table=nodes, source=alias) multiplies rows for duplicate node ids and joins null ids to each other #2034 (single-node predicate lookup keeps each duplicate-id row; the general path collapses them), pinned by row on both eager engines.
• The variants attenuate the lanes as expected (null/string ids decline the polars seeded lane; recorded in the xfail ledger, not counted in the serve-ratio pin).
• Everything else the variants tripped was harness or fixture: the oracle now normalizes null ids; known-divergence xfails are non-strict on variants where a shape can coincide.

Still narrow, and queued in the plan rather than grown by hand here: shapes should be harvested from the main GFQL shape suites (record every op list + small frames the suites execute and replay them through the harness) instead of cherry-picked; the harness wires three routes (native-fast, polars-plain, polars-seeded), not index-hop, indexed-kernel, cypher-fast or rows; no polars-gpu engine.

reviews/2054/ is no longer tracked; the replay ledger table stays in the comment above and the files live locally under plans/.