fix(polars): endpoint is_in shape errors on Polars 1.21 traversal

[lmeyerov]

Polars1.21.0 raises ComputeError: shapes don't match ... in is_in comparison during endpoint-resolvability filtering. This affects seeded typed-hop and OPTIONAL traversal queries on the Polars CPU engine installed in the RAPIDS25 test image.

Confirmed on both unmodified master 77bcbd337ad87c3d139e1f6f61fd041fd1563720 and combined PR2073 follow-ups 80271526f21e80e395b96c4fdbcaac6957b79c8c: the exact same33 test cases fail with the same error class. This is baseline compatibility debt, not a regression introduced by #2077/#2078/#2079 or #2080.

Reproduction environment: DGX Spark, graphistry/test-rapids-official:25.02-gfql, Polars1.21.0, cuDF25.2.2, pandas2.2.3, Python3.12. Commands were run sequentially under shared GPU locks and verified source manifests.

TEST_CUDF=1 TEST_POLARS_GPU=0 python -m pytest -q \
  graphistry/tests/compute/gfql/test_seeded_node_lookup_fastpath.py \
  graphistry/tests/compute/gfql/test_optional_match_semantics.py \
  -k polars --maxfail=0 --tb=short

Representative failing test: test_optional_varlen_arm_bound_endpoints_projects_bound_alias[polars]. Additional cases include indexed/scan seeded typed-hop property projections and categorical node columns. The master run reports33 failed,255 passed,261 deselected. Expanded candidate suite reports33 failed,929 passed,195 skipped. The same expanded candidate suite passes962 tests in RAPIDS26 and1045 in the Polars-GPU image; there are no cuDF test failures in the25 lane.

Root site: graphistry/compute/gfql/lazy/engine/polars/hop_eager.py, _keep_edges_with_both_endpoints_resolvable: universe = resolvable_ids.implode() followed by endpoint is_in(universe). Polars1.21 interprets this RHS incompatibly and reports expected lengths0/8/113/226 versus1. Blame traces the imploded-universe construction to 86013f47f0 (2026-08-15), with later endpoint casting in d0e23d881a (2026-08-19). These dates establish provenance of the current construction; they do not establish the first release affected.

Priority: P2 compatibility defect. Current Polars/RAPIDS26 paths work; older Polars traversal correctness coverage is affected. Do not silently skip these cases or substitute CPU execution for explicit GPU requests. A fix should exercise empty and nonempty universes, null endpoints, both directions, indexed/scan paths, categorical IDs and mixed matching/nonmatching endpoints on both supported Polars versions.

Related: #2074 (discovery and baseline separation), #1890 (GPU CI coverage), #1938 (existing is_in warning tracking), #2026 (endpoint filtering performance). This issue tracks the concrete old-version execution failure, rather than those broader coverage/performance tasks.

Evidence retained locally under plans/gfql-2073-followups/logs/: device-master2080/, device-baseline2080-v2/, master2080-baseline-failure-parity.json; immutable image metadata, full source/probe SHA256 manifests, JUnit case identities and complete stacks accompany the runs. Deferred from the existing PR2073 fix set because exact baseline comparison proves the independent pre-existing defect.

Current verification checkpoint

Refreshed exact master92ad8c0dbe0af7acc56923f42153594cf5ab30fc control reproduces the same33failure IDs/error types as the follow-up candidate. Final combinedffa52d8a expanded matrix:cuDF25image1053passed/215skipped/33baseline failures;cuDF26image1086passed/215skipped;PolarsGPU26image1301passed/0skipped. This supplements the historical77/802 evidence above; it does not establish first affected release. P2deferral remains unchanged; current parity artifact is logs/master2081-final/gate-and-baseline-audit.json under the local2074plan.

[lmeyerov]

Fix ready for review: #2085 (fix/gfql-2082-polars121-isin, base master 65c359b)

Root cause, pinned. polars changed the is_in right-hand-side contract in 1.28.0: below it a List-typed RHS (ids.implode()) is matched row-wise and must have the column's length, which is the reported shapes don't match: expected N elements in 'is_in' comparison, got 1; from 1.28 the bare-Series RHS is deprecated instead (pola-rs/polars#22149). Pinned by a per-version venv sweep over 14 releases 1.21.0 … 1.35.2: the imploded RHS fails through 1.27.1 and passes from 1.28.0; the bare RHS passes on every release and warns from 1.28.0.

Fix. One helper (lazy/engine/polars/membership.py: id_set, is_in_ids) spells the RHS for the installed polars; all 14 Series-membership sites route through it — hop_eager (the site reported here), chain_specializations/{hotpaths,point_rows}, pattern_apply, and two gfql_unified sites that still used the bare form and therefore still warned on >= 1.28 (#1938 item 5). On polars >= 1.28 the emitted expressions are unchanged, including the hoisted hop universe.

The blast radius was wider than the 33 cases here. On exact master in test-rapids-official:25.02-gfql (polars 1.21.0), graphistry/tests/compute/gfql -k polars fails 2361 cases and 1792 of them are this idiom — test_engine_polars_chain 729, hop boundary matrix 180, hop-labels differential 173, hotpaths 144, point_rows 97, and so on. The 33 named here are the two files this issue happened to sample. With the fix: 0 of that idiom, 1871 master failures pass, 0 new failures.

Verification (DGX Spark; exact trees shipped by git archive with sha256 manifests, shared benchmark lock, --memory capped, sequential):

lane	image (polars)	master	PR head
cuDF 25	25.02-gfql (1.21.0)	the 2 files: 33 failed / 255 passed	0 failed
cuDF 25	same	all gfql -k polars: 2361 failed	490 failed, 0 of this idiom, 0 new
cuDF 25	same	—	targeted + locks: 1237 passed, 88 failed — all polars-gpu params cudf-polars 25.02 cannot execute, 0 polars-CPU
cuDF 26	26.02-gfql (1.35.2)	288 passed	1325 passed / 0 failed
polars-GPU	26.02-gfql-polars, TEST_POLARS_GPU=1	288 passed	1325 passed / 0 failed

CI on the exact PR head (62e5b95aa): run 34938849389 SUCCESS — 85/85 jobs green, including all 7 test-polars cells (py3.9–3.14) and all 10 gfql-routes-off replays, which run the new tests again with each fast-path route disabled. Two jobs (test-dgl-cpu, test-full-ai (3.9)) were initially cancelled by their own 15-minute timeout inside dependency installation, before running any test; both passed on re-run, and neither touches anything this branch changes.

Also on the exact head: ruff, the type-hygiene and comment-density guards all clean; whole-tree mypy identical to master (258 pre-existing errors on both, 0 new); and the polars coverage lane plus coverage_audit --profile gfql-polars replicated, with the new module at 100% against its floor.

Controls. A mutant that always implodes re-breaks ~100 tests on 1.21. A mutant that always uses the bare form is caught only by recording warnings, because polars emits that deprecation from Rust and prints it under -W error — worth knowing for anyone writing similar pins. Beyond values, a static AST lock now fails on any is_in(...implode()) written outside the helper, since that spelling is green on every CI polars and fatal here; three mutations of hop_eager back to the pre-fix spelling each trip it.

One relabel, stated explicitly. Two of the 33 (test_categorical_node_columns_keep_parity_on_every_engine, "props from three aliases", scan+indexed) pass the is_in fix and then fail on polars 1.21 only because the fast path's categorical column comes back with categories ['f62'] while the full path keeps all 300: polars 1.21 keeps the full rev-map through filter/join and trims it after filter→join, 1.35 trims everywhere. The test's _canon now drops unused categories before comparing; values and categorical dtype still compare exactly. The category set is not a row contract.

Out of scope, pre-existing, still failing on polars 1.21: 365 polars-gpu NotImplementedError plus 54 GFQLUnsupportedError (cudf-polars 25.02 cannot execute those plans; the 25.02 GPU lane is not claimed), 3 Expr.round(mode=) (a polars >= 1.29 feature; the polars extra floor is 1.29), 4 row-op NotImplementedError, 4 harness artifacts.

Limitations. First affected release is still unknown: the imploded spelling landed 2026-08-15 (86013f4), so any release containing it fails on polars < 1.28. polars 1.21 remains below the polars extra floor — this makes the CPU engine run there, it does not claim feature parity. Separately, replicating the coverage lane showed lazy/engine/polars/dtypes.py at 87.30% against its 92.0 floor on unmodified master too, in the same image; this branch does not touch that file, so it is environmental (py3.13 with cuDF here vs py3.12 without in CI) and no baseline was changed for it.

[lmeyerov]

Merged as c9d7d53 via #2085.

Post-merge close-out:

• CI on the merged head c1959825b: run 35039348961 SUCCESS, 85/85 jobs, including all 7 test-polars cells (py3.9–3.14, polars 1.43.0) and all 10 gfql-routes-off replays.
• Version gate proven load-bearing. The suite previously only ever exercised whichever arm the installed polars selects, so a decorative gate would have passed everything. test_the_version_gate_is_load_bearing_on_the_installed_polars now forces the imploded arm and asserts the consequence version-agnostically: below 1.28 it must raise this issue's ComputeError, from 1.28 it must agree value-for-value with the routed arm. Neither lane skips.
• Performance, measured rather than argued. 12 of the 15 routed sites emit a byte-identical expression on polars >= 1.28; three (pattern_apply x2, the reachable gfql_unified OPTIONAL prune) genuinely change from a bare Series RHS to an imploded one. Interleaved A/B on DGX (polars 1.35.2, 300k nodes / 2M edges, 30 samples per arm per workload): two-hop endpoint gate (the perf(gfql): undirected k-hop chain re-runs the seed-rediscovery rule and the endpoint-resolvability filter on every pass #2026 path) median 80.6 -> 78.1 ms, seeded chain 9.5 -> 9.2 ms, EXISTS 312.6 -> 316.3 ms. All distributions overlap, nothing trips a 5%-and-2ms flag. The EXISTS minimum at +2.6% was isolated with a micro A/B of exactly the changed expression over 2M rows: imploded is 0.7-1.8% faster than bare, so that delta was box contention, not the diff. perf(gfql): undirected k-hop chain re-runs the seed-rediscovery rule and the endpoint-resolvability filter on every pass #2026 preserved.

Limitations unchanged: first affected release still unknown (the imploded spelling landed 2026-08-15 in 86013f4, so any release containing it breaks on polars < 1.28), and polars 1.21 remains below the polars extra floor of 1.29 — the CPU engine runs there, feature parity is not claimed.

[lmeyerov]

Closing the one limitation this issue was left with — the first affected release is 0.59.0, and it is the only affected release.

• 86013f47f (2026-08-15) introduced the imploded RHS; git tag --contains 86013f47f returns exactly one tag, 0.59.0 (released 2026-08-31).
• Verified in the release trees rather than inferred: 0.59.0:graphistry/compute/gfql/lazy/engine/polars/hop_eager.py:118 carries universe = resolvable_ids.implode(), and 0.58.0 (2026-07-21) contains no implode in that file at all.

So anyone on 0.59.0 running the polars CPU engine against polars < 1.28 — which RAPIDS 25.02 pins at 1.21 via cudf-polars — hits this; 0.58.0 and earlier are unaffected. Since 0.59.0 is the current release, that combination is broken on PyPI today, so this may be worth a patch release rather than waiting for the next minor.

Also checked while confirming: 0.59.0's hop_eager.py:557 had a second membership site with a bare-Series RHS (is_in(pl.Series(drop_seed_ids, ...))). It is gone from master — unrelated work removed it — so master's hop_eager now has exactly one is_in, the routed one, and the changelog's "removes the remaining bare-Series deprecation warnings" claim holds.