Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
f0c1260
perf(gfql): materialize indexed point rows before chain execution
lmeyerov Sep 7, 2026
ed4debe
perf(gfql): project point properties directly from indexed rows
lmeyerov Sep 7, 2026
c1860bd
perf(gfql): project joined point bindings from indexed edge rows
lmeyerov Sep 7, 2026
069ef37
perf(gfql): gather projected arrays and coalesce point properties
lmeyerov Sep 7, 2026
83f0379
fix(gfql): reject nulls in indexed scalar equality filters
lmeyerov Sep 7, 2026
4ecd505
perf(gfql): reduce indexed point materialization overhead
lmeyerov Sep 7, 2026
69ba33d
perf(gfql): batch Polars projection temporal text checks
lmeyerov Sep 7, 2026
70b97dd
perf(gfql): fuse Polars indexed join execution
lmeyerov Sep 7, 2026
564bc7d
perf(gfql): fuse Polars binding property attachment
lmeyerov Sep 7, 2026
75b27af
perf(gfql): avoid redundant Polars destination refilter
lmeyerov Sep 7, 2026
14b01e4
perf(gfql): gather native Polars point source rows directly
lmeyerov Sep 7, 2026
a93cccc
perf(gfql): project native Polars joined points from indexed frames
lmeyerov Sep 7, 2026
6447efa
test(gfql): register Polars point coverage and update alias admission…
lmeyerov Sep 7, 2026
135dec9
fix(gfql): honor pandas chain inputs and resolve CPU Dask consistently
lmeyerov Sep 8, 2026
7891565
perf(gfql): project singleton Polars paths without joins
lmeyerov Sep 8, 2026
a040bd8
fix(gfql): preserve cuDF bindings during point alias reordering
lmeyerov Sep 9, 2026
b296b55
perf(gfql): verify singleton Polars equalities without a filter collect
lmeyerov Sep 9, 2026
da2d1dc
fix(gfql): reuse valid property indexes across Polars execution targets
lmeyerov Sep 9, 2026
e88b9ee
perf(gfql): skip temporal regex scans for ordinary singleton strings
lmeyerov Sep 9, 2026
97b9296
perf(gfql): avoid redundant Polars point filters and alias collects
lmeyerov Sep 9, 2026
fa86d6f
perf(gfql): slice singleton Polars index hits without a gather
lmeyerov Sep 9, 2026
0ad725e
perf(gfql): validate chain wrappers once at explicit validation sites
lmeyerov Sep 9, 2026
26254fd
fix(gfql): preserve nullable Polars integer IDs in index lookups
lmeyerov Sep 9, 2026
4f0285c
perf(gfql): avoid redundant Polars named-hop collections
lmeyerov Sep 9, 2026
b74d2f5
perf(gfql): avoid repeated Polars schema reads on indexed queries
lmeyerov Sep 9, 2026
5f790c0
perf(gfql): reduce primitive filter and list validation overhead
lmeyerov Sep 9, 2026
ef6b172
perf(gfql): avoid uniqueness scans for singleton joined rows
lmeyerov Sep 9, 2026
afa372e
perf(gfql): filter small Polars equality results without expression p…
lmeyerov Sep 9, 2026
4161025
test(gfql): simulate missing cuGraph independently of installed backends
lmeyerov Sep 9, 2026
7de2090
perf(gfql): assemble tiny Polars index gathers from slices
lmeyerov Sep 9, 2026
edc237f
perf(gfql): skip singleton Polars frontier uniqueness scans
lmeyerov Sep 9, 2026
73565dc
fix(gfql): preserve Polars whole-entity identity after binding rows
lmeyerov Sep 9, 2026
62aa1cd
Harden point-route contracts and regression replay
lmeyerov Sep 12, 2026
58f3cf1
Keep review changes within type and comment hygiene guards
lmeyerov Sep 12, 2026
d53bcd3
Preserve traversal column order for empty pandas results
lmeyerov Sep 12, 2026
c73c200
Restore shadowed properties by entity key and register Polars tests
lmeyerov Sep 12, 2026
969bb9d
fix(gfql): preserve nullable integer identity in physical indexes
lmeyerov Sep 12, 2026
01fcf9e
fix(gfql): preserve row identity and order across execution routes
lmeyerov Sep 12, 2026
11c1f0c
fix(gfql): retain node selection edge schema and CI coverage
lmeyerov Sep 12, 2026
fc67887
ci(gfql): enforce replay for every registered route
lmeyerov Sep 12, 2026
642be44
test(gfql): scope seed engagement pins to their actual routes
lmeyerov Sep 12, 2026
b5703a3
Avoid expression plans for resolved Polars projections
lmeyerov Sep 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1598,19 +1598,18 @@ jobs:
./bin/test-graphviz.sh

gfql-routes-off:
# Non-blocking ledger: replays the GFQL suites with one hot path declined per cell
# Required replay: run the GFQL suites with one hot path declined per cell
# (GFQL_ROUTES_OFF); engagement pins are skipped via the route_engaged marker, so the
# uploaded lists hold route-vs-general result divergences only.
needs: [changes, test-gfql-core, generate-lockfiles]
if: ${{ needs.changes.outputs.gfql == 'true' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' }}
runs-on: ubuntu-latest
continue-on-error: true
timeout-minutes: 45

strategy:
fail-fast: false
matrix:
mode: [native-fast, polars-seeded, polars-plain, index-hop, indexed-kernel, cypher-fast, all-off]
mode: [polars-point-rows, point-rows, native-fast, polars-single-node, polars-seeded, polars-plain, index-hop, indexed-kernel, cypher-fast, all-off]

steps:

Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm

### Fixed

* GFQL: indexed scalar node and directed single-hop queries followed by `rows(source=...)` and optional `select(...)` now produce row tables directly on pandas, cuDF, and Polars. Joined property projections preserve repeated edge matches without constructing full traversal results.
* GFQL: indexed scalar equality filters no longer retain null-valued rows when the comparison produces a nullable boolean mask.

* GFQL: a native chain whose edge alias is named like the source, destination or edge-ID binding column silently overwrote that binding on pandas and cuDF (the seed's edges vanished, or the edge ids became `True`) and raised a raw polars `SchemaError`; it is now the same typed decline (E108) the node-ID collision already gets, on every engine, before execution — the rule the Cypher route already applied (#2050).
* GFQL polars: a colliding alias no longer leaves the internal `__gfql_shadow_restore__<alias>__` column on op-list results; the chain keeps the shadowed values under that name only while a compiled Cypher pipeline runs (its row pipeline reads them back), and on `gfql([...])` / `chain([...])` the marker simply shadows the column as on pandas and cuDF. Nothing is stripped from results: user-defined columns and pipelines composed from successive `gfql` calls are untouched.

Expand Down
11 changes: 11 additions & 0 deletions bin/test-polars.sh
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,12 @@ POLARS_TEST_FILES=(
graphistry/tests/compute/gfql/test_native_seed_skip_refilter.py
graphistry/tests/compute/gfql/test_polars_native_seed_resolution.py
graphistry/tests/compute/gfql/lazy/engine/polars/chain_specializations/test_polars_admission.py
graphistry/tests/compute/gfql/lazy/engine/polars/test_predicates.py
graphistry/tests/compute/gfql/lazy/engine/polars/chain_specializations/test_point_rows.py
graphistry/tests/compute/gfql/lazy/engine/polars/chain_specializations/test_hotpaths.py
graphistry/tests/compute/gfql/test_polars_indexed_join_helpers.py
graphistry/tests/compute/chain_specializations/test_native_admission.py
graphistry/tests/compute/chain_specializations/test_point_rows.py
graphistry/tests/compute/gfql/test_undirected_pairs_2026.py
graphistry/tests/compute/gfql/test_native_seed_lane_explain.py
# #1882/#1913-f4/#1879 crash-family pins: the polars params (filter helpers on polars
Expand All @@ -103,6 +108,7 @@ POLARS_TEST_FILES=(
graphistry/tests/compute/gfql/test_polars_rows_entity_groupby.py
graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py
graphistry/tests/compute/gfql/test_residual_polars_native.py
graphistry/tests/compute/gfql/index/test_engine_arrays.py
graphistry/tests/compute/gfql/index/test_auto_engine_agreement.py
graphistry/tests/compute/gfql/index/test_degree_consult.py
graphistry/tests/compute/gfql/test_single_alias_cache_key.py
Expand All @@ -127,6 +133,11 @@ POLARS_TEST_FILES=(
graphistry/tests/compute/test_chain_alias_column_collision.py
graphistry/tests/compute/test_gfql_op_list_hides_internal_columns.py
graphistry/tests/compute/gfql/routes/test_route_harness.py
graphistry/tests/compute/gfql/routes/test_point_boundaries.py
graphistry/tests/compute/gfql/routes/test_has_collision_contract.py
graphistry/tests/compute/gfql/routes/test_node_selection_rows.py
graphistry/tests/compute/gfql/test_join_backend_contracts.py
graphistry/tests/compute/test_chain_validation_execution.py
graphistry/tests/compute/gfql/test_engine_polars_semi_key_dedup.py
graphistry/tests/compute/gfql/test_engine_polars_call_modality.py
graphistry/tests/compute/gfql/test_engine_polars_gpu.py
Expand Down
13 changes: 8 additions & 5 deletions bin/test-routes-off.sh
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
#!/usr/bin/env bash
# Replay the GFQL suites with each hot path declined (GFQL_ROUTES_OFF, see graphistry/tests/conftest.py).
# Engagement pins carry the route_engaged marker and are skipped, so every remaining failure is a
# route-vs-general result divergence. Non-blocking ledger: always exits 0; per-mode logs + id lists in $OUT.
# route-vs-general result divergence. Fail on any failed replay; retain per-mode logs and id lists.
set -uo pipefail
cd "$(dirname "$0")/.."
MODES=${MODES:-native-fast polars-seeded polars-plain index-hop indexed-kernel cypher-fast all-off}
# One registry drives forcing and replay: new routes automatically join all-off.
ALL_ROUTES=$(python -c 'from graphistry.tests.compute.gfql.routes.switch import ROUTES; print(",".join(ROUTES))') || exit $?
MODES=${MODES:-${ALL_ROUTES//,/ } all-off}
SUITES=${SUITES:-graphistry/tests/compute/test_chain.py graphistry/tests/compute/test_hop.py graphistry/tests/compute/test_gfql.py graphistry/tests/compute/gfql}
OUT=${OUT:-build/routes-off}
mkdir -p "$OUT"
status=0
for mode in $MODES; do
if [ "$mode" = all-off ]; then routes=native-fast,polars-seeded,polars-plain,index-hop,indexed-kernel,cypher-fast; else routes=$mode; fi
GFQL_ROUTES_OFF=$routes python -m pytest $SUITES -q -p no:cacheprovider -o addopts="" -rfE > "$OUT/$mode.log" 2>&1
if [ "$mode" = all-off ]; then routes=$ALL_ROUTES; else routes=$mode; fi
GFQL_ROUTES_OFF=$routes python -m pytest $SUITES -q -p no:cacheprovider -o addopts="" -rfE > "$OUT/$mode.log" 2>&1 || status=1
grep -E "^(FAILED|ERROR) " "$OUT/$mode.log" | sed 's/ - .*//' | sort -u > "$OUT/$mode.divergences"
echo "$mode: $(wc -l < "$OUT/$mode.divergences") divergence id(s); $(tail -1 "$OUT/$mode.log")"
done
exit 0
exit "$status"
6 changes: 6 additions & 0 deletions graphistry/Engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,12 @@ def resolve_engine(
except ImportError:
pass

type_module = type(g_or_df).__module__
if 'dask' in type_module and 'cudf' not in type_module:
import dask.dataframe as dd
if isinstance(g_or_df, dd.DataFrame):
return Engine.PANDAS

if 'cudf.core.dataframe' in str(getmodule(g_or_df)):
has_cudf_dependancy_, _, _ = lazy_cudf_import()
if has_cudf_dependancy_:
Expand Down
38 changes: 36 additions & 2 deletions graphistry/compute/chain.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
from .typing import DataFrameT, SeriesT
from .util import generate_safe_column_name
from .chain_specializations.hotpaths import _try_chain_fast_path
from .chain_specializations.point_rows import _try_point_rows
from .engine_coercion import ensure_local_engine_match
from graphistry.compute.validate.validate_schema import validate_chain_schema, validate_graph_shape
from graphistry.compute.gfql.strictness import StrictInput
from graphistry.compute.gfql.same_path_types import (
Expand Down Expand Up @@ -545,6 +547,9 @@ def apply_output_slice(op: ASTObject, op_label: ASTObject, df):
out_df[base] = out_df[c_x].where(out_df[c_x].notna(), out_df[c])
out_df = out_df.drop(columns=[c, c_x])

# Empty pandas merges can move the binding column behind aliases or properties.
if out_df.columns[0] != id:
out_df = out_df[[id, *[column for column in out_df.columns if column != id]]]
return out_df


Expand Down Expand Up @@ -969,6 +974,8 @@ def _chain_with_strictness(
"Install RAPIDS/cudf_polars, or use engine='polars' for native CPU execution."
)
self = _coerce_input_formats(self, engine_concrete_early)
if engine_concrete_early == Engine.PANDAS:
self = ensure_local_engine_match(self, engine_concrete_early)

if engine_concrete_early in POLARS_ENGINES:
# Native polars chain lives in a dedicated dispatched module so the
Expand All @@ -977,7 +984,8 @@ def _chain_with_strictness(
# POLARS_GPU = the same lazy engine with the GPU execution target.
# (Dependency guards for polars / cudf_polars are above, pre-coercion.)
if validate_schema:
Chain(ops if not isinstance(ops, Chain) else ops.chain).validate(collect_all=False)
# Construct a fresh validator: the constructor validates children once.
Chain(ops if not isinstance(ops, Chain) else ops.chain)
validate_graph_shape(self, ops, collect_all=False) # pandas gets this via validate_chain_schema (#1889)
from graphistry.compute.gfql.lazy.engine.polars.chain import chain_polars
from graphistry.compute.gfql.lazy import target_mode, ExecutionTarget
Expand Down Expand Up @@ -1005,6 +1013,10 @@ def _chain_with_strictness(
finally:
call_thread_local.policy = old_policy
else:
point_ops = ops.chain if isinstance(ops, Chain) else ops
point_result = _try_point_rows(self, point_ops, engine_concrete_early, start_nodes, validate_schema)
if point_result is not None:
return point_result
return _chain_impl(self, ops, engine, validate_schema, policy, context, start_nodes)


Expand Down Expand Up @@ -1042,7 +1054,8 @@ def _chain_impl(
ops = ops.chain

if validate_schema:
Chain(ops).validate(collect_all=False)
# Revalidate mutable operations on every execution, including reused Chains.
Chain(ops)

from graphistry.compute.ast import ASTCall

Expand Down Expand Up @@ -1231,6 +1244,27 @@ def _chain_impl(
from .gfql.exec_context import clear_row_exec_context
g_out = clear_row_exec_context(g_out)
success = True
elif len(ops) == 1 and isinstance(ops[0], ASTNode):
# A node selection preserves each source row. Rejoining node IDs in
# the traversal combine would multiply duplicate rows and properties.
g_out = g_stack[0]
alias = ops[0]._name
if alias is not None:
cols = [c for c in g_out._nodes.columns if c != alias]
if g._node in cols:
cols = [g._node, *[c for c in cols if c != g._node]]
cols = ([*cols, alias] if alias in g._nodes.columns
else [*cols[:1], alias, *cols[1:]])
g_out = g_out.nodes(g_out._nodes[cols].reset_index(drop=True))
if synthesized_empty_edges:
g_out = self.nodes(g_out._nodes, g._node)
elif added_edge_index:
g_out = self.nodes(g_out._nodes, g._node).edges(
g_out._edges.drop(columns=[g._edge]), edge=original_edge)
elif g._edge is not None:
edge_cols = [g._edge, *[c for c in g_out._edges.columns if c != g._edge]]
g_out = g_out.edges(g_out._edges[edge_cols])
success = True
else:
# Phase 2: Backward pass to propagate downstream constraints.
g_stack_reverse : List[Plottable] = []
Expand Down
Loading
Loading