diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ac2f408572..a1cbbe585d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -921,6 +921,22 @@ jobs: printf 'Differential/parity tests:\n- %s\n' "${parity_tests[@]}" python -B -m pytest -vv -n auto "${parity_tests[@]}" + gpu-gate-audit: + # Ungated: a guard against a silent coverage gap must not itself be conditionally + # silent, and the path filters do not cover every file it cross-checks. + name: gpu-gate-audit + runs-on: ubuntu-latest + timeout-minutes: 3 + steps: + - name: Checkout repo + uses: actions/checkout@v4 + with: + persist-credentials: false + + - name: Audit cuDF test gates and the CI coverage they lack + run: | + python bin/ci_gpu_gate_audit.py + cypher-frontend-surface-guard: name: cypher-frontend-surface-guard needs: [changes] @@ -1577,7 +1593,9 @@ jobs: needs: [changes, test-minimal-python, test-gfql-core, generate-lockfiles] if: ${{ ((needs.changes.outputs.python == 'true' && needs.changes.outputs.narrow_python_only != 'true') || needs.changes.outputs.gfql == 'true' || needs.changes.outputs.pandas_compat == 'true' || needs.changes.outputs.core == 'true' || needs.changes.outputs.infra == 'true' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule') && !(needs.changes.outputs.docs_only_latest == 'true' && (github.event_name == 'push' || github.event_name == 'pull_request')) }} runs-on: ubuntu-latest - timeout-minutes: 10 + # The py3.12 coverage cell takes ~9m40s on a 2-vCPU hosted runner before + # setup, the coverage audit, and artifact upload; keep deterministic margin. + timeout-minutes: 15 strategy: matrix: diff --git a/CHANGELOG.md b/CHANGELOG.md index 78b4fe42e3..6d0d3c1f5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,14 +5,47 @@ All notable changes to the PyGraphistry are documented in this file. The PyGraph The changelog format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and all PyGraphistry-specific breaking changes are explictly noted here. -## [Development] +## [0.59.0 - 2026-08-31] +### Breaking + +- **GFQL strictness levels for absent labels/properties, defaulting to `warn` (#1916)**: an absent label or property no longer raises `GFQLSchemaError [column-not-found]` by default. Working on a subgraph with partial columns is normal usage, not a typo, so the default now follows openCypher and resolves an absent name to null: an absent label matches nothing (`MATCH (n:Nope)` is 0 rows, not an error), an absent property in `WHERE` or in a pattern map makes the predicate null so the row does not match (0 rows), `IS NULL` on an absent property is true (all rows), and an absent property in `RETURN` stays a null column. Three levels are selectable through the existing precedence chain (explicit `strict=` parameter, then `bind(schema=...)`'s `strict`, then its `metadata['strict']`, then the default): `"strict"` raises exactly the errors master raised, `"warn"` (default) warns once per distinct absent name per call, `"quiet"` is silent. The legacy boolean spelling maps on: `strict=True` is `"strict"` and `strict=False` is `"quiet"`, so both existing spellings are behavior-preserving and only the unset default moves. `strict=` is now accepted on `gfql()`, `chain()`, `gfql_validate()`, `gfql_remote()`, `gfql_remote_shape()`, `chain_remote()` and `chain_remote_shape()`. The validator and every executor (pandas, polars, cuDF, remote preflight) consult one shared resolution, so they cannot disagree the way they did on master, where `gfql_validate(strict=True)` rejected `RETURN n.nope_col` that execution then served. Two consequences of that agreement: under `"strict"`, `RETURN` of an absent property now raises where master returned a null column, and a direct `g.filter_nodes_by_dict({'absent': 1})` — not a GFQL call — keeps raising unchanged. A name absent from a DECLARED schema is still a typo and raises at every level; only a name the schema declares but this instance lacks is served leniently, which is the narrow-subgraph case `bind(schema=...)` exists to distinguish. Relatedly, a `type`/`labels` equality now resolves through a per-label boolean `label__X` column when the frame carries labels that way -- the mirror of the existing `label__X: True` rewrite -- so such a graph answers `-[:X]->` instead of matching nothing under the new default. +- **Remote GFQL sends the resolved strictness level (#1916)**: `gfql_remote()` previously hardcoded its client-side preflight to `strict=False` and sent the server nothing, so the same query was strict locally and loose remotely. The preflight now honors the resolved level, and the request body carries a new `strictness` field (`"strict"` / `"warn"` / `"quiet"`) alongside the existing `engine` field. **Server-side honoring is a server change and is not in this repository**: a server that does not read `strictness` applies its own default, so a non-default level requested remotely warns once, in the same shape as the existing Let/DAG compatibility warning (#1955). A client holding only a `dataset_id` can now also preflight names when `bind(schema=...)` supplied them, since a declared schema is names without data. + ### Fixed - **The 30M-edge GPlus filter/PageRank page now reports the completed Neo4j + GDS lane**: a locked twelve-slot follow-up produced a direct 354.47 s median-of-slot-medians with exact selected-node parity. The page and regenerated chart read the value from the vendored pyg-bench document. They publish no GFQL-vs-Neo4j ratio because Neo4j includes server round trips and a per-iteration GDS projection rebuild while GFQL retains resident frames. - **Polars graph-preserving Cypher CALLs retain their requested engine**: a CALL inside a compound `GRAPH ... USE ... CALL` query could return pandas frames after an igraph analytic even when the query requested `engine="polars"`. CALL-based graph constructors now restore the requested dataframe engine before returning, with value-parity and node/edge frame-type regression coverage. - **GFQL benchmark docs now enforce the benchmark contract v3 boundary**: the vendored pyg-bench artifact and contract suppress invalid cross-profile ratios. Filter/PageRank no longer divides resident in-process GFQL timings by a per-iteration Neo4j projection rebuild, and the GraphBench q1–q9 board no longer divides reused GFQL bindings by Kuzu's execute-text-per-call profile or treats cache-contaminated q8 values as results. The pages and generated charts retain direct timings and same-profile ratios only, and independently re-verify ratio operands, profiles, disclosure propagation, and derived-cell strength before rendering. - +- **Explicit unsupported remote engines now decline before side effects (#1957 completion)**: `gfql_remote` and `python_remote` preserve explicit `pandas` and `cudf` requests on the wire, while unsupported requests such as `polars` and `polars-gpu` raise typed `GFQLRemoteError` `E405` before credential refresh, upload, or POST. This is the remote service boundary, distinct from the release's local `polars`/`polars-gpu` GFQL engines; the existing `engine='auto'` policy is unchanged. +- **Strict GFQL validation now rejects relationship types that the edge schema proves absent (#1916)**: a strict binder previously deferred any relationship type when its catalog listed no known types, even when the edge schema had no generic `type` carrier, so `gfql_validate()` passed a query that execution rejected. It now raises typed `E301` when absence is provable. A generic `type` carrier with no declared catalog remains unjudgeable without scanning values and still defers, while an explicitly empty declared catalog is judgeable and rejects. Focused validator/executor and binder tests pin all three boundaries. +- **Cross-kind `WITH` whole-entity rebinds now fail early with typed `E108` (#1937)**: the local Cypher compiler guarded node-to-node and edge-to-edge rebinds but let a bare MATCH-bound node alias take a live edge alias's name, or the reverse, which could resolve rows and properties against different bindings. The guard now rejects any bare entity alias renamed onto another live pattern alias at compile time. Carries and self-renames, fresh targets, scalar/property shadows, terminal `RETURN` renames, `WITH`-to-`MATCH` reentry, and earlier, more specific validation errors keep their existing behavior. Focused tests pin both cross-kind error directions and the adjacent valid and precedence boundaries. +- **All-null Boolean `sum()` on `engine='polars-gpu'` now returns integer zero instead of null (#1997)**: cudf-polars 26.02 reports null for an all-null Boolean reduction, while GFQL's documented Boolean aggregate extension follows the Cypher `sum()` empty-input identity and returns `0`. The result normalization now fills only Boolean `sum` before the shared Int64 cast; Boolean `count` and non-Boolean `sum` keep their existing null behavior at this helper boundary. Direct boundary tests pin the positive cell and both negative controls. +- **Polars-GPU contract tests now distinguish correct fallback from fused-lane capability (#1997 follow-up)**: explicit `engine='polars-gpu'` strictness tests now cover absent-label and absent-property values plus strict/warn/quiet behavior. Grouped-aggregate engagement canaries runtime-xfail only when the GPU fused lane actually declines, and only after the generic GPU-targeted fallback matches its eager/pandas oracle; CPU Polars and any future serving GPU path remain strict must-serve assertions. This keeps a known cudf-polars capability gap visible without treating correct fallback answers as regressions or letting an expected failure mask a wrong answer. +- **A whole-entity endpoint projection (`RETURN b`) answered a deduplicated node set instead of the openCypher bag (#1994)**: on nodes 1-5 with edges (1,2) (1,3) (2,3) (3,4), `MATCH (a)-->(b) RETURN b` returned 3 rows where openCypher returns 4 — node 3 is bound twice, once from node 1 and once from node 2 — and `MATCH (a)-->(b) RETURN a` returned `[1, 2, 3]` for the 4-row bag `[1, 1, 2, 3]`. Parallel edges made it starker: two 1->2 edges are two matches, but the answer could not represent them at all. It was **silent**, and the engine disagreed with itself: every *property* spelling of the same projection (`RETURN b.id`, and even `RETURN a, b`) was already bag-correct, so only the single whole-entity spelling was wrong. Two independent vetoes sent it to the per-alias node table, which *is* a set: the multiplicity predicate bailed on any bare-alias projected item, and the projection lowering vetoed binding rows whenever the plan had a whole-row output. Fixing the lane alone was not enough — the polars projector could not render a whole entity off a binding-row frame at all, which is why `MATCH (a)-->(b) RETURN a, b` raised `NotImplementedError` on polars while pandas and cuDF answered it. That projector now resolves each alias's `{alias}.{field}` columns through a per-alias view (the polars twin of the pandas `_projection_alias_rows`), so single-entity and multi-entity binding rows render alike and the polars decline is gone. Four scopes are deliberately unchanged: `RETURN DISTINCT b` keeps the node-set lane (DISTINCT asks for exactly that dedup, and the binding-row frame carries sibling-alias columns a lone whole-row output does not functionally determine), a whole-row `WITH` carry into a trailing `MATCH` keeps it too (re-entry cannot yet separate matched from unmatched rows on a duplicated prefix, so #1935 item 1 stays open rather than turning into a decline), a variable-length arm keeps it (its bag is the relationship-unique walk expansion, not the edge bag this lane counts — that shape's own whole-entity/property disagreement is left open rather than swapped for a second unvalidated answer), and a pattern with no relationship has no multiplicity to keep. The seeded fast path recognizes the whole-entity bag lowering and re-expands one destination row per matched edge, so the LDBC IS5 entity shape (200k nodes / 1M edges, pandas, median of 20) stays on the fast lane at 7.8ms against 7.9ms before, rather than the 65ms the general lane costs; it defers on a zero-row bag so the empty-frame dtype contract stays single-sourced in the full path. An unseeded whole-entity scan necessarily gets slower in proportion to the rows it stopped dropping (94ms/199k rows before, 863ms/1.0M rows after). +- **Every string predicate over a CATEGORICAL column answered an empty/null result on cuDF where pandas answered rows**: `MATCH (n) WHERE searchAny(n, 'x', {columns: ['cat']}) RETURN n.id` over a categorical `cat` returned 15 rows on pandas and **0 rows on cuDF** — silently, with no warning and no error. A categorical-of-strings is string-VALUED on every engine, but only pandas lends it a `.str` accessor; cuDF raises `AttributeError` on `.str` for a categorical. The predicates' accessor probe read that raise as "this column is not string-valued" and returned the non-string result for the whole column — null, or `False` under the `na=False` that `searchAny` passes, so every row was dropped. `Contains`/`Startswith`/`Endswith`/`Match`/`Fullmatch` now decode a categorical whose CATEGORIES are strings back to its string values before the accessor, which is exact and null-preserving on both engines, and the unguarded `isalpha()`-family predicates take the same path instead of surfacing the raw `AttributeError` cuDF-only. A categorical with NUMERIC or temporal categories is unchanged and still refuses to stringify — `searchAny` keeps declining it on cuDF with a typed `NotImplementedError`, because that rendering diverges pandas↔cuDF. pandas answers are unchanged; polars already declined `searchAny` with explicit `columns=` and is unaffected. +- **NULL edge endpoints now follow one identity-resolution contract on all three engines (#1995)**: production answered this both ways -- eight sites implemented "a null never links" while the polars hop's `_keep_edges_with_both_endpoints_resolvable` (#1888 round 6) resolved a NULL endpoint to a NULL node id, so `MATCH (a)-[x]-(b) RETURN count(*)` over a graph with one NULL endpoint answered polars 4, pandas 6, cuDF 6. The contract is now stated in `docs/source/gfql/spec/language.md`: **a NULL id is not a graph identity**, so an edge with a NULL endpoint matches no pattern edge on any surface, from either direction, with a bound or synthesized node table. Input validity is a separate policy: this change preserves permissive DataFrame ingestion and current node-only row scans without declaring NULL-id source rows valid graph nodes, while `OPTIONAL MATCH` NULL bindings remain valid result values. Two defects that were wrong under either endpoint policy are fixed: the polars hop kept a NULL-endpoint edge whose NULL endpoint got no node row, and pandas/cuDF answered the same undirected chain with 2 edges unnamed and 3 edges named. Three kernels enforce endpoint resolution (shared pandas/cuDF `hop`, polars `hop_eager`, polars chain fast path). The seven strict-xfail cells from #1888 rounds 6-7 are removed and replaced by 13 green contract pins plus one non-strict compatibility probe (136 engine-parametrized cells, 39 red at the merge base) over both-sided-NULL, NULL-free, and string-id fixtures. +- **`hop()` on polars returned duplicate node rows where pandas returned one (#1895 residual)**: a node table with a repeated id (`id = [0, 0, 1]`) came back from `hop()` with both `id = 0` rows on polars and one on pandas — a silent cross-engine row-count divergence, not an error. The polars node output is a semi-join against the input table, which emits *every* matching left row; pandas de-dups by id in its edge-guarded output epilogue. The polars kernel (eager and lazy arms) now runs the same edge-guarded de-dup, so both engines emit one output node row per id. +- **Two silently-wrong row counts: a seeded typed 1-hop and a leading `OPTIONAL MATCH` both answered a deduplicated node set instead of the openCypher bag (#1899, #1903)**: `MATCH (a {name:'Ann'})-->(b) RETURN b.id` returned `[2]` where Ann has two parallel edges to Bob and openCypher returns `[2, 2]`, and `OPTIONAL MATCH (a)-->(b) RETURN a.id` returned `[1, 2, 3]` where the four-edge bag is `[1, 1, 2, 3]`. Both were **silent** — no warning, a plausible answer, and the unseeded/non-OPTIONAL spellings of the same patterns were already right, so the two shapes disagreed with their own siblings. Two compile-time carve-outs caused it. The first claimed a selectively-seeded single hop projecting only destination properties was "already answered value-correctly" by the seeded typed-hop fast path; the fast path answers with a destination-node SET, so the claim was false exactly when parallel edges exist. The second vetoed binding rows for *any* query containing an OPTIONAL clause, which is far broader than the null-extension it protects: a **leading** OPTIONAL MATCH binds nothing before it, so no row can go unmatched and it is a plain MATCH for row purposes (the zero-match case is served by the empty-result-row null extension, which never consults binding rows). Both carve-outs are gone, so the lowering is now unconditionally bag-correct and the fast path is a pure optimization: it recognizes the multiplicity-preserving `rows(binding_ops=...)` lowering of the same seeded shape and re-expands one destination row per matched edge, keeping the LDBC IS5 shape on the fast lane (measured ~7.6ms vs ~7.4ms before, against ~124ms on the general lane). Two consequences are deliberate and named. (a) Shapes where the Cypher fast path DECLINES (a datetime property, a requested-vs-actual engine mismatch) no longer reach the *native* chain fast path either, so on an un-indexed pandas graph they now show the same `int64 -> float64` rows-pivot upcast every other un-indexed pandas lane already shows. That artifact is pre-existing and still tracked; what changed is that the declined and served shapes now agree. (b) The fast path's pandas dtype rule now follows the lane it is standing in for: the upcast is a rows-pivot artifact, and an INDEXED bag lowering is served by the indexed connected-bindings kernel, which never pivots — so on an indexed graph the seeded projection keeps `int64`/`bool`, which is both its own generic answer and what polars and cuDF already return. Un-indexed keeps the pivot dtypes. Both directions are pinned (`test_indexed_bindings::test_destination_property_projection_dtype_parity`, `test_seeded_typed_hop_fastpath::test_pandas_int_bool_dtype_parity`). +- **`size()` over a non-list, non-string column answered with the TABLE ROW COUNT (#1985)**: `size(n.age)` on an `int64` column returned `3` for every row of a 3-row table and `7` for every row of a 7-row table — the height of the containing frame, not a property of the data, so adding unrelated rows changed the answer. In a predicate it was worse: `WHERE size(n.age) = 3` kept **every** row of a 3-row table and **no** row of a 4-row table. The pandas/cuDF row pipeline swallowed the typed `AttributeError` from its sequence-length helper and fell through to `len()`. Two sibling call sites swallowed the same failure and answered from a fabricated element count of zero: `any()`/`single()` said `false` and `all()`/`none()` said `true` about a column with no elements at all, and a list comprehension over such a column yielded `[]`. All three now raise a typed decline naming the limitation, matching the handling the slice-subscript and `ORDER BY`-list call sites two lines away already had, and matching the native polars lowering, which already declined a non-sequence operand rather than replicate the quirk. **What still answers is unchanged**: openCypher defines `size()` over strings, so `size()` remains CHARACTER length on pandas, polars and cuDF, `size()` remains the element count, `size([1,2,3])`/`size('abc')` still count the literal, and an all-null column still answers null rather than declining. Only inputs for which no size is defined — numeric, boolean, temporal — changed, and they changed from a wrong answer to a decline. +- **`OPTIONAL MATCH` after `WITH` blanked a carried whole-entity alias on its null-extended row, and cuDF rendered the blanks as `False` (#1897)**: for `MATCH (a:P) WITH a AS p, a.id AS pid LIMIT 2 OPTIONAL MATCH (p)-[:KNOWS]->(b) RETURN p, pid, b.id AS bid`, the row synthesized for the carried row that matched nothing came back with every `p.*` column NULL — even though `pid` on that same row proved `p` was still bound to `a2`. OPTIONAL MATCH cannot unbind an alias the prefix already bound, so only the suffix-bound outputs (`bid`) may go NULL; `p` keeps its own node columns. pandas answered NULL, and cuDF turned the NULL booleans into `False`, a definite wrong VALUE rather than a missing one. The null-fill copied only the carried SCALAR columns off the prefix frame; it now also copies the carried aliases' flat `alias.prop` entity columns, which the prefix frame already carries under the same names the result uses. Anti-join keys are unchanged (still the scalar columns), so no shape that previously null-extended changes which rows it fills. polars declines this shape earlier with its typed scalar-carry `NotImplementedError` and is unaffected. +- **An adjacency index changed the answer to `EXISTS { }` pattern predicates on polars (#1986)**: attaching an index (`gfql_index_all()`) made `MATCH (n) WHERE EXISTS { (n)-->(n) } RETURN n.id` return every node with *any* out-edge, where the un-indexed graph declined the shape with a `NotImplementedError`; `NOT EXISTS { (n)-->(n) }` mirrored it by dropping rows that genuinely satisfy the predicate. The adjacency-membership shortcut answered pattern participation from edge-table CSR keys alone, which drops two conditions the scan enforces: a **repeated endpoint alias** (`(n)-->(n)` means a self-loop, not "n has an out-edge"), and the **node-table intersection** (an edge endpoint absent from the node table is not a node, so the edge witnesses nothing). Both are now preconditions of the shortcut — a repeated alias never takes it, and neither does a graph whose node table does not cover every edge endpoint — so an indexed graph and an un-indexed graph return the same rows or decline identically for every EXISTS/NOT EXISTS shape. An index is an optimization and must never change an answer, least of all turn an honest decline into wrong rows. +- **`lazy_import_has_min_dependancy()` returned a 3-tuple on its generic-exception path, so a broken (not missing) scipy/sklearn raised `ValueError: too many values to unpack` instead of the real dependency error.** Every one of its five callers unpacks two values; only the `except ModuleNotFoundError` and success paths returned two. An ABI-mismatched or otherwise broken install therefore surfaced as an unrelated unpack error at `assert_imported()`, hiding the actual ImportError. The generic path now returns the same 2-tuple as the others, and a pin locks the arity of every return path (the sibling 3-tuple probes keep their own shape). +- **`sum()`/`avg()` over `BOOLEAN` adopted as a documented extension, with its return TYPES pinned across engines (#1820)**: `sum`/`avg` over a boolean column is a type error in Cypher (*"expected Float, Integer or Duration but was Boolean"*) and has always been answered here as a deliberate GFQL extension — a strict superset, so no Cypher-valid query changes meaning. The values already agreed on every engine; the **return types did not**. Polars answered `sum(BOOLEAN)` and *every* `count()` with `UInt32` where pandas and cuDF answered `int64`, the all-null substitution answered `sum` with an `Int32` literal, and cuDF answered `count(DISTINCT ...)` with `int32` — same values, four different return types, which is exactly the cross-engine divergence class the aggregate type contract exists to close. Exercising the cuDF arm (previously unverified) also turned up a **wrong value**, not just a wrong type: cuDF's grouped `sum` answered a group with no non-null values with NULL, where Cypher says **0** and pandas/polars already said 0 — on `BOOLEAN`, `Int64` and `float64` alike. The contract is now `sum(BOOLEAN) -> INTEGER(int64)`, `avg(BOOLEAN) -> FLOAT(float64)`, `min`/`max`(BOOLEAN) `-> BOOLEAN`, `count(BOOLEAN) -> INTEGER(int64)`, enforced at all seven aggregate sites across the pandas/cuDF row pipeline, the native polars row pipeline and both OLAP fast paths. `min`/`max` are stated and pinned as the standard boolean **ordering** `false < true` returning null over zero non-null values — NOT as an `AND`/`OR` fold, which agrees on populated input but predicts the conventional empty identities (`true`/`false`) where every engine answers null. `sum -> 0` over zero rows is Cypher conformance (SQL returns NULL), not a compromise. Non-boolean aggregates are unaffected: polars already summed every other numeric input to `Int64`/`Float64`/`Duration`, and `FLOAT`/`DURATION` sums are explicitly excluded from the widening. `count()` is now `INTEGER` on polars for every input type, matching pandas/cuDF and Cypher. (Registered on the #1665 per-engine semantics matrix and the #1664 openCypher conformance tracker.) +- **Scaling a duration raised instead of splitting a month (#1937)**: `duration('P1M') / 2` and `duration('P1M') * 0.5` declined with a `GFQLTypeError` whenever the result was fractional in month-space, even though `duration('P2M') / 2` answered `P1M` and the `duration('P0.5M')` constructor had always split a fractional month. Scaling now cascades the fractional month DOWN into days at the average month of 30.436875 days (365.2425 / 12, the constant the constructor already used) and on into time, so `duration('P1M') / 2` is `P15DT5H14M33S` as openCypher specifies. Every cascade step truncates toward zero rather than rounding, which also corrects the seconds group by a nanosecond (`duration('PT2S') / 3` is `PT0.666666666S`, previously `PT0.666666667S`) and makes a negative scale the exact mirror of its positive twin (`duration('P1M') / -2` and `duration('-P1M') / 2` both give `P-15DT-5H-14M-33S`). A result that is whole in month-space still stays in month-space — `P2M / 2` is `P1M`, `P1Y / 2` is `P6M`, `P1M * 2` is `P2M` — so the average month only appears where a month genuinely has to be split, and scaling is deliberately not round-trippable there: `(duration('P1M') / 2) * 2` is `P30DT10H29M6S`, not `P1M`. +- **`engine='polars-gpu'` ran the Cypher OLAP fast paths on CPU and reported them as GPU (#1824)**: the fast-path call sites pinned the lazy execution target to CPU no matter which engine was requested, and the connected-match-join two-star arms were never wrapped in a target at all, so every fast-path-served OLAP shape collected on CPU polars under a GPU label — undetectable downstream, and exactly the mislabelling `lazy._engine_for`'s `raise_on_fail=True` contract exists to prevent. One shared seam, `_run_fast_path_on_requested_target`, now runs every fast-path arm on the requested engine's target, so an explicit `polars-gpu` collects on the GPU or raises. A plan node cudf-polars cannot execute surfaces as the usual `NotImplementedError` and is treated as a fast-path DECLINE, letting the generic route — itself GPU-or-raise — answer; it is never quietly served on CPU. `engine='polars'`, `engine='auto'`, `pandas` and `cudf` are unchanged (still CPU target, and a `NotImplementedError` there is still a real error rather than a decline). Expect `polars-gpu` to get SLOWER on these shapes: the previous numbers were CPU numbers wearing a GPU label. +- **Remote GFQL/Python error surfacing leaked raw plumbing exceptions and could silently bind the wrong table (#1956)**: a shared `remote_response` helper now backs both `chain_remote` and `python_remote`, so a public remote call raises a typed `GFQLRemoteError` carrying the HTTP status and the server's own message instead of a `requests.exceptions.JSONDecodeError` (an error body with a JSON content-type but a non-JSON payload — `JSONDecodeError` subclasses `ValueError`, so the module's own "re-raise our ValueError" arm swallowed the fallback), a raw `HTTPError` (`python_remote` had no non-JSON branch at all, while `chain_remote` wrapped it), a `KeyError: 'edges'` (a 200 whose JSON body is an error document), or an `IndexError` (a zip missing an expected member). The zip handlers no longer build an informative message inside a `try` that catches and replaces it, so the server's real validation text survives. Most importantly, zip member selection no longer guesses: a member whose stem is exactly `nodes`/`edges` wins, and a looser name match is accepted only when it mentions one kind and not the other, so a prefixed `graph_nodes.parquet` still resolves while a compound `nodes_and_edges.parquet` — which previously bound the EDGE table as nodes with no error, and could be bound as BOTH tables at once when it was the only member — is now a typed decline naming the ambiguity. +- **Remote GFQL request/result plumbing: NaN leak, dropped `output=`, shape variant without `params`, stranded bindings (#1960)**: non-finite filter values (`float('nan')`, `inf`, and NaN inside a predicate) leaked a raw `requests.exceptions.InvalidJSONError` while every other non-JSONable value already got a typed `GFQLTypeError`; the wire body is now scanned before the POST and declined typed, naming the offending path. `output=` was silently dropped for non-Let queries — it names a binding to return, which a flat chain does not have, so it is now declined with a typed `GFQLSyntaxError` rather than ignored (Let/DAG queries are unaffected). `gfql_remote_shape()`/`chain_remote_shape()` now accept `params` and `output`, so a parameterized Cypher query can be shape-queried and not just executed. A `node_col_subset`/`edge_col_subset` that drops a column the returned graph is bound to now raises a typed `GFQLSchemaError` naming the column, instead of returning a Plottable whose `_node`/`_destination` points at a column that no longer exists. +- **`python_remote` rejected the function name its own contract mandates (#1959)**: the callable branch only converted `code` to source when the function was named something OTHER than `task`, so a function literally named `task` — the name `validate_python_str` requires and every doc example uses — fell through as a callable and hit `assert isinstance(code, str)`. Any other name worked. The same path never dedented, so the verbatim docstring example (an indented triple-quoted literal) died in `ast.parse` with `IndentationError`, as did source captured from a nested `def`. Normalization is now one shared `normalize_task_code()` — convert callables to source, rename to `task` only when needed, then `textwrap.dedent` — so name and indentation no longer decide whether the call works. Relative indentation inside the body is preserved, and already-flush source is returned unchanged. +- **Remote calls on unsupported frame types decline before the request (#1957 partial)**: `gfql_remote`/`chain_remote`/`python_remote_*` resolved the DataFrame library only *after* the POST, so a polars-backed graph sent the request, let the server do the work (and, with `persist=True`, create a dataset), and only then died with an untyped `ValueError: Unknown DataFrame types`. `python_remote` also inspected only `_edges`, so a polars node frame paired with pandas edges was never caught at all. Both surfaces now resolve the library once, before any request, via a shared helper, and decline with a typed `GFQLRemoteError` (`E404 remote-unsupported-frames`) naming the actual types (`nodes=polars.DataFrame`) and the remedy. The same resolution is reused at decode, so the pre-request check and the reader selection cannot drift. +- **Remote `format='csv'` no longer silently rewrites result values (#1958)**: `gfql_remote`/`chain_remote`/`python_remote_*` decoded csv responses with a bare `read_csv`, so pandas/cudf re-inferred dtypes from text: `'007'` came back as `7.0`, `'08'` as `8.0`, and `'NA'`/`''`/`'null'` as `NaN`. Worse, node `id` landed as `float64` while the edge endpoints stayed `int64`, so the returned graph could not join to itself and every downstream local op on it was wrong or empty. csv is untyped on the wire -- `format='parquet'` carries an Arrow schema, csv carries none -- so the client cannot reconstruct the server's schema; csv now emits a `UserWarning` naming the risk and pointing at `format='parquet'` (the default, and faithful), then serves the result -- `format='csv'` keeps working without new required arguments. Callers who want fidelity from csv pass the new `df_import_args` reader kwargs to take explicit control, e.g. `df_import_args={'dtype': {'id': str}, 'keep_default_na': False, 'na_values': []}`, which round-trips values and keeps the node/edge join coherent. (The warning predicate this entry described -- any dict silences it -- was corrected below.) A malformed (non-dict) `df_import_args` is rejected before the request is sent with a typed `GFQLRemoteError` (`E403 remote-format-lossy`, which also subclasses `ValueError`), so a caller typo never costs a round trip or an implicit upload. `format='parquet'` and `format='json'` are unchanged. +- **`group_in_a_box_layout()` returned duplicate node ids at conflicting coordinates (#1961)**: the default `bulk_mode=True` path lays out the WHOLE graph in one pass, but `partitioned_layout` then re-appended the singleton (`id_count == 1`), pair (`id_count == 2`) and edgeless (`degree_max == 0`) partitions on top of that already-positioned frame instead of replacing those rows. Every node in such a partition came back TWICE, each copy carrying a different `x`/`y`, so the returned node frame had more rows than it was given, plotted the same node at two positions, and fanned out any downstream node-keyed join; on a fully edgeless graph every node was duplicated. Those small-partition fallbacks exist for `bulk_mode=False` (where `layout_non_bulk_mode` only positions partitions with `id_count > 2 and degree_max > 0`) and are now scoped to it; the bulk pass already positions those nodes, and does so at least as well — pairs now spread across their box instead of clustering at `0`/`0.33`. Two adjacent defects in the same function are fixed alongside: the edgeless branch assigned `x` twice and never assigned `y`, and the NaN backstop asserted the negation of its own guard (`assert combined_nodes.y.isna().sum() == 0` inside `if combined_nodes.y.isna().any()`), so an unpositioned node raised a message-less `AssertionError` instead of reaching the `fillna` two lines below; the cuDF half of that backstop also built a 2-D `cupy` array that `cudf.Series` rejects. Pinned by an id-multiset invariant (output ids == input ids, no duplicates, no drops) across mixed partition sizes on both pandas and cuDF. Unseeded RNG for edgeless placement is unchanged and still undocumented. +- **GFQL serialization no longer drops `None` filter values, turning "match nothing" into "match everything" (#1954)**: `_filter_dict_to_json` skipped every entry whose value was `None`, so `n({'x': None})` serialized to `{"filter_dict": {}}` — an unconstrained match. Blast radius was every `to_json()` consumer, not just the remote path: `gfql_remote`/`chain_remote` put `filter_dict: {}` on the wire and asked the server for the entire graph while the identical in-process query returned zero rows; `Chain.to_json()`/`Chain.from_json()` round trips, saved query JSON, and any stored wire form silently widened the same way. The whole family was affected — node `filter_dict`, `edge_match`, `source_node_match`, `destination_node_match` — including Cypher patterns whose value came from a null parameter (`MATCH (a {x: $p})` with `params={'p': None}`). `None` values are now serialized as JSON `null`; `from_json` already revived them unchanged, so local and wire answers now agree. No query that previously answered correctly changes: the only affected inputs are ones whose serialized form did not mean what the caller wrote. +- **GFQL `min_hops` prune dropped qualifying branches that end below `max_hops` (#1944)**: `min_hops`/`max_hops` are documented as INCLUSIVE traversal bounds and the prune removes only "dead-end branches that do not reach `min_hops`", but the backward retention walk seeded its targets from the TOP hop level only, then narrowed them level by level. A branch that reached `min_hops` and then TERMINATED — strictly below `max_hops` — was therefore never a target when its own level was processed, so its terminating edge and everything feeding it exclusively were silently pruned. An edge traversed at a level `>= min_hops` now ends a qualifying walk ITSELF and is retained outright; only sub-`min_hops` levels still have to feed a retained longer walk. Landed PAIRED across engines because the polars chain mirror reproduced the identical under-retention: on `hop(min_hops=2, max_hops=3, direction='forward')` from a seed whose branch ends at hop 2, pandas and cuDF dropped both tail edges while LEAKING the tail's endpoint node (an incoherent node/edge frame) and the polars chain dropped the endpoint too; all four surfaces now return the hand oracle. Reverting either engine's half alone turns 31 cross-engine chain/hop parity cells red, which is what pins the pairing. Genuine sub-`min_hops` dead ends are still pruned (anti-vacuity control at `min_hops == max_hops`). +- **`gfql_validate` now agrees with execution on unqueryable graph shapes (#1889)**: the validator advertised itself as preflight ("validate without executing") yet returned `{ok: True, diagnostics: []}` for graphs it could not possibly answer — a graph with neither nodes nor edges bound then died at execution with a bare `ValueError: Missing edges` on pandas/cuDF or an empty-message `AssertionError` on polars, and an edge pattern against a graph with no edges bound was declined by the executor (`E304`) while the validator stayed silent. Validator and executors now consult one shared predicate, so both surfaces return the same typed verdict for the same shape: a new `ErrorCode.E305 graph-not-bound` when neither frame is bound, and the existing `E304` when an edge operation meets an unbound edge frame. This is a **behavior change for callers who treated `ok: True` as unconditional**: shapes that previously validated clean and then crashed now report a `GFQLSchemaError` up front, on both the chain and Cypher entry points and on every engine. Nothing that previously executed is refused — queries that answered still answer with identical values; only bare crashes became typed diagnostics. `schema=False` skips the check, so `chain_remote` preflight (whose frames live server-side) is unaffected. +- **GFQL Cypher temporal/error-leak family (#1915 B-5/B-7/B-8 + A-4, #1880 temporal half)**: (B-5) literal temporal comparisons are constant-folded engine-agnostically to the openCypher CIP2016-06-14 semantics — zoned instants compare on the UTC global timeline, so `datetime('2020-01-02T05:00:00+05:00') = datetime('2020-01-02T00:00:00Z')` is now `true` on every engine (polars compared rendered text and answered `false`, row-set-visibly in WHERE), while values of DIFFERENT temporal types are never equal and order as null — `datetime(...) = localdatetime(...)` was `true` on BOTH engines and is now `false`, with `<` etc. null. (B-7/#1880) temporal-vs-string comparisons no longer leak raw backend errors: polars filter predicates and scalar equality over Datetime/Duration/Date/Time columns parse the string with the SAME pandas parse the pandas engine applies (`pd.Timestamp`/`pd.to_timedelta`) and answer with pandas-identical rows in that SAFE subset, and otherwise raise the scalar half's typed `GFQLSchemaError` E302 instead of `polars.exceptions.InvalidOperationError`; tz-suffixed ISO temporal text comparisons stay a `where_rows` residual instead of a filter-dict pushdown, so `n.ts > '2021-01-01T00:00:00Z'` answers on pandas/cuDF instead of raising a raw numpy `bitwise_and` TypeError (and its `=` no longer silently matches zero rows); a tz-naive vs tz-aware datetime column pair in a same-path WHERE aligns onto UTC-naive (GFQL reads naive as UTC, matching the row pipeline) instead of raising a raw pandas TypeError, and the forward bounds prune skips incomparable dtypes rather than crashing. (B-8) non-reserved keywords are valid property names — `n.when`, `n.then`, `n.end`, `n.order`, `n.is`, `n.all`, `n.any`, `n.contains`, and `{when: 1}` property maps — via a dot/map-key-context `PROP_NAME` terminal in both grammars; keywords stay reserved everywhere else, and the WHERE-chain grammar keeps the filter-dict pushdown for them. (A-4) UNION branches projecting the same output names in a different order now align by name (Neo4j semantics; the output keeps the first branch's column order) on all three engines, and only a genuinely different name multiset keeps the typed decline. Pinned red-at-master across pandas/polars/cuDF with cross-type fold matrices, parity row sets, typed-decline assertions by name, and mutation-killing guards. +- **The remote csv warning was defeated by any dict, and silenced outright on GPU; master shipped a RED cuDF pin (#1958 follow-up)**: three linked defects behind the warn-and-serve `format='csv'` contract. (1) The warning fired only when `df_import_args` was `None`, so `df_import_args={}` or `{'sep': ','}` bought ZERO fidelity -- measured, both still decode `['007', '08', 'NA']` as `[7.0, 8.0, nan]` in `float64` -- while removing the only signal that the frame may be corrupt. The predicate now tracks whether the caller actually took control, per lossy axis, and the two axes are independent: `dtype=str` alone still maps `'NA'`/`''`/`'null'` to `NaN`, and `keep_default_na=False, na_values=[]` alone still reads `'007'` as `7`. dtype inference counts as governed by `dtype` or `converters`; NA substitution by `keep_default_na`, `na_values`, `na_filter` or `converters`; the warning names each axis left ungoverned and clears only when both are, so a partially-controlled read is told which half is still lossy. (2) On a cuDF graph the warning never reached anyone: `lazy_cudf_import()` called `warnings.filterwarnings('ignore')` outside any `catch_warnings`, so the FIRST engine resolution against a GPU frame installed a process-global ignore-everything filter -- silencing this warning and every other library's for the rest of the session. Probing for cudf (and cuml, which leaked a redundant unscoped filter alongside an already-scoped block) now restores the caller's filters. (3) The cuDF pin in `test_remote_csv_fidelity.py` still asserted the pre-#1974 hard decline (`pytest.raises(ValueError)`, `assert not mock_post.called`), so `TEST_CUDF=1` was RED on master; it now pins the shipped contract -- warns, issues the request, returns the rows. No refusal is re-introduced: `format='csv'` still serves, and `format='parquet'` still needs no reader args. +- **cuDF test gates are audited and the missing GPU lane is stated, not implied**: no CI lane runs a cuDF arm -- `ci.yml` never sets `TEST_CUDF` and installs no `cudf`, and `ci-gpu.yml` is gated on an unset `GRAPHISTRY_ENABLE_GPU_PUBLIC` variable, needs the `gpu_public` self-hosted runner, and hard-fails any manual trigger -- which is how a cuDF pin asserting a removed contract stayed red on master. A GPU lane cannot be wired from this repo (GitHub-hosted runners have no NVIDIA device, and cudf has no CPU fallback), so the gap is made loud instead: `bin/ci_gpu_gate_audit.py` (new `gpu-gate-audit` lane) counts the cuDF gates (37 across 23 files today), requires each to carry a `reason=` naming `TEST_CUDF` so `pytest -rs` on a CPU lane names what was not run, requires each to actually read the flag from the environment, and cross-checks the DEVELOP.md unprotected-receipts note against whether any workflow sets `TEST_CUDF` -- wiring a real GPU lane retires the note, deleting the note without wiring a lane fails the audit. The audit is static: it proves the gates are well formed, never that the gated assertions hold. DEVELOP.md now says plainly that a `TEST_CUDF=1` receipt is developer-local evidence only. ### Added - **Precomputed in/out degree facts (`DegreeFact`)**: the two-hop `count(*)` kernel spends O(E) per query on a `bincount` plus gather over every edge. With degrees precomputed the identical answer is `dot(indeg, outdeg)` — O(N). Measured at board scale (2.4M edges / 107k nodes): 6.76 ms of query work becomes 0.046 ms, a complexity-class change rather than a constant factor, built once per relationship type. Keyed by `(src, dst, type_column, type_value)` on the existing registry shape, because a typed pattern counts over one relationship type and a global degree array would be the wrong denominator for it. Declines by explicit precondition — absent column, non-integer or null-bearing endpoints, an endpoint outside the proved interval, or an oversized span — and an out-of-interval endpoint DECLINES rather than clamping, since a clamp would silently miscount. Note this is the first fact kind where staleness is a WRONG ANSWER rather than a lost optimization, so the identity+fingerprint guard is the correctness guard here and refuses on any mismatch. Pins: dot-vs-gather-sum equivalence on hand and random graphs, every decline, rebind/engine invalidation, and typed-vs-global key separation. - **Fast-path engagement is visible in `gfql_explain`**: each fast path now records whether it SERVED or declined, with the engine, under `op: fast_path`. Fast paths are contracted "same answer, faster" — every one falls back — so a dead one is otherwise invisible: the query still returns the right result and every value test still passes. `assert_fast_path(g, query, path, served=)` makes engagement assertable against a public surface rather than by monkeypatching private callees, which fails open when another module imported the name directly. Covers the single-hop grouped aggregate, the two-hop count and the seeded typed hop; free outside `index_trace()`. Pinned across pandas/polars/cuDF, including that a short-circuit is distinguishable from a decline (a path never consulted is absent, not False) and that the helper fails when the path did not fire. @@ -112,8 +145,11 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Infrastructure - **The polars CI lane runs in parallel, so the coverage cell stops racing its timeout**: `test-polars (3.12)` is the only cell that runs the coverage-instrumented pass *and* the per-file coverage audit, and it had already been cancelled twice at its 10-minute budget (615s, 611s) with zero test failures — pure budget exhaustion, with the same commit passing or failing depending on the runner it drew. `bin/test-polars.sh` now runs its main phase under `pytest-xdist` (`-n auto --maxprocesses 4 --dist load`); `pytest-xdist` was already in the `[test]` extra and `test-gfql-core` already runs `-n auto` under `--cov`, so no dependency changes. `auto` resolves to the runner's core count (4 on a GitHub-hosted `ubuntu-latest`) and scales *down* on a 2-vCPU runner where a fixed `-n 4` could be slower than serial; `--maxprocesses` keeps a many-core dev box from fanning out one polars process per core. Verified rather than assumed, on py3.12 / polars 1.43.1 / pandas 3.0.5 with `POLARS_COV=1`: the pass/skip **node-id sets** (not just counts) are identical serial vs parallel across 4-worker, 2-worker and `--dist loadfile` runs (2417 node ids, 2404 passed / 13 skipped every time); the merged coverage data is a strict superset of the serial data (28,480 vs 28,478 covered lines, zero lines lost, zero files dropped); the second `--cov-append` invocation still appends into the xdist-produced data file (+1,791 lines, nothing lost); and `bin/coverage_audit.py --profile gfql-polars` emits a byte-identical report (modulo its timestamp) from the parallel data, so the downstream `changed-line-coverage` check sees no change. Measured **on CI**, by this change's own run: the `test-polars (3.12)` cell goes **501s → 322s**, its script step **484s → 297s (1.63×)**, and the five non-coverage cells go 222–318s → 115–172s. (A local 4-CPU-pinned A/B with coverage on showed 326s → 109s, 2.98×; CI realizes less, because coverage tracing is per-worker CPU cost that does not parallelize away and the ~8s serial second phase plus 4× interpreter startup are fixed. The CI number is the one to quote.) `--dist load` rather than `loadfile` because one module (`test_engine_polars_chain.py`) is 69% of the lane's test time, capping file-level distribution at ~1.4×; `POLARS_XDIST=0` restores the serial path and `POLARS_XDIST_DIST` selects another distribution mode should a future order-dependent test need one. +- **The Polars coverage cell has deterministic runner-budget margin**: a later `ubuntu-latest` runner exposed only 2 CPUs, so xdist correctly created 2 workers and the main coverage pass took 566.14 s; the 13.89 s append pass and successful coverage audit then reached the 10-minute whole-job limit while the report was being printed. The job budget is now 15 minutes. This changes no test selection, parallelism, coverage floor, or runtime code; it prevents a fully passing audit from being reported as cancelled on the smaller runner shape. ### Documentation +- **Removed documentation that asserted behavior the code does not have.** A sweep re-measured every claim before rewriting it. (1) *"`auto` never selects Polars or Polars-GPU"* was false in five shipped places (`Engine.py`, `gfql/engines.rst` x3, `gfql/overview.rst`, plus a `gfql_unified.py` docstring): `resolve_engine(AUTO, )` returns `Engine.POLARS`, and `gfql` under AUTO additionally routes an all-cuDF graph to `polars-gpu` by re-entering with an explicit engine. The narrow fact — `resolve_engine` itself never returns `POLARS_GPU` for AUTO — is true and is now stated as such, separately from the routing that does reach GPU. The docs now describe the real rule: AUTO follows the input frames and falls back to pandas/cuDF only for shapes the native engine declines, while an explicit engine turns those declines into errors. The stale `gfql_index_all` caveat went with it — an AUTO build now keeps Polars frames native. (2) `spec/cypher_mapping.md` claimed the aggregate contract was pinned *"identically on pandas, polars, cuDF and polars-gpu"*; the suite is parametrized over all four but the polars-gpu arm skips for want of `cudf_polars`, so that arm is now described as specified-but-not-yet-verified. (3) Fifteen doc-vs-signature contradictions were corrected against the actual signatures, including `collapse` `unwrap` (documented True, actually False), three defaults in one `embed()` docstring (`use_feat`, `lr`, `evaluate`), `featurize` `min_samples` (5 -> 1, which means nothing is ever DBSCAN noise) and `strategy` (quantile -> uniform), `search` `top_n` (100 -> 10), ten `for_current` params documented "Default on" against a `False` signature, `hypergraph` `engine` (pandas -> auto), a `by=` kwarg on `fa2_layout` that does not exist (the real one is `circle_layout_params={'sort_by': ...}`), the polars `filter_expr_by_dict_polars` "schema only" claim (frame *height* gates its typed-error block), a `hop_eager` labeling comment contradicted by the code 60 lines below it, `partitioned_layout` `partition_key` (defaults to the literal `'partition'`, not the layout alg) and its CPU default (`fa2_layout` under the default `bulk_mode=True`, not igraph `fr`), and a `token_refresh_ms` parameter `register()` does not accept. +- **`circle_layout`'s `sort_by` / `ascending` / `na_position` / `ignore_index` are documented as having no effect, because they do not.** The sort they perform is discarded by an unconditional re-sort by node id before any angle is assigned, so ring order is always by node id. Rather than silently change everyone's layout coordinates in a documentation pass, the parameters are now described plainly as inert (including in `gfql/builtin_calls.rst`, which exposes them through the Call API) and a characterization test locks that, so a future change making `sort_by` real must update the docs in the same commit. The two residual effects are documented too: `sort_by=None` still attaches degree columns, and an unknown column still raises. - **`gfql/performance.rst` publishes the receipted q1–q9 board, and the prose around it matches the receipts.** The July matched lane (`results/graphbench-matched-q1q9-20260726` in graphistry/pyg-bench) carries no receipts and is withdrawn as nonreproducible; nothing cites it any more — including the restored q9 66.6 ms / 84.1 ms cell, which came from that lane. In its place, the receipted 2026-08-02 20k lane (`results/graphbench-board-20k-20260802`: 4 position-balanced slots × 51 timed runs after 5 warmups, per-second host-load and spike receipts, runner-script checksums, row-set-validated cells) is published as a full nine-row board: **5 wins, 2 ties, 2 losses** for GFQL-Polars against embedded Kuzu, with the two weak verdicts (q4 and q5, overlapping slot ranges) flagged in both the page and the published data, and the two losses printed with the same weight as the wins. The receipted 100k lane (`results/graphbench-board-100k-v2-20260802`, same protocol and receipts, 263 receipted spike seconds) is published the same way: **6 wins, 2 ties, 1 loss** — q8 loses at both scales (the comparator's verdicts: LOSE 2.94x at 20k, LOSE 2.44x at 100k) and q4 flips from a weak 20k loss to a clean 100k win. Numbers appear only as `:bench:` references resolved from `docs/source/_data/gfql_benchmarks.json`, with a single provenance block for the run. `engines.rst`'s "both sizes measured" and "gap grows with size" claims now cite the two receipted boards, and the vendor rows stay qualitative with links to the board. - **Benchmark numbers that could not be traced to a surviving measurement have been REMOVED rather than restated.** A provenance audit of every published performance figure found most of them can be neither confirmed nor refuted: the reproducers wrote their results to `/tmp` (or only printed them), recorded no commit, host or timestamp, and their raw artifacts were never committed and no longer exist. Withdrawn on those grounds: the four-engine Orkut/LiveJournal bulk table and its CPU-crossover ratios (`gfql/performance.rst`, `gfql/engines.rst`), the 0.58.0 tag-sweep tables (`gfql/performance.rst`, `gfql/indexing.rst`), the seeded-index synthetic and vs-Kuzu/Neo4j tables and the prepared-Kuzu figures (`gfql/index_adjacency.rst`), the LDBC-vs-Neo4j table (every row that maps onto a query id the current lane emits differs materially from the current board on both sides, and `recent-replies` maps onto no current query id at all), the LadybugDB head-to-head (whose competitor column came from a hardcoded `LADYBUG = {...}` literal in `benchmarks/gfql/bench_ladybug_cypher.py` with no URL, version or citation — a literal, not a measurement), the filter→PageRank-vs-Neo4j results and their two committed SVG charts (the chart generator reads `plans/gfql-gpu-pagerank-benchmark/results/`, a path that has never existed in any commit, so the charts cannot be regenerated by anyone), and every downstream echo of those figures (`~38x`, `9-28x vs Kuzu/Neo4j`, `43X+`, `10-50x`, `100X+`, `10X+`). `gfql/benchmark_graphframes.rst` is **kept intact** — its raw results are committed at `docs/source/gfql/_static/graphframes/results.json` and every headline cell round-trips against it. Removing an unverifiable number is the correct outcome, not a regression — but it is a large removal, so it is called out here rather than buried. - **Restored the charts and the narrative on `gfql/benchmark_filter_pagerank.rst`, and stopped the provenance blocks interrupting it.** `b467bfc36` did the right thing about the numbers and the wrong thing about the page: to stop republishing refuted figures it deleted `twitter_lifecycle.svg` and `gplus_lifecycle.svg`, the two per-graph narrative sections that embedded them and the "Why this matters" section, and dropped `.. bench-provenance::` / `.. bench-disclosures::` into the middle of the page, leaving a benchmark page with no benchmark visual and a compliance notice above the fold. The charts are back, **regenerated from the current published cells rather than restored** — the deleted SVGs asserted Twitter 13.83 s / 2.55 s / 0.30 s and GPlus 75.78 s / 3.33 s as glyph paths, none of which the artifact publishes any more. `docs/source/_ext/gfql_bench_charts.py` renders them from `docs/source/_data/gfql_benchmarks.json` through the same `format_cell` the `:bench:` role uses, so the table, the prose and the bars cannot disagree; it is stdlib-only and byte-reproducible, and `docs/test_bench_numbers.py` re-renders the SVGs on every test run and fails on drift, which is the check the old glyph-path charts could not have. The per-graph sections, "Why this matters", and the environment section are back; provenance and disclosures now sit at the bottom under "Benchmark environment and provenance", still mandatory, no longer above the fold. diff --git a/DEVELOP.md b/DEVELOP.md index b6d6faee77..0c147ad62b 100644 --- a/DEVELOP.md +++ b/DEVELOP.md @@ -248,12 +248,31 @@ Ruff additionally rejects `getattr(x, "const")` / `setattr(x, "const", v)` ### GPU CI -GPU CI can be manually triggered by core dev team members: +**Today, no CI lane executes cuDF.** `ci.yml` never sets `TEST_CUDF` and no lane +installs `cudf`, and `ci-gpu.yml` is disabled: its jobs are gated on the +`GRAPHISTRY_ENABLE_GPU_PUBLIC` repository variable (unset), it needs the +`gpu_public` self-hosted runner, and a `gpu-disabled-guard` job hard-fails any +manual trigger. So a `TEST_CUDF=1` receipt is **developer-local evidence only** -- +a cuDF-gated test can contradict the CPU contract, or rot outright, and stay green +on master indefinitely. Treat a GPU claim in a PR as unprotected until a GPU lane +exists: re-run it yourself rather than trusting the last receipt. + +`bin/ci_gpu_gate_audit.py` (lane `gpu-gate-audit`) keeps the size of that gap +visible: it counts the cuDF gates, requires each to be attributable (a `reason=` +naming `TEST_CUDF`, so `pytest -rs` names what was not run rather than reporting a +bare `s`) and to actually read the flag from the environment, and cross-checks this note against +whether any workflow sets `TEST_CUDF`. Wiring a real GPU lane retires the note; +deleting the note without wiring a lane fails the audit. The audit is static -- it +proves the gates are well formed, never that the gated assertions hold. + +GPU CI can be manually triggered by core dev team members, once the lane is +re-enabled: 1. Push intended changes to protected branches `gpu-public` or `master` 2. Manually trigger action [ci-gpu](https://github.com/graphistry/pygraphistry/actions/workflows/ci-gpu.yml) on one of the above branches -GPU tests can also be run locally via `./docker/test-gpu-local.sh` . +GPU tests can also be run locally via `./docker/test-gpu-local.sh` , or directly +with `TEST_CUDF=1 pytest ...` on a RAPIDS-equipped box. ## Debugging Tips diff --git a/bin/ci_comment_density_baseline.json b/bin/ci_comment_density_baseline.json index 88f6e8e868..3bdd5120a7 100644 --- a/bin/ci_comment_density_baseline.json +++ b/bin/ci_comment_density_baseline.json @@ -29,7 +29,7 @@ "graphistry/compute/gfql/cypher/_boolean_expr_text.py": 1, "graphistry/compute/gfql/cypher/ast.py": 5, "graphistry/compute/gfql/cypher/ast_normalizer.py": 1, - "graphistry/compute/gfql/cypher/lowering.py": 49, + "graphistry/compute/gfql/cypher/lowering.py": 48, "graphistry/compute/gfql/cypher/parser.py": 24, "graphistry/compute/gfql/cypher/reentry/compiletime.py": 2, "graphistry/compute/gfql/cypher/reentry/execution.py": 5, @@ -49,7 +49,7 @@ "graphistry/compute/gfql/index/explain.py": 1, "graphistry/compute/gfql/index/lookup.py": 1, "graphistry/compute/gfql/index/registry.py": 4, - "graphistry/compute/gfql/index/traverse.py": 15, + "graphistry/compute/gfql/index/traverse.py": 13, "graphistry/compute/gfql/index/types.py": 3, "graphistry/compute/gfql/index/wire.py": 2, "graphistry/compute/gfql/ir/pushdown_safety.py": 5, @@ -59,14 +59,13 @@ "graphistry/compute/gfql/lazy/engine/polars/chain.py": 47, "graphistry/compute/gfql/lazy/engine/polars/degrees.py": 3, "graphistry/compute/gfql/lazy/engine/polars/dtypes.py": 3, - "graphistry/compute/gfql/lazy/engine/polars/hop.py": 2, - "graphistry/compute/gfql/lazy/engine/polars/hop_eager.py": 19, + "graphistry/compute/gfql/lazy/engine/polars/hop_eager.py": 18, "graphistry/compute/gfql/lazy/engine/polars/lowering_context.py": 2, - "graphistry/compute/gfql/lazy/engine/polars/nan_clean.py": 2, + "graphistry/compute/gfql/lazy/engine/polars/nan_clean.py": 1, "graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py": 9, "graphistry/compute/gfql/lazy/engine/polars/predicates.py": 18, "graphistry/compute/gfql/lazy/engine/polars/projection.py": 5, - "graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py": 84, + "graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py": 82, "graphistry/compute/gfql/lazy/engine/polars/search.py": 3, "graphistry/compute/gfql/lazy/engine/polars/varlen_rows.py": 1, "graphistry/compute/gfql/logical_planner.py": 1, @@ -77,7 +76,7 @@ "graphistry/compute/gfql/row/entity_props.py": 1, "graphistry/compute/gfql/row/frame_ops.py": 6, "graphistry/compute/gfql/row/ordering.py": 1, - "graphistry/compute/gfql/row/pipeline.py": 40, + "graphistry/compute/gfql/row/pipeline.py": 38, "graphistry/compute/gfql/same_path/multihop.py": 1, "graphistry/compute/gfql/same_path/native_shortest_path.py": 3, "graphistry/compute/gfql/search_any.py": 4, @@ -89,7 +88,7 @@ "graphistry/compute/gfql_unified.py": 21, "graphistry/compute/gfql_validate.py": 3, "graphistry/compute/graph_operation.py": 1, - "graphistry/compute/hop.py": 6, + "graphistry/compute/hop.py": 4, "graphistry/compute/predicates/comparison.py": 9, "graphistry/compute/predicates/from_json.py": 1, "graphistry/compute/predicates/is_in.py": 4, @@ -145,7 +144,7 @@ "graphistry/ArrowFileUploader.py": 3, "graphistry/compute/ComputeMixin.py": 3, "graphistry/compute/ast.py": 1, - "graphistry/compute/chain.py": 7, + "graphistry/compute/chain.py": 6, "graphistry/compute/chain_fast_paths.py": 4, "graphistry/compute/chain_lean_combine.py": 2, "graphistry/compute/gfql/agg_types.py": 1, @@ -162,12 +161,8 @@ "graphistry/compute/gfql/index/explain.py": 1, "graphistry/compute/gfql/index/lookup.py": 3, "graphistry/compute/gfql/index/registry.py": 5, - "graphistry/compute/gfql/index/traverse.py": 9, "graphistry/compute/gfql/lazy/__init__.py": 7, - "graphistry/compute/gfql/lazy/engine/polars/chain.py": 7, "graphistry/compute/gfql/lazy/engine/polars/degrees.py": 1, - "graphistry/compute/gfql/lazy/engine/polars/hop_eager.py": 2, - "graphistry/compute/gfql/lazy/engine/polars/nan_clean.py": 2, "graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py": 1, "graphistry/compute/gfql/lazy/engine/polars/predicates.py": 1, "graphistry/compute/gfql/lazy/engine/polars/projection.py": 1, @@ -177,9 +172,8 @@ "graphistry/compute/gfql/row/pipeline.py": 8, "graphistry/compute/gfql/same_path/native_shortest_path.py": 1, "graphistry/compute/gfql/temporal/constructors.py": 1, - "graphistry/compute/gfql_fast_paths.py": 17, + "graphistry/compute/gfql_fast_paths.py": 1, "graphistry/compute/gfql_unified.py": 1, - "graphistry/compute/hop.py": 4, "graphistry/feature_utils.py": 1, "graphistry/layout/gib/gib.py": 1, "graphistry/layout/gib/partitioned_layout.py": 1, @@ -242,7 +236,7 @@ "graphistry/compute/gfql/cypher/_boolean_expr_text.py": 1, "graphistry/compute/gfql/cypher/ast.py": 3, "graphistry/compute/gfql/cypher/ast_normalizer.py": 1, - "graphistry/compute/gfql/cypher/lowering.py": 10, + "graphistry/compute/gfql/cypher/lowering.py": 8, "graphistry/compute/gfql/cypher/parser.py": 7, "graphistry/compute/gfql/cypher/reentry/execution.py": 2, "graphistry/compute/gfql/cypher/reentry/flatten.py": 1, @@ -257,11 +251,10 @@ "graphistry/compute/gfql/lazy/engine/polars/chain.py": 10, "graphistry/compute/gfql/lazy/engine/polars/degrees.py": 1, "graphistry/compute/gfql/lazy/engine/polars/hop.py": 1, - "graphistry/compute/gfql/lazy/engine/polars/hop_eager.py": 4, "graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py": 1, "graphistry/compute/gfql/lazy/engine/polars/projection.py": 6, "graphistry/compute/gfql/lazy/engine/polars/reserved_columns.py": 1, - "graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py": 10, + "graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py": 6, "graphistry/compute/gfql/lazy/engine/polars/varlen_rows.py": 1, "graphistry/compute/gfql/passes/predicate_pushdown.py": 1, "graphistry/compute/gfql/rollout.py": 1, diff --git a/bin/ci_cypher_surface_guard_baseline.json b/bin/ci_cypher_surface_guard_baseline.json index eb10cae5d1..d2cfba4088 100644 --- a/bin/ci_cypher_surface_guard_baseline.json +++ b/bin/ci_cypher_surface_guard_baseline.json @@ -13,5 +13,5 @@ "max_properties": 0 } }, - "lowering_py_max_lines": 9861 + "lowering_py_max_lines": 9895 } diff --git a/bin/ci_gpu_gate_audit.py b/bin/ci_gpu_gate_audit.py new file mode 100755 index 0000000000..a02166861c --- /dev/null +++ b/bin/ci_gpu_gate_audit.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +"""Audit the cuDF test gates and keep the size of the unprotected surface visible. + +No CI lane installs cudf or sets ``TEST_CUDF``, so every cuDF-gated test is +developer-local evidence only. This guard makes that gap loud rather than silent: + +1. every cuDF gate must be attributable -- a ``reason=`` naming ``TEST_CUDF``, so + ``pytest -rs`` names what was not run rather than reporting a bare ``s``; +2. every cuDF gate must read the flag from the environment, so a gate cannot + quietly become a constant; +3. ``DEVELOP.md`` must carry the unprotected-receipts note exactly while no + workflow sets ``TEST_CUDF`` -- wiring a real GPU lane retires the note, and + deleting the note without wiring a lane fails. + +The audit is static: it proves the gates are well formed and counts them. It does +not and cannot prove the gated assertions are true; only a GPU lane does that. +""" +import ast +import os +import sys +from pathlib import Path +from typing import List, Optional, Tuple + +REPO = Path(__file__).resolve().parent.parent +TESTS = REPO / "graphistry" / "tests" +WORKFLOWS = REPO / ".github" / "workflows" +DEVELOP = REPO / "DEVELOP.md" + +FLAG = "TEST_CUDF" +UNPROTECTED_NOTE = "no CI lane executes cuDF" + + +class Gate: + def __init__(self, path: Path, lineno: int, source: str, reason: Optional[str]) -> None: + self.path = path + self.lineno = lineno + self.source = source + self.reason = reason + + def where(self) -> str: + return f"{self.path.relative_to(REPO)}:{self.lineno}" + + +def _reason_of(call: ast.Call) -> Optional[str]: + for kw in call.keywords: + if kw.arg == "reason" and isinstance(kw.value, ast.Constant) and isinstance(kw.value.value, str): + return kw.value.value + return None + + +def _skip_message_of(call: ast.Call) -> Optional[str]: + for arg in call.args: + if isinstance(arg, ast.Constant) and isinstance(arg.value, str): + return arg.value + return _reason_of(call) + + +def _callee_name(call: ast.Call) -> str: + node = call.func + parts: List[str] = [] + while isinstance(node, ast.Attribute): + parts.append(node.attr) + node = node.value + if isinstance(node, ast.Name): + parts.append(node.id) + return ".".join(reversed(parts)) + + +def _gate_context(text: str, call: ast.Call, parents) -> str: + """Source that decides the gate: the call, widened to the ``if`` that guards a bare skip.""" + segment = ast.get_source_segment(text, call) or "" + node = call + while node in parents: + node = parents[node] + if isinstance(node, ast.If): + return (ast.get_source_segment(text, node.test) or "") + "\n" + segment + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, ast.Module)): + break + return segment + + +def collect_gates(root: Path) -> Tuple[List[Gate], List[str]]: + gates: List[Gate] = [] + parse_errors: List[str] = [] + for path in sorted(root.rglob("*.py")): + text = path.read_text(encoding="utf-8") + if FLAG not in text: + continue + try: + tree = ast.parse(text, filename=str(path)) + except SyntaxError as e: + parse_errors.append(f"{path.relative_to(REPO)}: {e}") + continue + parents = {} + for parent in ast.walk(tree): + for child in ast.iter_child_nodes(parent): + parents[child] = parent + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + name = _callee_name(node) + if not (name.endswith("skipif") or name.endswith("skip")): + continue + segment = ast.get_source_segment(text, node) or "" + if FLAG not in segment: + continue + reason = _reason_of(node) if name.endswith("skipif") else _skip_message_of(node) + gates.append(Gate(path, node.lineno, _gate_context(text, node, parents), reason)) + return gates, parse_errors + + +def workflows_setting_flag(root: Path) -> List[str]: + if not root.is_dir(): + return [] + assignments = (f"{FLAG}:", f"{FLAG}=") + return sorted( + p.name for p in root.glob("*.yml") + if any(a in p.read_text(encoding="utf-8") for a in assignments) + ) + + +def emit(line: str) -> None: + print(line) + summary = os.environ.get("GITHUB_STEP_SUMMARY") + if summary: + with open(summary, "a", encoding="utf-8") as fh: + fh.write(line + "\n") + + +def main() -> int: + gates, parse_errors = collect_gates(TESTS) + wired = workflows_setting_flag(WORKFLOWS) + develop = DEVELOP.read_text(encoding="utf-8") + failures: List[str] = [] + + failures.extend(f"test file does not parse: {e}" for e in parse_errors) + + if not gates: + failures.append( + f"found zero {FLAG} gates under {TESTS.relative_to(REPO)}; the gating convention moved " + "and this audit is now blind -- update it" + ) + + for gate in gates: + if not gate.reason or FLAG not in gate.reason: + failures.append( + f"{gate.where()}: cuDF gate has no reason naming {FLAG}, so `pytest -rs` cannot " + f"attribute the skip: {gate.source.splitlines()[0]}" + ) + if "environ" not in gate.source and "getenv" not in gate.source: + failures.append( + f"{gate.where()}: cuDF gate does not read {FLAG} from the environment, so it " + f"cannot be turned on: {gate.source.splitlines()[0]}" + ) + + note_present = UNPROTECTED_NOTE in develop + if wired and note_present: + failures.append( + f"workflow(s) {wired} now set {FLAG}; remove the '{UNPROTECTED_NOTE}' note from DEVELOP.md" + ) + if not wired and not note_present: + failures.append( + f"no workflow sets {FLAG}, so cuDF receipts are unprotected; DEVELOP.md must say " + f"'{UNPROTECTED_NOTE}'" + ) + + files = sorted({str(g.path.relative_to(REPO)) for g in gates}) + emit(f"cuDF gates: {len(gates)} across {len(files)} test files") + emit(f"workflows setting {FLAG}: {wired or 'NONE -- these gates are never executed by CI'}") + + if failures: + for f in failures: + print(f"ERROR: {f}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bin/test-polars.sh b/bin/test-polars.sh index ec9e47d74f..45dbe68761 100755 --- a/bin/test-polars.sh +++ b/bin/test-polars.sh @@ -41,6 +41,10 @@ POLARS_TEST_FILES=( # native polars aggregate guard and the raw-polars-exception wrap are exercised graphistry/tests/compute/gfql/test_aggregate_type_contract.py graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py + # #1985 size()/quantifier/comprehension declines: every case is parametrized pandas AND + # polars, and the polars params (native size() lowering must keep declining a + # non-sequence operand) only ever run here + graphistry/tests/compute/gfql/test_size_nonlist_decline_1985.py graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py graphistry/tests/compute/gfql/test_conformance_ledger.py @@ -49,12 +53,23 @@ POLARS_TEST_FILES=( graphistry/tests/compute/gfql/test_optional_match_semantics.py graphistry/tests/compute/gfql/test_optional_match_with_pipeline_boundaries.py graphistry/tests/compute/gfql/test_row_multiplicity_semantics.py + # whole-entity RETURN bag multiplicity: engine-parametrized pandas/polars/cudf, and the + # polars params (multi-entity binding-row rendering) only ever run here + graphistry/tests/compute/gfql/test_whole_entity_projection_bag_1994.py graphistry/tests/compute/gfql/test_aggregate_identity_row_semantics.py graphistry/tests/compute/gfql/test_numeric_conformance_semantics.py + # engine-parametrized absent-name strictness: the polars params of the level matrix + # (0-rows / null-column / 3VL) only ever run here + graphistry/tests/compute/gfql/test_strictness_levels.py graphistry/tests/compute/gfql/test_path_trail_semantics.py # #1911 alias-scoping pins: every case is parametrized pandas AND polars, and the # polars params (WITH-rebind decline parity, edge-identity collision crash) only run here graphistry/tests/compute/gfql/test_alias_scoping_semantics.py + graphistry/tests/compute/gfql/cypher/test_binding_seed_identity.py + # #1712 reentry-carry seed pins: no module-level importorskip (pandas params run in + # test-gfql-core), but the polars params — native carry restriction + the typed + # scalar-carry declines — only ever run here + graphistry/tests/compute/gfql/test_reentry_carry_seed_restriction.py graphistry/tests/compute/gfql/test_count_and_param_semantics.py graphistry/tests/compute/gfql/row/test_row_pipeline_boundaries.py graphistry/tests/compute/gfql/test_unary_op_surface.py @@ -67,6 +82,11 @@ POLARS_TEST_FILES=( # #1882/#1913-f4/#1879 crash-family pins: the polars params (filter helpers on polars # frames, polars prune_self_edges, nodes-only typed-decline advice) only run here graphistry/tests/compute/gfql/test_crash_family_1882_1879.py + # the polars param here asserts remote execution DECLINES polars frames pre-request + graphistry/tests/compute/test_remote_csv_fidelity.py + # #1889 validate-vs-execute agreement: the polars params (both-frames-None used to raise + # an empty-message AssertionError in ensure_nodes_polars) only ever run in this lane + graphistry/tests/compute/gfql/test_validate_execute_agreement_1889.py graphistry/tests/compute/gfql/test_polars_rows_entity_groupby.py graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py graphistry/tests/compute/gfql/test_residual_polars_native.py @@ -100,8 +120,14 @@ POLARS_TEST_FILES=( # polars params (Z-suffix text-temporal compare, IN [datetime(...)], mixed-type UNION # decline) only run here graphistry/tests/compute/gfql/test_temporal_and_union_semantics_1915.py + # #1915 B-5/B-7/B-8/A-4 + #1880 temporal-half pins: the polars cells (literal + # temporal fold, temporal-vs-string parse-or-E302, union name alignment) only run here + graphistry/tests/compute/gfql/test_temporal_leak_family_1915.py # #1934 incomparable-ordering-null pins: the polars typed-decline cells only run here graphistry/tests/compute/gfql/test_incomparable_ordering_null_1934.py + # #1937 split-month duration scaling: every case is parametrized pandas AND polars, + # and the polars params only run here + graphistry/tests/compute/gfql/test_duration_month_division_1937.py graphistry/tests/compute/gfql/index/test_indexed_bindings.py graphistry/tests/compute/gfql/test_reentry_caller_graph_immutability.py graphistry/tests/compute/gfql/test_rewrite_param_discard.py @@ -116,6 +142,8 @@ POLARS_TEST_FILES=( # index tests exercise the seeded-index hook in the polars hop entry (hop.py) — without # them the hook dominates the now-thin file and trips its per-file coverage floor graphistry/tests/compute/gfql/index/test_index.py + # every cell is polars-only: the indexed-vs-scan EXISTS/NOT EXISTS agreement matrix + graphistry/tests/compute/gfql/index/test_exists_pattern_index_agreement.py # engine-agnostic frame/series primitives (graphistry/Engine.py) — the polars branches of # these dispatch helpers are only measured when this lane covers graphistry (see cov widen below) graphistry/tests/test_engine_frame_helpers.py @@ -126,8 +154,9 @@ POLARS_TEST_FILES=( # of its CI budget; xdist is the lever that does not require a workflow edit (pytest-xdist is # already in the [test] extra, and test-gfql-core already runs `-n auto` under --cov, so # coverage+xdist is an established combination in this repo). -# * worker spec `auto` = os.cpu_count(): 4 on a GitHub-hosted ubuntu-latest runner, and it -# scales DOWN on a 2-vCPU runner where a fixed `-n 4` could be slower than serial. +# * worker spec `auto` = os.cpu_count(): currently 2 on a standard GitHub-hosted +# ubuntu-latest runner. It scales with the runner while avoiding a fixed worker count +# that could oversubscribe smaller runners. # * --maxprocesses caps the count so a 24-core dev box does not fan out 24 polars processes # that then oversubscribe polars' own thread pool. # * --dist load (xdist's default) balances per test. `loadfile` was measured too: it is diff --git a/docs/source/gfql/builtin_calls.rst b/docs/source/gfql/builtin_calls.rst index f31bf4fb12..1db68b9fbd 100644 --- a/docs/source/gfql/builtin_calls.rst +++ b/docs/source/gfql/builtin_calls.rst @@ -1098,19 +1098,19 @@ Arrange nodes in a circular layout. * - sort_by - string or list[string] - No - - Node column(s) for sort order + - Accepted but currently has no effect; circle order is always by node id * - ascending - boolean or list[boolean] - No - - Sort direction + - Accepted but currently has no effect * - na_position - string - No - - ``'first'`` or ``'last'`` + - ``'first'`` or ``'last'``; accepted but currently has no effect * - ignore_index - boolean - No - - Whether to ignore index during sort + - Accepted but currently has no effect * - engine - string - No diff --git a/docs/source/gfql/cypher.rst b/docs/source/gfql/cypher.rst index 8e75ef2063..546c2062c9 100644 --- a/docs/source/gfql/cypher.rst +++ b/docs/source/gfql/cypher.rst @@ -324,7 +324,11 @@ and ``RETURN`` expressions: ``lower`` / ``upper`` (the idiomatic case-insensitive compare, e.g. ``WHERE toLower(n.name) = 'bob'``), plus ``substring`` and ``size``, and conversions ``toInteger`` / ``toFloat`` / ``toString`` / - ``toBoolean`` and ``coalesce``. + ``toBoolean`` and ``coalesce``. ``size`` is defined over strings + (character count) and lists (element count) only; over a numeric, + boolean or temporal column it is a type error and declines. The same + applies to the list-walking forms ``any`` / ``all`` / ``none`` / + ``single`` and list comprehensions. - Regex ``=~`` (see WHERE Forms above). - ``searchAny(entity, term[, opts])`` — cross-column search predicate (WHERE position; GFQL extension for the viz filter pipeline): True where ANY of the diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index d24cce562d..71c599a897 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -44,19 +44,20 @@ than silently bridge), and the GPU engines only pay off on larger work. On CPU, Polars wins the common graph-query shapes (traversal, ``WHERE``/``ORDER``, aggregation) — see *When not to use Polars* below. -.. warning:: - **Already a Polars user? Pass** ``engine='polars'`` **— the default does not.** With the - default ``engine='auto'``, a graph built from ``polars.DataFrame`` is **silently coerced to - pandas** (``auto`` resolves to ``cudf`` for cuDF input and ``pandas`` for everything else, - *including Polars*; it never selects the Polars engine). To stay native end-to-end, pass - ``engine='polars'`` explicitly: +.. note:: + **Already a Polars user? The default now keeps you native.** With the default + ``engine='auto'``, a graph whose bound frames are all ``polars.DataFrame`` runs on the + Polars engine and returns Polars frames. If the query uses a shape the Polars engine + declines, GFQL falls back to pandas for that call — so ``auto`` is native *when it can + be*, and pandas otherwise. Pass ``engine='polars'`` explicitly when you want a decline + to raise instead of silently falling back: .. code-block:: python import polars as pl, graphistry g = graphistry.edges(edges_pl, 'src', 'dst').nodes(nodes_pl, 'id') # polars frames - out = g.gfql(query) # auto -> coerced to PANDAS (out._nodes is pandas!) - out = g.gfql(query, engine='polars') # native Polars in and out (out._nodes is polars) + out = g.gfql(query) # auto -> native Polars (out._nodes is polars) + out = g.gfql(query, engine='polars') # same, but a declined shape raises .. note:: **Result frames match the engine.** With ``engine='polars'`` or ``'polars-gpu'`` the @@ -97,8 +98,12 @@ The four engines - explicit - The Polars fused plan executed on GPU (cudf_polars); fastest on heavy multi-hop. -``engine='auto'`` resolves to ``cudf`` for cuDF input and ``pandas`` otherwise. **AUTO -never selects Polars or Polars-GPU** — they are explicit opt-in (see *Why opt-in?* below). +``engine='auto'`` follows the input frames: Polars frames run on ``polars``, cuDF frames on +``cudf``, everything else on ``pandas``. Two AUTO fast paths go further — all-Polars frames +are tried on ``polars``, and all-cuDF frames are tried on ``polars-gpu`` when a GPU collect +probes usable — each falling back to ``pandas`` / ``cudf`` respectively if the query uses a +shape that engine declines. Passing the engine explicitly turns those declines into errors +instead of a fallback (see *What auto does* below). How the engines compare ----------------------- @@ -300,17 +305,18 @@ The build frame type and the run engine are independent — GFQL coerces the inp frames to the engine you ask for. A pandas graph runs on ``engine='polars'``, a Polars graph runs on ``engine='pandas'``, and so on. The only cost is a **one-time convert** of the input frames at the start of the call; the query then -runs fully on the chosen engine. Note that ``engine='auto'`` (the default) -resolves to ``cudf`` for cuDF input and ``pandas`` for everything else — **it -never selects Polars or Polars-GPU**, so those two are always an explicit opt-in. +runs fully on the chosen engine. Note that ``engine='auto'`` (the default) follows +the input frames — Polars frames run natively on ``polars``, cuDF frames on +``cudf`` (or ``polars-gpu`` when that GPU path probes usable), everything else on +``pandas`` — falling back to ``pandas`` / ``cudf`` only for query shapes the native +engine declines. .. tip:: For selective, seeded traversal, build the CSR adjacency index once with ``g.gfql_index_all()`` (or ``index_policy=``) — it works on all four engines - and turns the O(E) scan into an O(degree) gather. **Polars frames currently need - the engine passed explicitly** — ``g.gfql_index_all(engine='polars')`` — because an - AUTO build swaps Polars frames to pandas (fix tracked in PR #1767). - See :doc:`index_adjacency`. + and turns the O(E) scan into an O(degree) gather. An AUTO build on Polars frames now + keeps them native, so ``g.gfql_index_all()`` and ``g.gfql_index_all(engine='polars')`` + build the same index. See :doc:`index_adjacency`. .. _gfql-offengine-calls: @@ -524,17 +530,27 @@ Then change one keyword — your existing graph and query are unchanged: g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars') # CPU columnar g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars-gpu') # same plan on GPU -Why opt-in? ------------ +What auto does +-------------- + +``auto`` prefers the native engine for your frames and keeps a safety net. A few exotic +Cypher features still require ``engine='pandas'``: the Polars engine **declines them before +execution** rather than silently bridging. Under ``auto`` that decline is caught and the +call is re-served on ``pandas`` (all-cuDF frames decline back to ``cudf``), so a query that +works today keeps working while everything the native engine does support stays native. -Polars and Polars-GPU are explicit (``engine='polars'`` / ``'polars-gpu'``; ``auto`` never -picks them). The main reason is robustness, not speed: a few exotic Cypher features still -require ``engine='pandas'`` and are **rejected before execution** rather than silently -bridge, so auto-selecting Polars would turn queries that work today on pandas into hard -errors. (Performance is rarely the -downside — CPU Polars wins common graph queries past small/interactive sizes; only -trivially small operations favor pandas, immaterially.) Opting in keeps the default -behavior unchanged and guarantees a working result. +Pass the engine explicitly when you would rather know: ``engine='polars'`` / +``'polars-gpu'`` raise ``NotImplementedError`` on a declined shape instead of falling back, +which is what you want in a benchmark or a pipeline that must not silently change engines. +``engine='polars-gpu'`` is additionally GPU-or-error and never quietly runs on CPU. + +Performance is rarely the downside — CPU Polars wins common graph queries past +small/interactive sizes; only trivially small operations favor pandas, immaterially. + +.. note:: + Non-GFQL surfaces (layouts, plotting, featurization) still consume Polars frames as an + *input format* and compute in pandas, so ``auto`` coerces there. The native-under-auto + behavior described above is specific to GFQL query execution. See also -------- diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index 2bca4bc5b8..9dd893260e 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -316,7 +316,7 @@ Key advantages of GFQL Let: Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -GFQL runs the same query on four interchangeable engines, all returning identical results: ``pandas`` (CPU, default), ``polars`` (CPU columnar — often an order of magnitude faster on query-heavy workloads, **no GPU**), ``cudf`` (NVIDIA GPU), and ``polars-gpu`` (NVIDIA GPU). ``engine='auto'`` resolves to ``cudf`` for cuDF input and ``pandas`` otherwise; ``polars`` / ``polars-gpu`` are explicit opt-in (``auto`` never selects them — **so a Polars-frame graph run with the default is coerced to pandas; pass** ``engine='polars'`` **to stay native**). Neither silently bridges: ``polars-gpu`` is GPU-or-error, and unsupported Polars/Cypher shapes are declined during validation, compilation, or planning before execution rather than falling back to pandas. See :doc:`Choosing an Engine ` for the decision matrix and benchmarks. +GFQL runs the same query on four interchangeable engines, all returning identical results: ``pandas`` (CPU, default), ``polars`` (CPU columnar — often an order of magnitude faster on query-heavy workloads, **no GPU**), ``cudf`` (NVIDIA GPU), and ``polars-gpu`` (NVIDIA GPU). ``engine='auto'`` follows the input frames — **a Polars-frame graph runs natively on Polars under the default** — resolving to ``cudf`` for cuDF input and ``pandas`` otherwise; an all-cuDF graph is additionally tried on ``polars-gpu`` when that GPU path probes usable. A query shape the native engine declines falls back to ``pandas`` (or ``cudf``); pass the engine explicitly to get an error instead of a fallback. Neither engine silently bridges mid-query: ``polars-gpu`` is GPU-or-error, and unsupported Polars/Cypher shapes are declined during validation, compilation, or planning — before execution — so the fallback re-runs the query from the start on pandas rather than half-executing. See :doc:`Choosing an Engine ` for the decision matrix and benchmarks. When you use cuDF (GPU) dataframes with ``engine='auto'``, GFQL executes queries on the GPU for massive speedups. diff --git a/docs/source/gfql/schema.rst b/docs/source/gfql/schema.rst index ae87a815b4..fa800532d9 100644 --- a/docs/source/gfql/schema.rst +++ b/docs/source/gfql/schema.rst @@ -92,9 +92,14 @@ Schema Objects ``GraphSchema(node_types, edge_types, strict=True, ...)`` Groups node/edge contracts and adapts them to the internal - ``GraphSchemaCatalog`` used by binder/preflight validation. ``strict=False`` - makes schema-bound ``g.gfql_validate(...)`` permissive by default; callers can - still override per call with ``g.gfql_validate(..., strict=True)``. A physical + ``GraphSchemaCatalog`` used by binder/preflight validation. ``strict`` accepts a + strictness level -- ``"strict"`` (raise), ``"warn"``, ``"quiet"`` -- or the legacy + booleans (``True`` is ``"strict"``, ``False`` is ``"quiet"``). It sets the default for + both ``g.gfql_validate(...)`` and ``g.gfql(...)`` on the bound graph; callers can still + override per call with ``strict=``. A name the schema does not declare is treated as a + typo and raises at every level, while a declared name this instance happens to lack + resolves to null -- the narrow-subgraph case. Without a bound schema the default level is + ``"warn"``. A physical node property column must have the same logical type for every node type that declares it, and a physical edge property column must have the same logical type for every edge type that declares it. Use separate column names when two diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index c85b2408a5..b33f0a4969 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -449,12 +449,38 @@ Boolean"*); GFQL accepts it because summing an indicator column is idiomatic in surface GFQL also serves, and every engine already agrees on the answer. `sum` over a boolean counts the true values; `avg` gives their fraction. +The extension is a strict **superset**: it only accepts input Cypher rejects outright, so no +Cypher-valid query changes meaning under it. + Any other input type **raises**. This is stricter than earlier releases, where a string column returned its *concatenation* on pandas and leaked a raw polars error on polars — wrong in two different directions. Empty and all-null inputs follow Cypher rather than SQL: `sum` returns **0**, `avg` returns -**null**. +**null**. This is conformance, not a compromise — SQL's `SUM` returns NULL over zero rows and +Cypher's returns 0, and every GFQL engine already matched Cypher on both. + +### Aggregates over `BOOLEAN`: return types + +Values *and* return types are specified identically for pandas, polars, cuDF and polars-gpu. +The conformance suite is parametrized over all four, but only pandas, polars and cuDF are +**executed today** — the polars-gpu arm skips for want of `cudf_polars` (RAPIDS 26.02+), so +that arm is specified and not yet verified: + +| Aggregate | Returns | Definition | +|-----------|---------|------------| +| `sum(BOOLEAN)` | `INTEGER` (int64) | count of `true`, nulls skipped; **0** over zero non-null values | +| `avg(BOOLEAN)` | `FLOAT` (float64) | `true_count / non_null_count`; **null** over zero non-null values | +| `min(BOOLEAN)` | `BOOLEAN` | standard boolean ordering `false < true`; **null** over zero non-null values | +| `max(BOOLEAN)` | `BOOLEAN` | standard boolean ordering `false < true`; **null** over zero non-null values | +| `count(BOOLEAN)` | `INTEGER` (int64) | non-null count | + +`min`/`max` are an **ordering**, not a logical fold. `min == AND` and `max == OR` follow from +`false < true` on populated input, but they disagree on the empty one: the conventional identity of +`AND` over zero elements is `true` and of `OR` over zero elements is `false`, while GFQL — like +`ORDER BY`, which already orders booleans this way — answers **null**. + +`count` returns `INTEGER` for every input type, not only `BOOLEAN`. ## Key Differences diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index 0e7ca40e28..74f4b58436 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -39,6 +39,41 @@ Graphs consist of node and edge dataframes: - Edge destination attribute: `g._destination` (e.g., "destination", "to") - GFQL infers nodes from edge references when only edges are provided +#### NULL Identity Resolution and Edge Endpoints + +**A NULL id is not a graph identity.** A source row with NULL in a bound node-id or +edge-endpoint column does not define that identity. An edge whose source or destination is +NULL therefore matches no pattern edge, on every surface (`hop`, chains, Cypher rows, Cypher +aggregates), from either direction, and whether or not the node table holds a row with a NULL +id. A NULL seed id likewise resolves to no node. + +This follows openCypher's three-valued logic: `null = null` evaluates to UNKNOWN rather than +TRUE, so an endpoint that cannot be shown equal to any node identity binds nothing. It is also +the only reading under which a result is self-consistent — a linkable NULL endpoint makes +`MATCH (a)-[x]-(b) RETURN count(*)` disagree with the edges the same pattern returns, and lets +a result frame carry an edge whose endpoint has no node row. + +The rule constrains identity and endpoint **resolution**. It does not define a new input +validation policy: + +- Existing permissive DataFrame ingestion and node-only row scans remain unchanged for + compatibility. That pass-through behavior does not make NULL a valid node identity and is + outside this endpoint-resolution contract. +- `OPTIONAL MATCH` still produces NULL **bindings** for an unmatched optional pattern. Those + are result values, not source graph identities or edge endpoints. +- This release does not add an error for NULL source identity values. Pattern matching and + traversal tolerate them by treating them as unresolved. +- `get_degrees` / `get_indegrees` / `get_outdegrees` are raw edge-row tallies, not pattern + matches, so they still count a NULL-endpoint edge on whichever endpoint is an identity. A + degree column can therefore exceed the number of edges a pattern will match at that node. + + +```python +# nodes id = [0, 1, 2, NULL]; edges (0,1) (1,2) (NULL,2) (2,NULL) +g.gfql([n(), e_undirected(), n()]) # edges (0,1) and (1,2); nodes {0, 1, 2} +g.gfql('MATCH (a)-[x]-(b) RETURN count(*)') # 4 -- two edges, two orientations each +``` + #### GFQL Programs GFQL programs are declarative graph-to-graph transformations: diff --git a/docs/source/gfql/strict_mode.rst b/docs/source/gfql/strict_mode.rst index 3cb1d17745..4fdace7148 100644 --- a/docs/source/gfql/strict_mode.rst +++ b/docs/source/gfql/strict_mode.rst @@ -11,8 +11,33 @@ when you want a report without running the query. Use :py:meth:`g.gfql(..., validate=True) ` when you want the same checks before execution. -Local Cypher execution uses these schema checks. Environment variables or -keyword arguments do not switch local Cypher execution back to a looser mode. +Strictness Levels +----------------- + +How an absent label or property is reported is chosen by a strictness level: + +``"strict"`` + Raise ``GFQLValidationError`` / ``GFQLSchemaError``, before execution where + possible. + +``"warn"`` (default) + Emit one ``UserWarning`` per distinct absent name per call and resolve the name + to null, which is openCypher: an absent label matches nothing, an absent + property makes a predicate null so the row does not match, ``IS NULL`` on an + absent property is true, and an absent property in ``RETURN`` is a null column. + +``"quiet"`` + Same answers as ``"warn"``, silently. + +``warn`` is the default because working on a subgraph with partial columns is +normal usage, not a typo. Pass ``strict=`` to ``g.gfql(...)``, ``g.chain(...)``, +``g.gfql_validate(...)`` and the ``gfql_remote`` family to choose per call. The +legacy booleans still work: ``strict=True`` means ``"strict"`` and +``strict=False`` means ``"quiet"``. + +The level is resolved once and consulted by both the validator and every +executor, so ``g.gfql_validate(q, strict=L)`` and ``g.gfql(q, strict=L)`` always +agree about whether ``q`` is acceptable. What Gets Checked ----------------- @@ -24,8 +49,9 @@ For Cypher queries, strict schema checks verify: in scope. * Property names exist for the node or edge variable they are read from. -Invalid queries raise ``GFQLValidationError`` before execution. Valid queries -run the same as before. +Under ``strict``, invalid queries raise ``GFQLValidationError`` before execution. +Under ``warn``/``quiet``, an absent name resolves to null instead. Valid queries +run the same at every level. It does **not** check every dataframe value's Python or Arrow type. This page is about Cypher names and schema references. @@ -69,8 +95,14 @@ handlers, notebooks, and CI checks. Configuration Notes ------------------- -Most users do not need to configure these checks directly. Prefer -``g.gfql_validate(...)`` or ``g.gfql(..., validate=True)``. +Most users do not need to configure these checks directly. Prefer ``strict=`` on +the call, or ``g.gfql_validate(...)``. + +Declaring a schema with ``bind(schema=...)`` sharpens the levels: a name the +schema does not declare is a typo and raises at every level, while a name the +schema declares but this instance happens to lack is the narrow-subgraph case and +resolves to null. Its ``strict=`` field also supplies the default level for the +bound graph. Code can also set a catalog metadata flag: @@ -94,12 +126,9 @@ or a process-wide environment variable: Truthy values: ``1``, ``true``, ``yes``, ``on`` (case-insensitive). Falsy / unset: anything else (default ``false``). -Treat these as opt-in signals, not as switches that disable validation. Setting -them to ``false`` or leaving them unset does not make local Cypher execution -looser. - -The explicit validation APIs (``g.gfql_validate(strict=True)`` and -``g.gfql(validate=True)``) are unaffected by these helpers. +The environment variable is inert for query behavior: it feeds +``strict_schema_env_default()`` and nothing else reads it. Use ``strict=`` or the +catalog metadata flag to choose a level. Error Messages -------------- diff --git a/graphistry/Engine.py b/graphistry/Engine.py index 2ac1e842bf..d9fe793d8a 100644 --- a/graphistry/Engine.py +++ b/graphistry/Engine.py @@ -31,7 +31,10 @@ class Engine(Enum): POLARS = 'polars' # GPU execution TARGET of the lazy Polars engine (cudf_polars): frames stay # ``pl.DataFrame`` (handled exactly like POLARS in all frame ops); only the - # lazy ``.collect()`` runs on GPU. Explicit opt-in only — AUTO never selects it. + # lazy ``.collect()`` runs on GPU. ``resolve_engine`` never RETURNS this for + # AUTO. ``gfql`` still reaches it under AUTO by a separate route that re-enters + # with an explicit engine when every bound frame is cuDF and a GPU collect + # probes usable; that route declines back to the legacy CUDF path. POLARS_GPU = 'polars-gpu' # Engines whose frames use the polars API (unique/with_columns/...) rather than the diff --git a/graphistry/Plottable.py b/graphistry/Plottable.py index 52fdb5a03a..8dd9e1377c 100644 --- a/graphistry/Plottable.py +++ b/graphistry/Plottable.py @@ -4,7 +4,7 @@ from graphistry.io.types import ComplexEncodingsDict from graphistry.models.ModelDict import ModelDict -from graphistry.models.compute.chain_remote import FormatType, OutputTypeAll, OutputTypeDf, OutputTypeGraph +from graphistry.models.compute.chain_remote import DFImportArgs, FormatType, OutputTypeAll, OutputTypeDf, OutputTypeGraph from graphistry.models.compute.dbscan import DBSCANEngine from graphistry.models.compute.umap import UMAPEngineConcrete from graphistry.models.compute.features import GraphEntityKind @@ -559,7 +559,9 @@ def chain_remote( edge_col_subset: Optional[List[str]] = None, engine: Optional[Literal["pandas", "cudf"]] = None, validate: bool = True, - persist: bool = False + persist: bool = False, + df_import_args: Optional[DFImportArgs] = None, + strict: Any = None, # hygiene-ok: explicit-any -- bool | strictness level | None; see gfql.strictness.StrictInput ) -> 'Plottable': """ chain is Union[List[ASTObject], Chain] @@ -577,7 +579,11 @@ def chain_remote_shape( edge_col_subset: Optional[List[str]] = None, engine: Optional[Literal["pandas", "cudf"]] = None, validate: bool = True, - persist: bool = False + persist: bool = False, + df_import_args: Optional[DFImportArgs] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() + output: Optional[str] = None, + strict: Any = None, # hygiene-ok: explicit-any -- bool | strictness level | None; see gfql.strictness.StrictInput ) -> pd.DataFrame: """ chain is Union[List[ASTObject], Chain] @@ -596,7 +602,11 @@ def gfql_remote( edge_col_subset: Optional[List[str]] = None, engine: EngineAbstractType = 'auto', validate: bool = True, - persist: bool = False + persist: bool = False, + df_import_args: Optional[DFImportArgs] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars + output: Optional[str] = None, + strict: Any = None, # hygiene-ok: explicit-any -- bool | strictness level | None; see gfql.strictness.StrictInput ) -> 'Plottable': """ chain is Union[List[ASTObject], Chain] @@ -614,7 +624,11 @@ def gfql_remote_shape( edge_col_subset: Optional[List[str]] = None, engine: EngineAbstractType = 'auto', validate: bool = True, - persist: bool = False + persist: bool = False, + df_import_args: Optional[DFImportArgs] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() + output: Optional[str] = None, + strict: Any = None, # hygiene-ok: explicit-any -- bool | strictness level | None; see gfql.strictness.StrictInput ) -> pd.DataFrame: """ chain is Union[List[ASTObject], Chain] @@ -630,7 +644,8 @@ def python_remote_g( output_type: Optional[OutputTypeAll] = 'all', engine: EngineAbstractType = 'auto', run_label: Optional[str] = None, - validate: bool = True + validate: bool = True, + df_import_args: Optional[DFImportArgs] = None, ) -> 'Plottable': ... @@ -643,7 +658,8 @@ def python_remote_table( output_type: Optional[OutputTypeDf] = 'table', engine: EngineAbstractType = 'auto', run_label: Optional[str] = None, - validate: bool = True + validate: bool = True, + df_import_args: Optional[DFImportArgs] = None, ) -> pd.DataFrame: ... diff --git a/graphistry/PlotterBase.py b/graphistry/PlotterBase.py index ba0920e8ae..ac1a5ee4b4 100644 --- a/graphistry/PlotterBase.py +++ b/graphistry/PlotterBase.py @@ -688,7 +688,7 @@ def encode_point_color( :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :returns: Plotter @@ -757,7 +757,7 @@ def encode_edge_color( :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :returns: Plotter @@ -793,7 +793,7 @@ def encode_point_size( :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :returns: Plotter @@ -881,7 +881,7 @@ def encode_point_icon( :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :param as_text: Values should instead be treated as raw strings, instead of icons and images. (Default False.) @@ -958,7 +958,7 @@ def encode_edge_icon( :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :param as_text: Values should instead be treated as raw strings, instead of icons and images. (Default False.) @@ -4094,7 +4094,7 @@ def hypergraph( and the renderable result Plotter. Hypergraphs reveal relationships between rows and between column values. This transform is useful for lists of events, samples, relationships, and other structured high-dimensional data. - Specify local compute engine by passing `engine='pandas'`, 'cudf', 'dask', 'dask_cudf' (default: 'pandas'). + Specify local compute engine by passing `engine='pandas'`, 'cudf', 'dask', 'dask_cudf' (default: 'auto', which selects the engine from the input dataframe type). If events are not in that engine's format, they will be converted into it. The transform creates a node for every unique value in the entity_types columns (default: all columns). diff --git a/graphistry/compute/ComputeMixin.py b/graphistry/compute/ComputeMixin.py index bba5024185..6551a2af30 100644 --- a/graphistry/compute/ComputeMixin.py +++ b/graphistry/compute/ComputeMixin.py @@ -11,6 +11,7 @@ from .chain_let import chain_let as chain_let_base from .gfql_unified import gfql as gfql_base from .gfql_validate import gfql_validate as gfql_validate_base +from .gfql.strictness import StrictInput from .chain_remote import ( chain_remote as chain_remote_base, chain_remote_shape as chain_remote_shape_base @@ -20,7 +21,7 @@ python_remote_table as python_remote_table_base, python_remote_json as python_remote_json_base ) -from graphistry.models.compute.chain_remote import OutputTypeGraph, FormatType +from graphistry.models.compute.chain_remote import DFImportArgs, OutputTypeGraph, FormatType from .collapse import collapse_by from .hop import hop as hop_base from .filter_by_dict import ( @@ -829,8 +830,10 @@ def gfql_remote( engine: EngineAbstractType = 'auto', validate: bool = True, persist: bool = False, - params: Optional[Dict[str, Any]] = None, + df_import_args: Optional[DFImportArgs] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() output: Optional[str] = None, + strict: StrictInput = None, ) -> Plottable: """Run GFQL query remotely. @@ -846,6 +849,8 @@ def gfql_remote( Cypher string (compiled locally before sending). :param params: Optional parameter dict for Cypher string queries (e.g., ``params={"val": 10}`` for ``$val`` references). + :param strict: Absent-label/property strictness for the local preflight, also sent + to the server as the ``strictness`` request field; see :meth:`gfql`. Example:: @@ -866,7 +871,7 @@ def gfql_remote( return chain_remote_base( self, chain, api_token, dataset_id, output_type, format, df_export_args, node_col_subset, edge_col_subset, engine, validate, persist, - params=params, output=output, + params=params, output=output, df_import_args=df_import_args, strict=strict, ) def gfql_remote_shape( @@ -880,18 +885,29 @@ def gfql_remote_shape( edge_col_subset: Optional[List[str]] = None, engine: EngineAbstractType = 'auto', validate: bool = True, - persist: bool = False + persist: bool = False, + df_import_args: Optional[DFImportArgs] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() + output: Optional[str] = None, + strict: StrictInput = None, ) -> pd.DataFrame: """Get shape metadata for remote GFQL query execution. This is the remote shape version of :meth:`gfql`. Returns metadata about the resulting graph without downloading the full data. + :param params: Optional parameter dict for Cypher string queries + (e.g., ``params={"cutoff": 10}`` for ``$cutoff`` references). + :param output: Optional Let/DAG binding name to return; requires a Let/DAG query. + :param strict: Absent-name strictness sent to the server as ``strictness``; + see :meth:`gfql`. + See :meth:`chain_remote_shape` for detailed documentation (chain_remote_shape is deprecated). """ return chain_remote_shape_base( self, chain, api_token, dataset_id, format, df_export_args, - node_col_subset, edge_col_subset, engine, validate, persist + node_col_subset, edge_col_subset, engine, validate, persist, + df_import_args=df_import_args, params=params, output=output, strict=strict, ) def python_remote_g(self, *args, **kwargs) -> Any: diff --git a/graphistry/compute/ast.py b/graphistry/compute/ast.py index 8f68292e2c..c25c93db71 100644 --- a/graphistry/compute/ast.py +++ b/graphistry/compute/ast.py @@ -121,10 +121,10 @@ def maybe_filter_dict_from_json(d: Dict, key: str) -> Optional[Dict]: def _filter_dict_to_json(filter_dict: Dict[str, Any]) -> Dict[str, Any]: + # Keep None values: dropping an entry widens the filter to match-everything. return { k: v.to_json() if isinstance(v, ASTPredicate) else v for k, v in filter_dict.items() - if v is not None } diff --git a/graphistry/compute/chain.py b/graphistry/compute/chain.py index b37497189d..8b87ac6c0c 100644 --- a/graphistry/compute/chain.py +++ b/graphistry/compute/chain.py @@ -12,18 +12,20 @@ from .typing import DataFrameT, SeriesT from .util import generate_safe_column_name from .chain_fast_paths import _seeded_typed_hop_pandas_cudf, _tag_fast_path_aliases -from graphistry.compute.validate.validate_schema import validate_chain_schema +from graphistry.compute.validate.validate_schema import validate_chain_schema, validate_graph_shape +from graphistry.compute.gfql.strictness import StrictInput from graphistry.compute.gfql.same_path_types import ( WhereComparison, normalize_where_entries, parse_where_json, where_to_json, ) -from .gfql.policy import PolicyContext, PolicyException +from .gfql.policy import PolicyContext, PolicyException, PolicyFunction from .gfql.policy.stats import extract_graph_stats from graphistry.otel import otel_traced, otel_detail_enabled if TYPE_CHECKING: + from .execution_context import ExecutionContext from graphistry.compute.exceptions import GFQLSchemaError, GFQLValidationError from graphistry.compute.gfql.index.handoff import IndexedBindingsHandoff @@ -516,11 +518,23 @@ def apply_output_slice(op: ASTObject, op_label: ASTObject, df): out_df[op._name] = label_mask cols = list(out_df.columns) + # An alias named like a user column collides here (marker `_x` from the step frames + # vs user values `_y` from the base frame). The marker is authoritative (null = + # unbound row); user values are restored from the base frame at property-read time, + # never coalesced into the marker (mixed bool/user dtypes also crash cuDF). + alias_marker_names = { + op._name for op, _ in steps + if isinstance(op, op_type) and isinstance(getattr(op, '_name', None), str) + } for c in cols: if c.endswith('_x'): base = c[:-2] c_y = base + '_y' if c_y in out_df.columns: + if base in alias_marker_names: + out_df[base] = out_df[c].fillna(False).astype(bool) + out_df = out_df.drop(columns=[c, c_y]) + continue if len(out_df) > 0: out_df[base] = out_df[c].where(out_df[c].notna(), out_df[c_y]) out_df = out_df.drop(columns=[c, c_y]) @@ -1006,7 +1020,8 @@ def chain( validate_schema: bool = True, policy=None, context=None, - start_nodes: Optional[DataFrameT] = None + start_nodes: Optional[DataFrameT] = None, + strict: StrictInput = None, ) -> Plottable: """ Chain a list of ASTObject (node/edge) traversal operations @@ -1023,10 +1038,33 @@ def chain( :param policy: Optional policy dict for hooks :param context: Optional ExecutionContext for tracking execution state :param start_nodes: Optional node wavefront for the first traversal step + :param strict: Absent-name strictness: ``"strict"`` raises, ``"warn"`` (default) warns + once per absent name and resolves it to null, ``"quiet"`` resolves silently. + ``True``/``False`` map to ``"strict"``/``"quiet"``. ``None`` consults + ``bind(schema=...)``, then the ``"warn"`` default. :returns: Plotter :rtype: Plotter """ + from graphistry.compute.gfql.strictness import ( + resolve_strict_level, schema_declared_names, strictness_scope) + + with strictness_scope( + resolve_strict_level(self, strict=strict), declared=schema_declared_names(self) + ): + return _chain_with_strictness( + self, ops, engine, validate_schema, policy, context, start_nodes) + + +def _chain_with_strictness( + self: Plottable, + ops: Union[List[ASTObject], Chain], + engine: Union[EngineAbstract, str], + validate_schema: bool, + policy: Optional[Dict[str, PolicyFunction]], + context: Optional['ExecutionContext'], + start_nodes: Optional[DataFrameT], +) -> Plottable: if context is None: from .execution_context import ExecutionContext context = ExecutionContext() @@ -1068,6 +1106,7 @@ def chain( # (Dependency guards for polars / cudf_polars are above, pre-coercion.) if validate_schema: Chain(ops if not isinstance(ops, Chain) else ops.chain).validate(collect_all=False) + validate_graph_shape(self, ops, collect_all=False) # pandas gets this via validate_chain_schema (#1889) from graphistry.compute.gfql.lazy.engine.polars.chain import chain_polars from graphistry.compute.gfql.lazy import target_mode, ExecutionTarget # NO pandas fallback here (no-silent-fallback policy): chain_polars raises diff --git a/graphistry/compute/chain_remote.py b/graphistry/compute/chain_remote.py index 90e06892ce..8beaf9e32e 100644 --- a/graphistry/compute/chain_remote.py +++ b/graphistry/compute/chain_remote.py @@ -9,17 +9,33 @@ import warnings import zipfile -from graphistry.Engine import Engine, EngineAbstractType, resolve_input_engine +from graphistry.Engine import EngineAbstractType from graphistry.Plottable import Plottable from graphistry.client_session import DatasetInfo from graphistry.compute.ast import ASTLet, ASTObject from graphistry.compute.chain import Chain from graphistry.compute.gfql.cypher.lowering import compile_cypher_query from graphistry.compute.gfql.cypher.parser import parse_cypher +from graphistry.compute.gfql.strictness import ( + DEFAULT_STRICT_LEVEL, StrictInput, resolve_strict_level, schema_declared_names, strictness_scope) from graphistry.compute.gfql_validate import gfql_validate as gfql_preflight_validate from graphistry.io.metadata import deserialize_plottable_metadata -from graphistry.models.compute.chain_remote import OutputTypeGraph, FormatType, output_types_graph -from graphistry.utils.json import JSONVal +from graphistry.compute.exceptions import ErrorCode, GFQLSyntaxError, GFQLTypeError +from graphistry.compute.remote_df_io import ( + require_supported_frame_library, + resolve_csv_reader, + resolve_remote_engine, + validate_csv_import_args) +from graphistry.compute.remote_response import ( + check_subset_result_bindings, + decode_json_result, + error_document_error, + raise_for_remote_error, + require_json_result_keys, + select_zip_member, +) +from graphistry.models.compute.chain_remote import DFImportArgs, OutputTypeGraph, FormatType, output_types_graph +from graphistry.utils.json import JSONVal, find_non_finite from graphistry.otel import inject_trace_headers @@ -129,24 +145,18 @@ def chain_remote_generic( engine: EngineAbstractType = 'auto', validate: bool = True, persist: bool = False, - params: Optional[Dict[str, Any]] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() output: Optional[str] = None, + df_import_args: Optional[DFImportArgs] = None, + strict: StrictInput = None, ) -> Union[Plottable, pd.DataFrame]: - if not api_token: - self._pygraphistry.refresh() - api_token = self.session.api_token + strict_level = resolve_strict_level(self, strict=strict) if output_type not in output_types_graph: raise ValueError(f"Unknown output_type, expected one of {output_types_graph}, got: {output_type}") - # Resolve engine: auto -> pandas/cudf based on graph DataFrame type - engine_resolved = resolve_input_engine(engine, self) - if engine_resolved not in [Engine.PANDAS, Engine.CUDF]: - raise ValueError(f"Remote GFQL only supports 'pandas' or 'cudf' engines (or 'auto' which resolves to one of them). " - f"Got engine='{engine}' which resolved to '{engine_resolved.value}'. " - f"Dask engines are not supported for remote execution.") - engine_str = engine_resolved.value + engine_str = resolve_remote_engine(engine, self, "gfql_remote").value if format is None: if output_type == "shape": @@ -154,6 +164,9 @@ def chain_remote_generic( else: format = "parquet" + validate_csv_import_args(df_import_args, "gfql_remote") + frame_lib = require_supported_frame_library(self._nodes, self._edges, "gfql_remote") + # Validate persist compatibility early if persist and output_type in ["nodes", "edges"]: raise ValueError(f"persist=True is not supported with output_type='{output_type}'. " @@ -192,16 +205,32 @@ def chain_remote_generic( else: raise TypeError(f"gfql_remote() query must be Chain, List, ASTLet, Dict, or str. Got {type(chain)}") - if validate: - gfql_preflight_validate( - self, - chain, - params=params, - strict=False, - collect_all=False, - schema=False, + if output is not None and not is_let: + raise GFQLSyntaxError( + ErrorCode.E109, + "output= names a binding to return and requires a Let/DAG query; " + "this query compiled to a flat chain, which has no bindings", + field="output", + value=output, + suggestion="Drop output=, or express the query as a Let/DAG (or Cypher with named graph bindings)", ) + if validate: + declared = schema_declared_names(self) # a declared schema is names without data (#1916) + with strictness_scope(strict_level, declared=declared): + gfql_preflight_validate( + self, + chain, + params=params, + strict=strict_level, + collect_all=False, + schema=False, + ) + + if not api_token: + self._pygraphistry.refresh() + api_token = self.session.api_token + if not dataset_id: dataset_id = self._dataset_id @@ -243,6 +272,16 @@ def chain_remote_generic( if df_export_args is not None: request_body["df_export_args"] = df_export_args request_body["engine"] = engine_str + request_body["strictness"] = strict_level + if strict_level != DEFAULT_STRICT_LEVEL: + warnings.warn( + f"gfql_remote() is requesting strictness={strict_level!r}. Servers that do not " + "read the strictness field apply their own default, so absent labels/properties " + "may still be reported differently than requested. Upgrade to a server that reads " + "strictness for end-to-end parity.", + UserWarning, + stacklevel=2, + ) if persist: request_body["persist"] = persist @@ -250,6 +289,15 @@ def chain_remote_generic( if hasattr(self, '_privacy') and self._privacy is not None: request_body["privacy"] = dict(self._privacy) + non_finite = find_non_finite(request_body) + if non_finite is not None: + raise GFQLTypeError( + ErrorCode.E201, + "Filter values must be predicates or JSON-serializable: NaN and infinity have no JSON representation", + field=non_finite, + suggestion="Use is_na()/notna() predicates, or a finite bound", + ) + url = f"{self.base_url_server()}/api/v2/etl/datasets/{dataset_id}/gfql/{output_type}" # Prepare headers @@ -261,46 +309,27 @@ def chain_remote_generic( response = requests.post(url, headers=headers, json=request_body, verify=self.session.certificate_validation) - # Enhanced error handling for GFQL validation errors - if not response.ok: - try: - # Try to parse JSON error response for more details - if response.headers.get('content-type', '').startswith('application/json'): - error_data = response.json() - error_msg = error_data.get('error', str(error_data)) - raise ValueError(f"GFQL remote operation failed: {error_msg} (HTTP {response.status_code})") - else: - # Fallback to generic error with response text - raise ValueError(f"GFQL remote operation failed: {response.text[:500]} (HTTP {response.status_code})") - except (ValueError,) as ve: - # Re-raise our custom ValueError - raise ve - except Exception: - # If JSON parsing fails, re-raise the original HTTP error - response.raise_for_status() + raise_for_remote_error(response, "GFQL remote operation") # deserialize based on output_type & format - # Determine DataFrame library by checking both edges and nodes - edges_is_cudf = self._edges is not None and 'cudf.core.dataframe' in str(getmodule(self._edges)) - nodes_is_cudf = self._nodes is not None and 'cudf.core.dataframe' in str(getmodule(self._nodes)) - - if edges_is_cudf or nodes_is_cudf: + # Library was resolved pre-request; reuse it so the two cannot drift. + if frame_lib == "cudf": import cudf df_cons = cudf.DataFrame read_csv = cudf.read_csv read_parquet = cudf.read_parquet - elif (self._edges is None or isinstance(self._edges, pd.DataFrame) or 'unittest.mock' in str(type(self._edges))) and \ - (self._nodes is None or isinstance(self._nodes, pd.DataFrame) or 'unittest.mock' in str(type(self._nodes))): + else: df_cons = pd.DataFrame read_csv = pd.read_csv read_parquet = pd.read_parquet - else: - raise ValueError(f"Unknown DataFrame types - edges: {type(self._edges)}, nodes: {type(self._nodes)}") + + if format == "csv": + read_csv = resolve_csv_reader(read_csv, df_import_args, "gfql_remote") if output_type == "shape": if format == "json": - return pd.DataFrame(response.json()) + return pd.DataFrame(decode_json_result(response, "GFQL remote operation")) elif format == "csv": return read_csv(BytesIO(response.content)) elif format == "parquet": @@ -310,79 +339,69 @@ def chain_remote_generic( elif output_type == "all" and format in ["csv", "parquet"]: zip_buffer = BytesIO(response.content) try: - with zipfile.ZipFile(zip_buffer, "r") as zip_ref: - nodes_file = [f for f in zip_ref.namelist() if "nodes" in f][0] - edges_file = [f for f in zip_ref.namelist() if "edges" in f][0] + zip_ref_cm = zipfile.ZipFile(zip_buffer, "r") + except zipfile.BadZipFile as e: + raise error_document_error(response, "GFQL remote operation", "a zip archive") from e + with zip_ref_cm as zip_ref: + names = zip_ref.namelist() + nodes_file = select_zip_member(names, "nodes", "GFQL remote operation") + edges_file = select_zip_member(names, "edges", "GFQL remote operation") - nodes_data = zip_ref.read(nodes_file) - edges_data = zip_ref.read(edges_file) + nodes_data = zip_ref.read(nodes_file) + edges_data = zip_ref.read(edges_file) - if len(nodes_data) > 0: - nodes_df = read_parquet(BytesIO(nodes_data)) if format == "parquet" else read_csv(BytesIO(nodes_data)) - else: - nodes_df = df_cons() + if len(nodes_data) > 0: + nodes_df = read_parquet(BytesIO(nodes_data)) if format == "parquet" else read_csv(BytesIO(nodes_data)) + else: + nodes_df = df_cons() - if len(edges_data) > 0: - edges_df = read_parquet(BytesIO(edges_data)) if format == "parquet" else read_csv(BytesIO(edges_data)) - else: - edges_df = df_cons() + if len(edges_data) > 0: + edges_df = read_parquet(BytesIO(edges_data)) if format == "parquet" else read_csv(BytesIO(edges_data)) + else: + edges_df = df_cons() - result = self.edges(edges_df).nodes(nodes_df) + result = self.edges(edges_df).nodes(nodes_df) - # Check for metadata.json in zip (both persist and GFQL metadata) - if 'metadata.json' in zip_ref.namelist(): - try: - metadata_content = zip_ref.read('metadata.json') - metadata = json.loads(metadata_content.decode('utf-8')) + # Check for metadata.json in zip (both persist and GFQL metadata) + if 'metadata.json' in zip_ref.namelist(): + try: + metadata_content = zip_ref.read('metadata.json') + metadata = json.loads(metadata_content.decode('utf-8')) - if persist: - # Extract dataset_id for URL generation - if 'dataset_id' in metadata: - result._dataset_id = metadata['dataset_id'] - - # Generate URL using existing infrastructure - if result._dataset_id: # Type guard - _refresh_url_from_dataset_id(result) - - # Optionally restore privacy settings - if 'privacy' in metadata: - result._privacy = metadata['privacy'] - - if 'gfql_metadata' in metadata: - result = deserialize_plottable_metadata(metadata['gfql_metadata'], result) - _apply_persist_axis_defaults(result) - if persist: + if persist: + # Extract dataset_id for URL generation + if 'dataset_id' in metadata: + result._dataset_id = metadata['dataset_id'] + + # Generate URL using existing infrastructure + if result._dataset_id: # Type guard _refresh_url_from_dataset_id(result) - except Exception as e: + # Optionally restore privacy settings + if 'privacy' in metadata: + result._privacy = metadata['privacy'] + + if 'gfql_metadata' in metadata: + result = deserialize_plottable_metadata(metadata['gfql_metadata'], result) + _apply_persist_axis_defaults(result) if persist: - warnings.warn(f"persist=True requested but failed to parse metadata.json: {e}. " - f"URL generation will not be available. This may indicate an older server version.", - UserWarning, stacklevel=2) - else: - warnings.warn(f"Failed to parse metadata.json: {e}. GFQL metadata will not be hydrated.", - UserWarning, stacklevel=2) - elif persist: - warnings.warn("persist=True requested but server did not return metadata.json. " - "URL generation will not be available. This indicates an older server version that doesn't support zip format persistence.", + _refresh_url_from_dataset_id(result) + + except Exception as e: + if persist: + warnings.warn(f"persist=True requested but failed to parse metadata.json: {e}. " + f"URL generation will not be available. This may indicate an older server version.", UserWarning, stacklevel=2) + else: + warnings.warn(f"Failed to parse metadata.json: {e}. GFQL metadata will not be hydrated.", + UserWarning, stacklevel=2) + elif persist: + warnings.warn("persist=True requested but server did not return metadata.json. " + "URL generation will not be available. This indicates an older server version that doesn't support zip format persistence.", + UserWarning, stacklevel=2) - return result - except zipfile.BadZipFile as e: - # Server likely returned an error response instead of zip data - # Try to parse the response as JSON for a better error message - try: - if response.headers.get('content-type', '').startswith('application/json'): - error_data = response.json() - error_msg = error_data.get('error', str(error_data)) - raise ValueError(f"GFQL remote operation failed with validation error: {error_msg}") - else: - # Show the response text for debugging - raise ValueError(f"GFQL remote operation failed - server returned non-zip response: {response.text[:500]}") - except Exception: - # If all else fails, re-raise the original BadZipFile error with context - raise ValueError(f"GFQL remote operation failed - server response is not a valid zip file. " - f"This usually indicates a server validation error. Response status: {response.status_code}") from e + check_subset_result_bindings(result, node_col_subset, edge_col_subset, "GFQL remote operation") + return result elif output_type in ["nodes", "edges"] and format in ["csv", "parquet"]: data = BytesIO(response.content) if len(response.content) > 0: @@ -396,11 +415,12 @@ def chain_remote_generic( out = self.edges(df) out._nodes = None - + check_subset_result_bindings(out, node_col_subset, edge_col_subset, "GFQL remote operation") return out elif format == "json": - o = response.json() + o = decode_json_result(response, "GFQL remote operation") if output_type == "all": + o = require_json_result_keys(o, ['nodes', 'edges'], response, "GFQL remote operation") result = self.edges(df_cons(o['edges'])).nodes(df_cons(o['nodes'])) elif output_type == "nodes": result = self.nodes(df_cons(o)) @@ -430,6 +450,7 @@ def chain_remote_generic( if persist: _refresh_url_from_dataset_id(result) + check_subset_result_bindings(result, node_col_subset, edge_col_subset, "GFQL remote operation") return result else: raise ValueError(f"Unsupported format {format}, output_type {output_type}") @@ -446,7 +467,11 @@ def chain_remote_shape( edge_col_subset: Optional[List[str]] = None, engine: EngineAbstractType = 'auto', validate: bool = True, - persist: bool = False + persist: bool = False, + df_import_args: Optional[DFImportArgs] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() + output: Optional[str] = None, + strict: StrictInput = None, ) -> pd.DataFrame: """ Like chain_remote(), except instead of returning a Plottable, returns a pd.DataFrame of the shape of the resulting graph. @@ -473,6 +498,12 @@ def chain_remote_shape( shape_df = g1.chain_remote_shape([n(), e(), n()], engine='cudf') print(shape_df) + + :param params: Optional parameter dict for Cypher string queries (e.g. ``params={"cut": 10}`` for ``$cut``). + :type params: Optional[Dict[str, Any]] + + :param output: Optional Let/DAG binding name to return. Requires a Let/DAG query. + :type output: Optional[str] """ out_df = chain_remote_generic( @@ -487,7 +518,11 @@ def chain_remote_shape( edge_col_subset, engine, validate, - persist + persist, + params=params, + output=output, + df_import_args=df_import_args, + strict=strict, ) assert isinstance(out_df, pd.DataFrame) return out_df @@ -505,8 +540,10 @@ def chain_remote( engine: EngineAbstractType = 'auto', validate: bool = True, persist: bool = False, - params: Optional[Dict[str, Any]] = None, + params: Optional[Dict[str, Any]] = None, # hygiene-ok: explicit-any -- Cypher params are heterogeneous JSON scalars, matching gfql_remote() output: Optional[str] = None, + df_import_args: Optional[DFImportArgs] = None, + strict: StrictInput = None, ) -> Plottable: """Remotely run GFQL chain query on a remote dataset. @@ -524,12 +561,15 @@ def chain_remote( :param output_type: Whether to return nodes and edges ("all", default), Plottable with just nodes ("nodes"), or Plottable with just edges ("edges"). For just a dataframe of the resultant graph shape (output_type="shape"), use instead chain_remote_shape(). :type output_type: OutputType - :param format: What format to fetch results. We recommend a columnar format such as parquet, which it defaults to when output_type is not shape. + :param format: What format to fetch results. We recommend a columnar format such as parquet, which it defaults to when output_type is not shape. ``'csv'`` is untyped on the wire: the client re-infers dtypes and can rewrite values, so it warns and serves. Pass ``df_import_args`` to control the reader. :type format: Optional[FormatType] :param df_export_args: When server parses data, any additional parameters to pass in. :type df_export_args: Optional[Dict, str, Any]] + :param df_import_args: Reader kwargs the client applies when decoding a ``format='csv'`` response. Optional; without it csv dtypes are re-inferred from text, which can rewrite values (``'007'`` -> ``7.0``) and break the returned graph's own node/edge id join. The warning names each lossy axis your kwargs do not govern, and clears only once they govern both: dtype inference (``dtype``/``converters``) and NA substitution (``keep_default_na``/``na_values``/``na_filter``/``converters``). Prefer ``format='parquet'``, which is faithful and needs no reader args. + :type df_import_args: Optional[Dict[str, Any]] + :param node_col_subset: When server returns nodes, what property subset to return. Defaults to all. :type node_col_subset: Optional[List[str]] @@ -596,6 +636,8 @@ def chain_remote( persist, params=params, output=output, + df_import_args=df_import_args, + strict=strict, ) assert isinstance(g, Plottable) return g diff --git a/graphistry/compute/collapse.py b/graphistry/compute/collapse.py index e4e8752bf6..cd2212f440 100644 --- a/graphistry/compute/collapse.py +++ b/graphistry/compute/collapse.py @@ -447,7 +447,8 @@ def normalize_graph( :param g: graphistry instance :param self_edges: bool, whether to keep duplicates from ndf, edf, default False - :param unwrap: bool, whether to unwrap node text with `~`, default True + :param unwrap: bool, whether to strip the `~` wrapping from collapsed node/src/dst ids + (readability only, but it changes the emitted ids), default False :returns: final graphistry instance """ diff --git a/graphistry/compute/endpoint_utils.py b/graphistry/compute/endpoint_utils.py new file mode 100644 index 0000000000..789c7ac22b --- /dev/null +++ b/graphistry/compute/endpoint_utils.py @@ -0,0 +1,34 @@ +from __future__ import annotations + +import typing + +from graphistry.Engine import is_polars_df +from graphistry.compute.typing import DataFrameT + +# Preserve the caller's pandas/cuDF/Polars frame flavor; DataFrameT is pandas-only in type checks. +EndpointFrameT = typing.TypeVar("EndpointFrameT") + + +def _drop_null_endpoint_edges_pandas_cudf( + frame: DataFrameT, source: str, destination: str +) -> DataFrameT: + source_null = frame[source].isna() + destination_null = frame[destination].isna() + if not (bool(source_null.any()) or bool(destination_null.any())): + return frame + return frame.loc[~(source_null | destination_null)] + + +def drop_null_endpoint_edges( + frame: EndpointFrameT, source: str, destination: str +) -> EndpointFrameT: + """Return only edges whose source and destination are identities.""" + if is_polars_df(frame): + import polars as pl + + result: DataFrameT = frame.filter( + pl.col(source).is_not_null() & pl.col(destination).is_not_null() + ) + return result + result = _drop_null_endpoint_edges_pandas_cudf(frame, source, destination) + return result diff --git a/graphistry/compute/exceptions.py b/graphistry/compute/exceptions.py index eb88f8d037..55d4f876a3 100644 --- a/graphistry/compute/exceptions.py +++ b/graphistry/compute/exceptions.py @@ -10,6 +10,7 @@ class ErrorCode: - E1xx: Syntax errors (structural issues) - E2xx: Type errors (type mismatches) - E3xx: Schema errors (data-related issues) + - E4xx: Remote transport/response errors """ # Syntax errors (E1xx) @@ -21,6 +22,7 @@ class ErrorCode: E106 = "empty-chain" E107 = "invalid-cypher-syntax" E108 = "unsupported-cypher-query" + E109 = "output-requires-let-query" # Type errors (E2xx) E201 = "type-mismatch" @@ -34,6 +36,14 @@ class ErrorCode: E302 = "incompatible-column-type" E303 = "invalid-node-reference" E304 = "invalid-edge-reference" + E305 = "graph-not-bound" + + # Remote transport/response errors (E4xx) + E401 = "remote-request-failed" + E402 = "remote-response-malformed" + E403 = "remote-format-lossy" + E404 = "remote-unsupported-frames" + E405 = "remote-unsupported-engine" # Graph constructor errors (E150-E159) E150 = "duplicate-graph-binding" @@ -124,3 +134,12 @@ class GFQLTypeError(GFQLValidationError): class GFQLSchemaError(GFQLValidationError): """Schema validation errors (column existence, type compatibility).""" pass + + +class GFQLRemoteError(GFQLValidationError, ValueError): + """Remote call failed or returned a response the client cannot use. + + Also a ``ValueError`` so callers written against the previous untyped + remote errors keep working. + """ + pass diff --git a/graphistry/compute/filter_by_dict.py b/graphistry/compute/filter_by_dict.py index 9a2704ff0e..2b272cbb5e 100644 --- a/graphistry/compute/filter_by_dict.py +++ b/graphistry/compute/filter_by_dict.py @@ -82,6 +82,10 @@ def resolve_filter_column(df: DataFrameT, col: str, val: Any) -> Tuple[str, Any] if "type" in df.columns and not _looks_like_edge_dataframe(df): return "type", label + # mirror of the rewrite above, for frames carrying labels as per-label boolean columns + if col in ("type", "labels") and isinstance(val, str) and f"label__{val}" in df.columns: + return f"label__{val}", True + from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError raise GFQLSchemaError( @@ -93,6 +97,26 @@ def resolve_filter_column(df: DataFrameT, col: str, val: Any) -> Tuple[str, Any] ) +def resolve_filter_column_or_absent( + df: DataFrameT, + col: str, + val: Any, # hygiene-ok: explicit-any -- filter values are heterogeneous by contract + *, + context: Optional[str] = None, +) -> Optional[Tuple[str, Any]]: # hygiene-ok: explicit-any -- mirrors resolve_filter_column's heterogeneous value contract + """``resolve_filter_column``, but ``None`` when the resolved strictness level + says an absent column resolves to null rather than raising.""" + from graphistry.compute.exceptions import GFQLSchemaError + from graphistry.compute.gfql.strictness import absent_filter_key_is_lenient + + try: + return resolve_filter_column(df, col, val) + except GFQLSchemaError: # only an absent column is leniency-eligible; other errors are real + if absent_filter_key_is_lenient(col, val, context=context): + return None + raise + + def filter_by_dict(df: DataFrameT, filter_dict: Optional[dict] = None, engine: Union[EngineAbstract, str] = EngineAbstract.AUTO) -> DataFrameT: """ return df where rows match all values in filter_dict @@ -125,10 +149,17 @@ def filter_mask_by_dict(df: DataFrameT, filter_dict: Dict[str, Any]) -> SeriesT: """ from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError + from graphistry.compute.gfql.strictness import absent_column_matches + predicates: Dict[str, Tuple[str, ASTPredicate]] = {} concrete_filters: Dict[str, Tuple[str, Any]] = {} + absent_never_matches = False for col, val in filter_dict.items(): - resolved_col, resolved_val = resolve_filter_column(df, col, val) + resolved = resolve_filter_column_or_absent(df, col, val) + if resolved is None: + absent_never_matches = absent_never_matches or not absent_column_matches(val) + continue + resolved_col, resolved_val = resolved # Type checking for non-predicate values if not isinstance(resolved_val, ASTPredicate): @@ -190,7 +221,9 @@ def filter_mask_by_dict(df: DataFrameT, filter_dict: Dict[str, Any]) -> SeriesT: predicates[col] = (resolved_col, resolved_val) - hits = df[[]].assign(x=True).x + hits = df[[]].assign(x=False if absent_never_matches else True).x + if absent_never_matches: + return hits if concrete_filters: for original_col, (resolved_col, resolved_val) in concrete_filters.items(): if original_col.startswith("label__") and resolved_col == "labels" and isinstance(resolved_val, str): diff --git a/graphistry/compute/gfql/agg_types.py b/graphistry/compute/gfql/agg_types.py index add4e7408e..f58bd8e48f 100644 --- a/graphistry/compute/gfql/agg_types.py +++ b/graphistry/compute/gfql/agg_types.py @@ -34,8 +34,28 @@ DELIBERATE GFQL EXTENSION, not an oversight: ``sum``/``avg`` over BOOLEAN is a type error in Neo4j ("expected Float, Integer or Duration but was Boolean") but is accepted here on every engine, because summing an indicator column is idiomatic in the dataframe surface GFQL also -serves and both engines already agreed on it. It is recorded here so the divergence is a choice -with a reason rather than an accident. +serves and both engines already agreed on it. It is a strict SUPERSET -- no Cypher-valid query +changes meaning -- so the only cost is documenting it, which the aggregates docs now do. + +THE BOOLEAN RETURN-TYPE CONTRACT (adopted 2026-07-28; values AND dtypes, on every engine):: + + sum(BOOLEAN) -> INTEGER (int64) count of true, nulls skipped; 0 over zero non-null + avg(BOOLEAN) -> FLOAT (float64) true_count / non_null_count; NULL over zero non-null + min(BOOLEAN) -> BOOLEAN ordering false < true; NULL over zero non-null + max(BOOLEAN) -> BOOLEAN ordering false < true; NULL over zero non-null + count(BOOLEAN) -> INTEGER (int64) non-null count + +``min``/``max`` are stated as ORDERING, not as a logical fold. ``min == AND`` / ``max == OR`` is a +DERIVATION from ``false < true`` and it gets the empty case backwards: the conventional identity of +AND over zero elements is ``true`` and of OR over zero elements is ``false``, but every engine here +answers NULL -- the same answer ``ORDER BY`` already gives, and the same answer ``min``/``max`` give +over any other empty input. + +``sum -> 0`` over zero rows is CONFORMANCE, not a compromise: Cypher's ``sum()`` returns 0 where +SQL's returns NULL, and Cypher's ``avg()`` returns null; the engines here already match Cypher on +both. Pinning the DTYPES is what was still missing -- polars answered ``sum(BOOLEAN)`` and every +``count()`` with ``UInt32`` while pandas/cuDF answered ``int64``, so the values agreed and the +return types did not. Each engine classifies its OWN dtypes (a pandas dtype and a polars ``DataType`` are not comparable) and then funnels into the one raiser below, so the diagnostic text, the error class @@ -73,6 +93,60 @@ {"collect", "collect_distinct"} ) +#: Aggregates whose Cypher return type is INTEGER for EVERY input type. +CYPHER_INTEGER_RESULT_AGGREGATIONS: Final[FrozenSet[str]] = frozenset( + {"count", "count_distinct"} +) + + +def agg_result_is_integer(func: str, input_is_boolean: bool) -> bool: + """True when this aggregate's return type is INTEGER (int64) on this input. + + ``count``/``count_distinct`` are INTEGER over ANY input. ``sum`` is INTEGER over BOOLEAN -- + the documented extension counts the true values, so its result is a count, not a boolean. + ``avg`` stays FLOAT and ``min``/``max`` stay BOOLEAN, so neither is retyped here. + """ + if func in CYPHER_INTEGER_RESULT_AGGREGATIONS: + return True + return func == "sum" and input_is_boolean + + +def polars_agg_result_cast(func: str, input_dtype: "Optional[pl.DataType]") -> "Optional[pl.DataType]": + """The dtype polars' own aggregate kernel does NOT produce, or ``None`` when it conforms. + + Polars answers EVERY ``count()`` with ``UInt32`` and ``sum()`` over ``Boolean`` with ``UInt32``, + where pandas and cuDF answer ``int64`` -- the values agree and the return types do not, which is + the divergence class the aggregate type contract exists to close. Every OTHER numeric input + already sums to ``Int64``/``Float64``/``Duration`` on polars, so the ``sum`` half of this cast + can only fire on a boolean column; the ``count`` half is input-independent on both sides. + """ + import polars as pl + + is_boolean = input_dtype is not None and input_dtype == pl.Boolean + return pl.Int64 if agg_result_is_integer(func, is_boolean) else None + + +def polars_conform_agg_dtype(expr: "pl.Expr", func: str, input_dtype: "Optional[pl.DataType]", + alias: str) -> "pl.Expr": + """Land a polars aggregate on its CONTRACT dtype rather than on its kernel dtype.""" + import polars as pl + + target = polars_agg_result_cast(func, input_dtype) + if target is None: + return expr.alias(alias) + if func == "sum" and input_dtype == pl.Boolean: + expr = expr.fill_null(0) + return expr.cast(target).alias(alias) # hygiene-ok: explicit-cast -- polars dtype conversion + + +def polars_all_null_agg_literal(func: str, alias: str) -> "pl.Expr": + """Cypher's all-null answer as a TYPED literal: a bare ``pl.lit(0)`` is ``Int32``, which + neither pandas nor cuDF ever produces for a ``sum``.""" + import polars as pl + + value = numeric_agg_all_null_value(func) + return pl.lit(value, dtype=pl.Int64 if value is not None else None).alias(alias) + def _describe_agg_input(column: str, alias: Optional[str]) -> str: """How to point the user at the offending value in THEIR query text. @@ -121,6 +195,43 @@ def numeric_agg_all_null_value(func: str) -> Optional[int]: return 0 if func == "sum" else None +#: Integer widths a pandas/cuDF aggregate may land on that the INTEGER contract widens to int64. +_NARROW_INTEGER_DTYPES: Final[FrozenSet[str]] = frozenset( + {"int8", "int16", "int32", "uint8", "uint16", "uint32"} +) + + +def pandas_conform_agg_dtype(result: "SeriesT", func: str, input_is_boolean: bool) -> "SeriesT": + """Widen a pandas/cuDF aggregate whose kernel answered narrower than the INTEGER contract. + + cuDF's grouped ``nunique`` answers ``int32`` where pandas answers ``int64`` -- the same value + behind a different return type, on an aggregate Cypher declares INTEGER. Only the narrow + integer widths are eligible: ``int64``/``Int64`` are already the contract, and a float, boolean + or object result must never be retyped by this. + """ + if not agg_result_is_integer(func, input_is_boolean): + return result + if str(getattr(result, "dtype", "")).lower() not in _NARROW_INTEGER_DTYPES: + return result + return result.astype("int64") # hygiene-ok: explicit-cast -- dataframe dtype conversion + + +def pandas_agg_kernel_null_fill(func: str, series: "SeriesT") -> Optional[int]: + """The value a pandas/cuDF aggregate kernel's NULL answer must be repaired to, else ``None``. + + Cypher's ``sum()`` never returns null -- 0 is its zero-row answer -- but cuDF's grouped ``sum`` + over a group with no non-null values answers ````, on boolean AND on ``Int64``/``float64``, + where pandas answers 0. The two engines therefore disagreed on a VALUE, not merely a dtype, on + exactly the all-null row. Applied to the kernel's OUTPUT, so it repairs the per-group answer + that :func:`numeric_agg_all_null_value` (a whole-column pre-substitution) cannot see. + """ + if func != "sum": + return None + if str(getattr(series, "dtype", "")).lower() == "object": + return None # untyped kernel answer; the object-bool retype already owns this column + return 0 + + def pandas_dtype_is_numeric_for_agg(series: "SeriesT") -> bool: """True when the pandas/cuDF dtype ITSELF proves the column is a valid sum/avg input. diff --git a/graphistry/compute/gfql/call/executor.py b/graphistry/compute/gfql/call/executor.py index f0300de793..40aae41d69 100644 --- a/graphistry/compute/gfql/call/executor.py +++ b/graphistry/compute/gfql/call/executor.py @@ -16,7 +16,7 @@ execute_row_pipeline_call, is_row_pipeline_call, ) -from graphistry.compute.exceptions import ErrorCode, GFQLTypeError +from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError, GFQLTypeError from graphistry.compute.engine_coercion import ensure_engine_match from graphistry.compute.gfql.policy import PolicyContext, PolicyException from graphistry.compute.gfql.policy.stats import extract_graph_stats @@ -304,6 +304,8 @@ def execute_call(g: Plottable, function: str, params: Dict[str, Any], engine: En ) from error if isinstance(error, GFQLTypeError): raise error + if isinstance(error, GFQLSchemaError): + raise error # absent-name verdicts keep their own E301 taxonomy (#1916) if isinstance(error, NotImplementedError) and ( engine in (Engine.POLARS, Engine.POLARS_GPU) or is_row_pipeline_call(function) ): diff --git a/graphistry/compute/gfql/cypher/ast.py b/graphistry/compute/gfql/cypher/ast.py index 516357ba1a..9551238c18 100644 --- a/graphistry/compute/gfql/cypher/ast.py +++ b/graphistry/compute/gfql/cypher/ast.py @@ -327,6 +327,7 @@ class CypherQuery: reentry_unwinds: Tuple[UnwindClause, ...] = () graph_bindings: Tuple[GraphBinding, ...] = () use: Optional[UseClause] = None + return_is_reentry_carry: bool = False @property def match(self) -> Optional[MatchClause]: diff --git a/graphistry/compute/gfql/cypher/lowering.py b/graphistry/compute/gfql/cypher/lowering.py index f689f960d9..88ab4f3ff7 100644 --- a/graphistry/compute/gfql/cypher/lowering.py +++ b/graphistry/compute/gfql/cypher/lowering.py @@ -155,7 +155,6 @@ fold_temporal_constructor_ast, rewrite_temporal_constructors_in_expr, ) -from graphistry.compute.gfql.row.entity_props import LABEL_FLAG_PREFIX from graphistry.compute.gfql.same_path_types import ( EDGE_IDENTITY_COLUMN, NODE_IDENTITY_COLUMN, @@ -2593,6 +2592,22 @@ def _binds_one_route_per_pair_undirected(clause: MatchClause) -> bool: return False +def _sole_leading_optional_match(query: CypherQuery) -> bool: + """One OPTIONAL MATCH with nothing bound before it: no row can go unmatched. + + Over the single empty incoming row, such a clause either matches (and is a plain + MATCH, so binding rows are sound) or matches nothing (and the empty-result-row + null extension emits the one null row, without ever consulting binding rows).""" + return ( + len(query.matches) == 1 + and query.matches[0].optional + and not query.reentry_matches + and not query.with_stages + and not query.unwinds + and query.call is None + ) + + def _forces_relationship_multiplicity_projection_bindings( query: CypherQuery, *, @@ -2600,6 +2615,7 @@ def _forces_relationship_multiplicity_projection_bindings( relationship_count: int, items: Sequence[ReturnItem], order_by: Optional[OrderByClause], + bag_preserving_whole_row_aliases: AbstractSet[str] = frozenset(), ) -> bool: """Non-aggregate projections over relationship patterns run on binding rows: the per-alias node table collapses row multiplicity (bag semantics). @@ -2610,7 +2626,7 @@ def _forces_relationship_multiplicity_projection_bindings( conservative source-table path; variable-length arms are excluded.""" if relationship_count <= 0 or not alias_targets: return False - if any(clause.optional for clause in query.matches): + if any(clause.optional for clause in query.matches) and not _sole_leading_optional_match(query): return False if not all(isinstance(target, (ASTNode, ASTEdge)) for target in alias_targets.values()): return False @@ -2623,6 +2639,10 @@ def _forces_relationship_multiplicity_projection_bindings( referenced_aliases: Set[str] = set() for text in texts: stripped = text.strip() + if stripped in bag_preserving_whole_row_aliases and isinstance(alias_targets.get(stripped), ASTNode): + referenced_aliases.add(stripped) + saw_node_prop_ref = True + continue if stripped == "*" or stripped in alias_targets: return False tokens = { @@ -2640,42 +2660,25 @@ def _forces_relationship_multiplicity_projection_bindings( saw_node_prop_ref = True if not saw_node_prop_ref: return False - if _seeded_typed_hop_reduction_is_value_correct( - query, alias_targets=alias_targets, referenced_aliases=referenced_aliases - ): - return False return True -def _seeded_typed_hop_reduction_is_value_correct( +def _bag_preserving_whole_row_aliases( query: CypherQuery, *, - alias_targets: Mapping[str, ASTObject], - referenced_aliases: AbstractSet[str], -) -> bool: - """A selectively-seeded single-hop pattern projecting only destination props, which - the seeded typed-hop fast path already answers without binding rows.""" - if len(query.matches) != 1 or len(query.matches[0].patterns) != 1: - return False - pattern = query.matches[0].patterns[0] - if not ( - len(pattern) == 3 - and isinstance(pattern[0], NodePattern) - and isinstance(pattern[1], RelationshipPattern) - and pattern[1].min_hops is None - and pattern[1].max_hops is None - and not getattr(pattern[1], "to_fixed_point", False) - and isinstance(pattern[2], NodePattern) + plan: "_ProjectionPlan", +) -> AbstractSet[str]: + """Return whole-row aliases that may use relationship binding rows.""" + if query.return_.distinct or query.return_is_reentry_carry: + return frozenset() + if any( + _is_variable_length_relationship_pattern(element) + for clause in query.matches + for element in _match_pattern_elements(clause) + if isinstance(element, RelationshipPattern) ): - return False - seed_alias = pattern[0].variable - dest_alias = pattern[2].variable - seed_target = alias_targets.get(seed_alias) if seed_alias is not None else None - seed_filter = getattr(seed_target, "filter_dict", None) - has_selective_seed = seed_filter is not None and any( - not str(key).startswith(LABEL_FLAG_PREFIX) for key in seed_filter - ) - return has_selective_seed and dest_alias is not None and referenced_aliases <= {dest_alias} + return frozenset() + return frozenset(plan.whole_row_sources.values()) def _is_pure_count_star_shortcircuit( @@ -4573,13 +4576,13 @@ def _lower_projection_chain( merged_match = _merged_match_clause(query) force_multiplicity_bindings = ( plan.all_source_aliases is None - and not plan.whole_row_output_names and _forces_relationship_multiplicity_projection_bindings( query, alias_targets=alias_targets, relationship_count=_match_relationship_count(merged_match) if merged_match is not None else 0, items=query.return_.items, order_by=query.order_by, + bag_preserving_whole_row_aliases=_bag_preserving_whole_row_aliases(query, plan=plan), ) ) allowed_match_aliases = ({plan.source_alias} | plan.all_source_aliases | binding_row_aliases) if plan.all_source_aliases is not None else binding_row_aliases @@ -6429,14 +6432,12 @@ def _reject_with_rebind_onto_live_alias(query: CypherQuery) -> None: continue for item in clause.items: source = item.expression.text.strip() - source_kind = pattern_aliases.get(source) - if source_kind is None: + if source not in pattern_aliases: # Not a bare entity alias: a scalar column, which openCypher lets shadow freely. continue if item.alias is None or item.alias == source: continue - if pattern_aliases.get(item.alias) != source_kind: - # Cross-kind rebinds are the binder's; a fresh name already declines downstream. + if item.alias not in pattern_aliases: continue raise GFQLValidationError( ErrorCode.E108, @@ -6871,6 +6872,11 @@ def lower_match_query( ) ) continue + if _is_zoned_iso_temporal_comparison(predicate): + zoned_row_expr = _row_where_predicate_text(predicate) + if zoned_row_expr is not None: + row_where_predicates.append(zoned_row_expr) + continue _apply_literal_where( alias_targets, left=cast(PropertyRef, predicate.left), @@ -6927,6 +6933,23 @@ def _render_row_where_operand_text(value: Union[PropertyRef, CypherLiteral]) -> return str(value) +_ZONED_ISO_TEMPORAL_TEXT_RE = re.compile( + r"^(?:\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?|\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?)" + r"(?:Z|[+-]\d{2}:?\d{2})$" +) + + +def _is_zoned_iso_temporal_comparison(predicate: WherePredicate) -> bool: + """Comparison against tz-suffixed ISO temporal TEXT: keep it a ``where_rows`` + residual. Pushed down, the raw pandas compare raises on a naive datetime + column and its equality silently matches zero rows; the row pipeline's + temporal path compares instants on both column shapes.""" + if predicate.op not in {"==", "!=", "<>", "<", "<=", ">", ">="}: + return False + right = predicate.right + return isinstance(right, str) and _ZONED_ISO_TEMPORAL_TEXT_RE.match(right) is not None + + def _row_where_predicate_text(predicate: WherePredicate) -> Optional[str]: if isinstance(predicate.left, LabelRef): return None @@ -8352,9 +8375,8 @@ def _connected_join_pushable_value( Pushdown is an optimization: anything not exactly representable must stay a `where_rows` residual rather than push a filter that means something else. - - `None` is unrepresentable: `_filter_dict_to_json` drops null-valued entries, so a - pushed `nick = null` would vanish on the executor's serialization round-trip and - silently return unfiltered rows. + - `None` pushes down to a `filter_dict` equality, not to the three-valued-logic + comparison Cypher specifies for `nick = null`; the residual owns that meaning. - Ordering/inequality ops lower to `NumericASTPredicate`, which admits only int/float (`bool` is an `int` subclass but is not a numeric column predicate), so pushing a string or bool would raise where the residual answers correctly. @@ -8382,6 +8404,9 @@ def _connected_join_pushable_value( # The 64-bit literal guard lives on the row-expr path, so pushing an out-of-range # int would evade it and reach pandas, which overflows with a raw OverflowError. return False + if isinstance(resolved, str) and _ZONED_ISO_TEMPORAL_TEXT_RE.match(resolved) is not None: + # tz-suffixed ISO temporal text string-compares wrongly when pushed; the residual compares instants. + return False if op in _CONNECTED_JOIN_STRING_OPS: if not isinstance(resolved, str): return False @@ -8968,6 +8993,11 @@ def _apply_where_to_ops( ) ) continue + if _is_zoned_iso_temporal_comparison(predicate): + zoned_row_expr = _row_where_predicate_text(predicate) + if zoned_row_expr is not None: + row_expr_filters.append(ExpressionText(text=zoned_row_expr, span=predicate.span)) + continue _apply_literal_where( alias_targets, left=cast(PropertyRef, predicate.left), @@ -9379,9 +9409,10 @@ def compile_cypher_query( output_names = _cypher_return_output_names(branch.return_) if branch_output_names is None: branch_output_names = output_names - elif output_names != branch_output_names: + elif sorted(output_names) != sorted(branch_output_names): + # Same names in a different ORDER align by name at execution; only a different multiset errors. raise _unsupported( - "Cypher UNION branches must project the same output names in the same order", + "Cypher UNION branches must project the same output names", field="union", value={"expected": branch_output_names, "actual": output_names}, line=branch.return_.span.line, diff --git a/graphistry/compute/gfql/cypher/parser.py b/graphistry/compute/gfql/cypher/parser.py index 1135eb646c..eda183d787 100644 --- a/graphistry/compute/gfql/cypher/parser.py +++ b/graphistry/compute/gfql/cypher/parser.py @@ -151,7 +151,7 @@ variable: NAME properties: "{" [property_entry ("," property_entry)*] "}" -property_entry: NAME ":" expr +property_entry: PROP_NAME ":" expr // Unified: every WHERE parses as a generic boolean ``expr`` (so LALR(1) accepts // OR/XOR/NOT/parenthesized clauses, no Earley). ``generic_where_clause`` lifts the @@ -205,8 +205,11 @@ skip_clause: "SKIP"i expr limit_clause: "LIMIT"i expr -qualified_name: NAME ("." NAME)* -property_ref.2: NAME "." NAME +// PROP_NAME (dot/map-key contexts only) admits non-reserved keywords as property +// names (n.when, n.order, {when: 1}) — openCypher property keys are unreserved; +// the contextual lexer only expects it where NAME's keyword exclusions cannot apply. +qualified_name: NAME ("." PROP_NAME)* +property_ref.2: NAME "." PROP_NAME unwind_expr: expr order_expr: expr @@ -261,7 +264,7 @@ | postfix_composite ?postfix_composite: primary_composite | postfix "[" subscript_key "]" -> subscript - | postfix_composite "." NAME -> property_access + | postfix_composite "." PROP_NAME -> property_access ?primary_composite: parameter | literal @@ -362,6 +365,7 @@ MINUS: /-(?!-)/ NAME: /(?!(?i:MATCH|RETURN|WITH|ORDER|BY|SKIP|LIMIT|UNWIND|WHERE|AS|ASC|ASCENDING|DESC|DESCENDING|AND|OR|XOR|NOT|IN|IS|NULL|TRUE|FALSE|CONTAINS|STARTS|ENDS|ANY|ALL|NONE|SINGLE|CASE|WHEN|THEN|ELSE|END)\b)[A-Za-z_][A-Za-z0-9_]*/ MAP_KEY_NAME: /[A-Za-z_][A-Za-z0-9_]*/ +PROP_NAME: /[A-Za-z_][A-Za-z0-9_]*/ NUMBER: /[+-]?(?:0[xX][0-9A-Fa-f]+|0[oO][0-7]+|(?:\d+\.\d+(?:[eE][+-]?\d+)?|\.\d+(?:[eE][+-]?\d+)?|\d+(?:[eE][+-]?\d+)?))/ INT: /[0-9]+/ STRING : /'(?:\\.|[^'\\])*'|"(?:\\.|[^"\\])*"/ diff --git a/graphistry/compute/gfql/cypher/projection_columns.py b/graphistry/compute/gfql/cypher/projection_columns.py new file mode 100644 index 0000000000..93656c94f0 --- /dev/null +++ b/graphistry/compute/gfql/cypher/projection_columns.py @@ -0,0 +1,23 @@ +from __future__ import annotations + +import typing + + +def alias_field_sources( + columns: typing.Iterable[str], + alias: str, +) -> typing.Optional[typing.Mapping[str, str]]: + column_names = tuple(str(column) for column in columns) + prefix = f"{alias}." + sources = { + column[len(prefix):]: column + for column in column_names + if column.startswith(prefix) + } + if sources: + if alias in column_names: + sources.setdefault(alias, alias) + return sources if alias in sources else None + if alias in column_names: + return {column: column for column in column_names} + return None diff --git a/graphistry/compute/gfql/cypher/reentry/compiletime.py b/graphistry/compute/gfql/cypher/reentry/compiletime.py index 4761302f03..0515a56410 100644 --- a/graphistry/compute/gfql/cypher/reentry/compiletime.py +++ b/graphistry/compute/gfql/cypher/reentry/compiletime.py @@ -332,6 +332,7 @@ def _compile_bounded_reentry_query( limit=prefix_stage.limit, trailing_semicolon=False, reentry_unwinds=(), + return_is_reentry_carry=True, ) prefix_compiled = compile_cypher_query(prefix_query, params=params) if not isinstance(prefix_compiled, CompiledCypherQuery): diff --git a/graphistry/compute/gfql/cypher/reentry/execution.py b/graphistry/compute/gfql/cypher/reentry/execution.py index daadd506a0..a704c85264 100644 --- a/graphistry/compute/gfql/cypher/reentry/execution.py +++ b/graphistry/compute/gfql/cypher/reentry/execution.py @@ -3,7 +3,7 @@ from __future__ import annotations from dataclasses import dataclass -from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple, Union, cast +from typing import Any, Dict, List, Mapping, Optional, Sequence, Set, Tuple, Union, cast from graphistry.Engine import ( EngineAbstract, @@ -259,8 +259,15 @@ def _optional_reentry_carried_null_rows( ) if not carried_columns: return None + # OPTIONAL MATCH cannot unbind an alias the prefix bound; the keys stay the scalars. + copied_columns = carried_columns + _carried_entity_columns( + prefix_df, + result_columns=result_columns, + reentry_plan=reentry_plan, + exclude=set(carried_columns), + ) - prefix_records = _records_for_columns(prefix_df, carried_columns) + prefix_records = _records_for_columns(prefix_df, copied_columns) prefix_keys = [_optional_reentry_key(record, carried_columns) for record in prefix_records] if len(set(prefix_keys)) != len(prefix_keys): return None @@ -280,12 +287,33 @@ def _optional_reentry_carried_null_rows( fill_rows: List[CypherFillRow] = [] for record in missing_records: row = dict(null_row) - for col in carried_columns: + for col in copied_columns: row[col] = record[col] fill_rows.append(row) return fill_rows +def _carried_entity_columns( + prefix_df: DataFrameT, + *, + result_columns: Set[str], + reentry_plan: ReentryPlan, + exclude: Set[str], +) -> Tuple[str, ...]: + """Flat ``alias.prop`` columns of the carried whole-entity aliases, in prefix order.""" + prefixes = tuple(f"{alias.output_name}." for alias in reentry_plan.aliases) + if not prefixes: + return () + return tuple( + str(col) + for col in prefix_df.columns + if isinstance(col, str) + and col.startswith(prefixes) + and col in result_columns + and col not in exclude + ) + + def _optional_reentry_key( record: Mapping[str, CypherFillValue], columns: Tuple[str, ...] ) -> Tuple[CypherFillValue, ...]: @@ -310,6 +338,46 @@ def _optional_reentry_key_value(value: CypherFillValue) -> CypherFillValue: return value +def restrict_connected_join_rows_to_reentry_seed( + joined_rows: DataFrameT, + *, + start_nodes: DataFrameT, + reentry_alias: Optional[str], + node_col: str, +) -> DataFrameT: + """Keep only comma-pattern join rows whose reentry alias is a carried seed id. + + The connected comma-pattern join re-matches every arm from the whole graph, so a + ``WITH p MATCH (p)-..., (p)-...`` suffix must be narrowed back to the carried ``p`` + rows here; seeds are non-null by construction (``aligned_reentry_rows``).""" + if reentry_alias is None or node_col not in start_nodes.columns: + raise reentry_validation_error( + "Cypher MATCH after WITH could not recover the carried seed ids for the connected comma-pattern join", + value=reentry_alias, + suggestion=REENTRY_WHOLE_ROW_SUGGESTION, + ) + alias_col = next( + ( + col + for col in (reentry_alias, f"{reentry_alias}.{node_col}") + if col in joined_rows.columns + ), + None, + ) + if alias_col is None: + raise reentry_validation_error( + "Cypher MATCH after WITH could not recover the carried alias binding column from the connected comma-pattern join", + value=reentry_alias, + suggestion=REENTRY_WHOLE_ROW_SUGGESTION, + ) + seed_ids = start_nodes[node_col] + if _is_polars_df(joined_rows): + import polars as pl + seed_values = seed_ids.to_list() if hasattr(seed_ids, "to_list") else list(seed_ids) + return joined_rows.filter(pl.col(alias_col).is_in(seed_values)) # type: ignore[attr-defined] + return joined_rows[joined_rows[alias_col].isin(seed_ids)] + + def compiled_query_reentry_state( base_graph: Plottable, plan: ReentryPlan, diff --git a/graphistry/compute/gfql/cypher/result_postprocess.py b/graphistry/compute/gfql/cypher/result_postprocess.py index 02e3c9b3a8..df0fd68094 100644 --- a/graphistry/compute/gfql/cypher/result_postprocess.py +++ b/graphistry/compute/gfql/cypher/result_postprocess.py @@ -8,6 +8,8 @@ from graphistry.Plottable import Plottable from graphistry.compute.typing import DataFrameT, SeriesT from graphistry.Engine import is_polars_df +from graphistry.compute.gfql.cypher.projection_columns import alias_field_sources +from graphistry.compute.gfql.identifiers import shadow_restore_column from graphistry.compute.gfql.series_str_compat import is_non_textual_scalar_dtype from .lowering import ResultProjectionColumn, ResultProjectionPlan @@ -263,20 +265,15 @@ def _projection_alias_rows( *, alias: str, ) -> Optional[DataFrameT]: - prefix = f"{alias}." - alias_columns = [column for column in rows_df.columns if str(column).startswith(prefix)] - if alias_columns: - alias_rows = cast( - DataFrameT, - rows_df[alias_columns].rename(columns={column: str(column)[len(prefix):] for column in alias_columns}), - ) - if alias in rows_df.columns and alias not in alias_rows.columns: - alias_rows = cast(DataFrameT, alias_rows.assign(**{alias: rows_df[alias]})) - if alias in alias_rows.columns: - return alias_rows - if alias in rows_df.columns: + field_sources = alias_field_sources(rows_df.columns, alias) + if field_sources is None: + return None + if all(field == source for field, source in field_sources.items()): return rows_df - return None + source_fields = list(field_sources.values()) + return rows_df[source_fields].rename( + columns={source: field for field, source in field_sources.items()} + ) def apply_result_projection( @@ -370,7 +367,12 @@ def _apply_result_projection_pandas( output_columns.append(column.output_name) if column.kind == "property": property_rows_df = alias_rows_df - if ( + self_shadow_col = shadow_restore_column(projection.alias) + if column.source_name == projection.alias and self_shadow_col in rows_df.columns: + # 'alias.alias': the plain column is the marker; rows() re-keyed the user values + column = replace(column, source_name=self_shadow_col) + property_rows_df = rows_df + elif ( column.source_name is not None and column.source_name not in alias_rows_df.columns and column.source_name in rows_df.columns diff --git a/graphistry/compute/gfql/df_executor.py b/graphistry/compute/gfql/df_executor.py index 47b4755b4c..ccb68c2350 100644 --- a/graphistry/compute/gfql/df_executor.py +++ b/graphistry/compute/gfql/df_executor.py @@ -196,14 +196,18 @@ def _apply_mask(alias: str, frame: DataFrameT, mask: Any) -> bool: elif clause.op in INEQ_WHERE_OPS: left_vals = left_frame[left_col] right_vals = right_frame[right_col] - left_min, left_max = left_vals.min(), left_vals.max() - right_min, right_max = right_vals.min(), right_vals.max() - masks = { - "<": (left_vals < right_max, right_vals > left_min), - "<=": (left_vals <= right_max, right_vals >= left_min), - ">": (left_vals > right_min, right_vals < left_max), - ">=": (left_vals >= right_min, right_vals <= left_max), - } + try: + left_min, left_max = left_vals.min(), left_vals.max() + right_min, right_max = right_vals.min(), right_vals.max() + masks = { + "<": (left_vals < right_max, right_vals > left_min), + "<=": (left_vals <= right_max, right_vals >= left_min), + ">": (left_vals > right_min, right_vals < left_max), + ">=": (left_vals >= right_min, right_vals <= left_max), + } + except TypeError: + # Incomparable dtypes: this bounds prune is optional — the real WHERE filter answers. + continue left_mask, right_mask = masks[clause.op] changed |= _apply_mask(left_alias, left_frame, left_mask) changed |= _apply_mask(right_alias, right_frame, right_mask) diff --git a/graphistry/compute/gfql/expr_parser.py b/graphistry/compute/gfql/expr_parser.py index 51357a1130..48fe178473 100644 --- a/graphistry/compute/gfql/expr_parser.py +++ b/graphistry/compute/gfql/expr_parser.py @@ -215,7 +215,7 @@ def __init__(self, message: str, *, line: Optional[int] = None, column: Optional ?postfix: primary | postfix "[" subscript_key "]" -> subscript - | postfix "." NAME -> property_access + | postfix "." PROP_NAME -> property_access ?subscript_key: expr -> subscript_index | expr ".." expr -> subscript_slice_between @@ -249,7 +249,10 @@ def __init__(self, message: str, *, line: Optional[int] = None, column: Optional distinct_func_args: "DISTINCT"i func_arg ?func_arg: expr | "*" -> star_arg -identifier: NAME ("." NAME)* +// PROP_NAME (dot contexts only) admits non-reserved keywords as property names +// (n.when, n.order) — openCypher property keys are unreserved; only after "." can +// no keyword ambiguity arise, so NAME's exclusions need not apply there. +identifier: NAME ("." PROP_NAME)* case_expr: searched_case_expr | simple_case_expr @@ -278,6 +281,7 @@ def __init__(self, message: str, *, line: Optional[int] = None, column: Optional COMP_OP: __GFQL_COMPARISON_GRAMMAR_ALTS__ MINUS: /-(?!-)/ NAME: /(?!(?i:AND|OR|XOR|NOT|IN|IS|NULL|CASE|WHEN|THEN|ELSE|END|CONTAINS|STARTS|WITH|ENDS|ANY|ALL|NONE|SINGLE)\b)[A-Za-z_][A-Za-z0-9_]*/ +PROP_NAME: /[A-Za-z_][A-Za-z0-9_]*/ MAP_KEY_NAME: /[A-Za-z_][A-Za-z0-9_]*/ NUMBER: /[+-]?(?:0[xX][0-9A-Fa-f]+|0[oO][0-7]+|(?:\d+\.\d+(?:[eE][+-]?\d+)?|\.\d+(?:[eE][+-]?\d+)?|\d+(?:[eE][+-]?\d+)?))/ STRING : /'(?:\\.|[^'\\])*'|"(?:\\.|[^"\\])*"/ @@ -416,7 +420,7 @@ def string_lit(self, items: Sequence[Any]) -> Literal: return Literal(_parse_string_token(str(items[0]))) def identifier(self, items: Sequence[Any]) -> Identifier: - names = [str(i) for i in items if _is_token(i) and str(getattr(i, "type", "")) == "NAME"] + names = [str(i) for i in items if _is_token(i) and str(getattr(i, "type", "")) in ("NAME", "PROP_NAME")] if len(names) == 0: raise GFQLExprParseError("Invalid identifier") return Identifier(".".join(names)) @@ -666,7 +670,7 @@ def property_access(self, items: Sequence[Any]) -> PropertyAccessExpr: if len(stripped) != 1: raise GFQLExprParseError("Invalid property access") value = cast(ExprNode, stripped[0]) - names = [str(i) for i in items if _is_token(i) and str(getattr(i, "type", "")) == "NAME"] + names = [str(i) for i in items if _is_token(i) and str(getattr(i, "type", "")) in ("NAME", "PROP_NAME")] if len(names) == 0: raise GFQLExprParseError("Invalid property access") return PropertyAccessExpr(value=value, property=names[-1]) diff --git a/graphistry/compute/gfql/frontends/cypher/binder.py b/graphistry/compute/gfql/frontends/cypher/binder.py index 0bfecf0afb..c19320d26e 100644 --- a/graphistry/compute/gfql/frontends/cypher/binder.py +++ b/graphistry/compute/gfql/frontends/cypher/binder.py @@ -708,7 +708,14 @@ def _validate_relationship_pattern_schema( scoped_columns = _catalog_edge_columns_for_types(state.catalog, relationship_pattern.types) columns = scoped_columns if scoped_columns is not None else edge_columns available_types = _catalog_edge_types(state.catalog) - if available_types: + declared_types = state.catalog.metadata.get("edge_types") + has_declared_type_catalog = isinstance( + declared_types, (list, tuple, set, frozenset) + ) + can_judge_types = ( + bool(available_types) or has_declared_type_catalog or "type" not in edge_columns + ) + if can_judge_types: for rel_type in relationship_pattern.types: if rel_type not in available_types: raise _missing_relationship_type_in_schema_error( diff --git a/graphistry/compute/gfql/identifiers.py b/graphistry/compute/gfql/identifiers.py index 662a47cf61..fc89ecc3d9 100644 --- a/graphistry/compute/gfql/identifiers.py +++ b/graphistry/compute/gfql/identifiers.py @@ -55,6 +55,11 @@ def identifier_tokens(text: str) -> Set[str]: #: Stable per-edge identity: openCypher TRAIL semantics bind a relationship at most once per path. TRAIL_EDGE_IDENT_COL: Final[str] = '__gfql_edge_ident__' + +def shadow_restore_column(alias: str) -> str: + """Row-table column carrying user values of a column the alias marker overwrote.""" + return f'__gfql_shadow_restore__{alias}__' + #: Prefix of the per-hop column recording WHICH relationship that hop bound. TRAIL_COLUMN_PREFIX: Final[str] = '__gfql_trail_' diff --git a/graphistry/compute/gfql/index/bindings.py b/graphistry/compute/gfql/index/bindings.py index 8102e86154..0d040597a0 100644 --- a/graphistry/compute/gfql/index/bindings.py +++ b/graphistry/compute/gfql/index/bindings.py @@ -117,12 +117,15 @@ def _filter_compatible(frame: DataFrameT, filter_dict: Optional[dict]) -> bool: from graphistry.compute.filter_by_dict import ( _is_numeric_dtype_safe, _is_string_dtype_safe, - resolve_filter_column, + resolve_filter_column_or_absent, ) try: for col, value in filter_dict.items(): - resolved, resolved_value = resolve_filter_column(frame, col, value) + resolved_pair = resolve_filter_column_or_absent(frame, col, value) + if resolved_pair is None: + return False # absent name; the canonical path applies the 3VL verdict (#1916) + resolved, resolved_value = resolved_pair series = ( frame.get_column(resolved) # type: ignore[operator] if "polars" in type(frame).__module__ diff --git a/graphistry/compute/gfql/lazy/engine/polars/chain.py b/graphistry/compute/gfql/lazy/engine/polars/chain.py index 42b789ea47..de683f940b 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/chain.py +++ b/graphistry/compute/gfql/lazy/engine/polars/chain.py @@ -14,6 +14,7 @@ # Runtime import (not TYPE_CHECKING): AggSpec is a pure typing Union of builtins (engine- # neutral wire type), and it keeps _GroupByParams introspectable (get_type_hints) at runtime. from graphistry.compute.gfql.call.support import AggSpec +from graphistry.compute.endpoint_utils import drop_null_endpoint_edges from graphistry.Plottable import Plottable from graphistry.compute.ast import ASTObject, ASTNode, ASTEdge @@ -1009,7 +1010,7 @@ def _plain_edge(op): ncol, scol, dcol = gf._node, gf._source, gf._destination assert ncol is not None and scol is not None and dcol is not None gf, restore = _align_edge_endpoints(gf, ncol, scol, dcol) - edges = gf._edges + edges = drop_null_endpoint_edges(gf._edges, scol, dcol) n_from, n_to = (n0, n2) if e1.direction != "reverse" else (n2, n0) all_ids = gf._nodes.select(pl.col(ncol)) diff --git a/graphistry/compute/gfql/lazy/engine/polars/hop_eager.py b/graphistry/compute/gfql/lazy/engine/polars/hop_eager.py index 795fa501c8..4a4c2c6b33 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/hop_eager.py +++ b/graphistry/compute/gfql/lazy/engine/polars/hop_eager.py @@ -13,6 +13,7 @@ from typing import TYPE_CHECKING, Any, Optional, Tuple from graphistry.Plottable import Plottable +from graphistry.compute.endpoint_utils import drop_null_endpoint_edges from graphistry.compute.util import generate_safe_column_name from .dtypes import endpoint_ids from .predicates import filter_by_dict_polars @@ -32,6 +33,19 @@ def _unsupported(**kwargs: Any) -> None: ) +def _dedup_output_node_rows( + out_nodes: "pl.DataFrame", out_edges: "pl.DataFrame", node_col: str +) -> "pl.DataFrame": + """One output node row per id, matching pandas' edge-guarded hop epilogue. + + The node output is a semi-join against the input table, which emits every + matching row, so a duplicated input id survives as two rows here where pandas + emits one. Guarded on ``out_edges`` exactly as pandas guards its drop_duplicates.""" + if out_edges.height == 0: + return out_nodes + return out_nodes.unique(subset=[node_col], keep="first", maintain_order=True) + + def ensure_nodes_polars(g: Plottable) -> Plottable: """Materialize a polars node table from edges when absent (native — avoids the pandas-idiom ``materialize_nodes`` path, which uses drop_duplicates/reset_index).""" @@ -98,20 +112,13 @@ def _keep_edges_with_both_endpoints_resolvable( edges_idx: "PolarsT", src: str, dst: str, node_dtype: "pl.DataType", resolvable_ids: "pl.Series", ) -> "PolarsT": - """`.implode()` makes the id series ONE membership collection; bare `is_in` is deprecated. - - A NULL endpoint resolves iff the id universe holds a NULL id: `is_in` answers NULL for a - NULL input (dropped by `filter`), where the pandas/cuDF `isin` this mirrors answers True. - """ + """Filter both endpoints against the non-null identity universe.""" import polars as pl universe = resolvable_ids.implode() - a_null_id_is_resolvable = resolvable_ids.null_count() > 0 def _resolvable(endpoint_col: str) -> "pl.Expr": - endpoint = pl.col(endpoint_col).cast(node_dtype) - member = endpoint.is_in(universe) - return (member | endpoint.is_null()) if a_null_id_is_resolvable else member + return pl.col(endpoint_col).cast(node_dtype).is_in(universe) return edges_idx.filter(_resolvable(src) & _resolvable(dst)) @@ -239,6 +246,7 @@ def hop_polars( # resolved_max_hops comes from the shared resolver above (None == run-to-closure). FROM, TO, NID, EID, edges_idx, synth_eid, node_dtype = _hop_setup_columns( edges, all_nodes, node_col, g._edge) + edges_idx = drop_null_endpoint_edges(edges_idx, src, dst) serves_single_bounded_hop = ( not to_fixed_point and resolved_max_hops == 1 @@ -316,6 +324,7 @@ def _idframe_lf(lf: "pl.LazyFrame", col: str) -> "pl.LazyFrame": needed_lf = pl.concat([needed_lf, endpoints_lf], how="vertical_relaxed").unique(subset=[NID]) out_nodes_lf = all_nodes.lazy().join(needed_lf.rename({NID: node_col}), on=node_col, how="semi") out_edges_c, out_nodes_c = collect_all([out_edges_lf, out_nodes_lf]) + out_nodes_c = _dedup_output_node_rows(out_nodes_c, out_edges_c, node_col) return g.nodes(out_nodes_c, node_col).edges(out_edges_c, src, dst) allowed_source = None @@ -360,12 +369,13 @@ def _idframe_lf(lf: "pl.LazyFrame", col: str) -> "pl.LazyFrame": empty_ids = all_nodes.select(pl.col(node_col).cast(node_dtype).alias(NID)).clear() - # Hop labeling — plain (non-min_hops) BFS only. pandas labels a node with the hop at - # which the ANTI-JOINED wavefront first discovers it (hop.py:581-603 over new_node_ids), i.e. - # its shortest-path distance; that is exactly `new_frontier` here. Seeds are already in - # `visited_nodes` after the first iteration, so a seed re-reached by a backtracking undirected - # walk stays UNLABELED (null) — that is the divergence. label_seeds writes hop 0 for - # seeds instead. Edges take the hop that first traversed them (hop.py:555-557), min-aggregated. + # Hop labeling — plain (non-min_hops) BFS only. Labels come from every DESTINATION of the + # hop (`cand`), first-wins against `label_seen_nodes` — NOT from `new_frontier`, which is + # anti-joined against `visited_nodes` and drives traversal only. That matches pandas + # (hop.py:540 new_node_ids = all TO ids), so under fwd/rev a seed re-entered at hop 1 IS + # labeled 1. Only undirected pre-seeds `label_seen_nodes` with the seeds, leaving a seed + # re-reached by backtracking UNLABELED (null); label_seeds writes hop 0 for seeds instead. + # Edges take the hop that first traversed them (hop.py:555-557), min-aggregated. track_node_hops = (label_node_hops is not None or label_seeds) and not min_hops_active node_hop_frames = [] # list[DataFrame[NID, NHOP]] label_seen_nodes = empty_ids # first-wins guard for node labels @@ -495,7 +505,10 @@ def _idframe_lf(lf: "pl.LazyFrame", col: str) -> "pl.LazyFrame": valid_edge_frames = [] for level in range(max_edge_hop, 0, -1): lvl = edge_rec.filter(pl.col(HOP) == level) - reaching = lvl.join(current_targets.rename({NID: TO}), on=TO, how="semi") + # An edge at >= min_hops ends a qualifying walk itself; only sub-min levels feed one. + level_is_goal = level >= min_hops # paired with hop.py, fixes #1944 + reaching = lvl if level_is_goal else lvl.join( + current_targets.rename({NID: TO}), on=TO, how="semi") valid_edge_frames.append(reaching.select(pl.col(EID))) current_targets = reaching.select(pl.col(FROM).alias(NID)).unique() valid_node = pl.concat([valid_node, current_targets], how="vertical_relaxed").unique(subset=[NID]) @@ -554,6 +567,7 @@ def _idframe_lf(lf: "pl.LazyFrame", col: str) -> "pl.LazyFrame": out_nodes = _min_hops_labeled_node_output(all_nodes, needed, reached_for_attrs, node_col, NID) else: out_nodes = all_nodes.join(needed.rename({NID: node_col}), on=node_col, how="semi") + out_nodes = _dedup_output_node_rows(out_nodes, out_edges, node_col) if track_node_hops and label_node_hops is not None: node_labels = ( diff --git a/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py b/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py index 68edc3b488..309d4c2e9a 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py +++ b/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py @@ -11,6 +11,7 @@ from typing import TYPE_CHECKING, Dict, List, Optional, Sequence from graphistry.utils.json import JSONVal from graphistry.compute.gfql.index.types import HopDirection +from graphistry.compute.typing import ArrayLike from graphistry.Plottable import Plottable @@ -107,6 +108,26 @@ def rows_binding_ops_polars( return _rewrap(g, lookup) +def _nodes_cover_keys(base_graph: Plottable, node_id: str, keys: ArrayLike) -> bool: + """True when every id in ``keys`` (the index's edge-endpoint keys) is present in the + node table — the precondition under which edge-derived membership equals the scan's + node-table-intersected answer. False (decline) on a missing/lazy/mistyped node table, + a null id, or any backend the keys cannot be compared against natively.""" + import polars as pl + import numpy as np + nodes = base_graph._nodes + if nodes is None or is_lazy(nodes) or node_id not in nodes.columns: + return False + try: + key_frame = pl.DataFrame({node_id: pl.Series(node_id, np.asarray(keys))}) + node_col = nodes.select(node_id) # type: ignore[union-attr] # eager polars frame, guarded above + if key_frame.get_column(node_id).null_count() > 0 or node_col.get_column(node_id).null_count() > 0: + return False + return key_frame.join(node_col, on=node_id, how="anti").height == 0 + except Exception: + return False + + def _pattern_alias_keys_polars( g: Plottable, binding_ops: Sequence[Dict[str, JSONVal]], alias: str, neq: Optional[Sequence[str]] = None ) -> Optional["pl.DataFrame"]: @@ -139,10 +160,13 @@ def _pattern_alias_keys_polars( # edge filters, no drop-self neq) -> participating nodes == "has an edge in this # direction" = CSR adjacency membership. Skips the O(E) chain_polars below. # Strict guard; anything richer (filters/neq/multi-hop) falls through unchanged. + # Repeated endpoint alias means SELF-LOOP, which adjacency membership cannot + # express, so that shape must not take this path. from graphistry.compute.gfql.index import get_index_policy if ( neq is None and get_index_policy(g) != "off" + and n0._name != n2._name and not n0.filter_dict and not n2.filter_dict and not edge_op.edge_match and edge_op.edge_query is None @@ -168,7 +192,17 @@ def _pattern_alias_keys_polars( if isinstance(_src, str) and isinstance(_dst, str): _eng = _Engine.POLARS_GPU if _active_target() == _ExecutionTarget.GPU else _Engine.POLARS _mk = adjacency_membership_keys(_reg, _mdir, base_graph._edges, (_src, _dst), _eng) - if _mk is not None: + # Both endpoints must be nodes, so BOTH directions' keys need covering. + _opp: HopDirection = "reverse" if _mdir == "forward" else "forward" + _cover = None if _mdir == "undirected" else adjacency_membership_keys( + _reg, _opp, base_graph._edges, (_src, _dst), _eng + ) + _needed = [k for k in (_mk, _cover) if k is not None] + if ( + _mk is not None + and (_mdir == "undirected" or _cover is not None) + and all(_nodes_cover_keys(base_graph, node_id, k) for k in _needed) + ): return pl.DataFrame({node_id: pl.Series(node_id, _np.asarray(_mk))}) if neq: # EXISTS { (n)--(m) WHERE m <> n } — for the single-edge shape, endpoint diff --git a/graphistry/compute/gfql/lazy/engine/polars/predicates.py b/graphistry/compute/gfql/lazy/engine/polars/predicates.py index ccf338f72b..c539922f15 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/predicates.py +++ b/graphistry/compute/gfql/lazy/engine/polars/predicates.py @@ -14,7 +14,8 @@ from graphistry.compute.predicates.ASTPredicate import ASTPredicate from graphistry.compute.predicates.str import Contains, Endswith, Fullmatch, Match, Startswith -from graphistry.compute.filter_by_dict import resolve_filter_column +from graphistry.compute.filter_by_dict import resolve_filter_column_or_absent +from graphistry.compute.gfql.strictness import absent_column_matches from .dtypes import is_numeric as _dtype_numeric, is_stringlike as _dtype_stringlike if TYPE_CHECKING: @@ -75,11 +76,86 @@ def _orders_boolean_column_against_number(op: object, val: object, dtype: "Optio return dtype == pl.Boolean +def _dtype_is_temporal(dtype: "Optional[pl.DataType]") -> bool: + import polars as pl + return dtype is not None and ( + isinstance(dtype, (pl.Datetime, pl.Duration)) or dtype == pl.Date or dtype == pl.Time + ) + + +def _parse_temporal_filter_scalar( + val: str, dtype: "pl.DataType" +) -> "Optional[Union[datetime.date, datetime.time, datetime.datetime, datetime.timedelta]]": + """The python temporal scalar a TEMPORAL column can compare ``val`` against, or None. + + Parses with pandas (``pd.Timestamp`` / ``pd.to_timedelta``) — the SAME parse pandas + comparison ops apply to a string operand, so the compared instant is + parity-equal by construction. SAFE subset only: a NAIVE Datetime column takes a + naive parse (tz-suffixed text and sub-microsecond precision decline — pandas + itself raises/zero-rows on the tz mix), Duration takes a ``to_timedelta`` parse, + Date/Time take exact ISO parses. None means not comparable, so the caller + raises the typed schema error.""" + import datetime as _dt + import pandas as pd + import polars as pl + try: + if isinstance(dtype, pl.Datetime): + if dtype.time_zone is not None: + return None + ts = pd.Timestamp(val) + if ts.tz is not None or ts.nanosecond != 0: + return None + return ts.to_pydatetime() + if isinstance(dtype, pl.Duration): + td = pd.to_timedelta(val) + if td.nanoseconds % 1000 != 0: + return None + return td.to_pytimedelta() + if dtype == pl.Date: + return _dt.date.fromisoformat(val) + if dtype == pl.Time: + return _dt.time.fromisoformat(val) + except (ValueError, TypeError): + return None + return None + + +def _raise_temporal_str_mismatch(col: str, dtype: "pl.DataType", val: str) -> None: + from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError + raise GFQLSchemaError( + ErrorCode.E302, + f'Type mismatch: column "{col}" is temporal ({dtype}) but filter value is a ' + f'string it cannot be compared to', + field=col, + value=val, + column_type=str(dtype), + suggestion='Use matching temporal text (e.g. a naive ISO datetime for a naive ' + 'datetime column) or a temporal value such as date(...)', + ) + + +def _temporal_str_cmp_expr( + col: str, + col_expr: "pl.Expr", + op: "Callable[[pl.Expr, pl.Expr], pl.Expr]", + val: str, + dtype: "pl.DataType", +) -> "pl.Expr": + """Temporal column vs string comparison: parse-and-compare in the SAFE subset, + typed GFQLSchemaError otherwise — never a raw polars error.""" + import polars as pl + parsed = _parse_temporal_filter_scalar(val, dtype) + if parsed is None: + _raise_temporal_str_mismatch(col, dtype, val) + return op(col_expr, pl.lit(parsed)) + + def _cmp_expr( col_expr: "pl.Expr", op: Callable[[Any, Any], Any], val: CmpValue, dtype: "Optional[pl.DataType]" = None, + col: str = "", ) -> "Optional[pl.Expr]": import datetime as _dt @@ -114,6 +190,10 @@ def _cmp_expr( if _orders_boolean_column_against_number(op, val, dtype): import polars as pl return pl.lit(False) + # Temporal column vs raw string raises InvalidOperationError at collect; parse-or-typed-error instead. + if isinstance(val, str) and _dtype_is_temporal(dtype) and op in _CMP_OPS: + assert dtype is not None + return _temporal_str_cmp_expr(col, col_expr, op, val, dtype) if op in _CMP_OPS: return op(col_expr, val) return None @@ -145,7 +225,7 @@ def predicate_to_expr(col: str, pred: ASTPredicate, dtype: "Optional[pl.DataType op = getattr(pred, "op", None) if op is not None and hasattr(pred, "val"): - expr = _cmp_expr(c, op, pred.val, dtype) + expr = _cmp_expr(c, op, pred.val, dtype, col) if expr is not None: return expr @@ -162,8 +242,8 @@ def predicate_to_expr(col: str, pred: ASTPredicate, dtype: "Optional[pl.DataType # returns None -> honest NIE (tz-aware DateTimeValue, TimeValue, raw datetime, mixed # bounds, non-Datetime dtype all decline this way — never a silent mismatch). inclusive = getattr(pred, "inclusive", True) - lo_expr = _cmp_expr(c, operator.ge if inclusive else operator.gt, lo, dtype) - hi_expr = _cmp_expr(c, operator.le if inclusive else operator.lt, hi, dtype) + lo_expr = _cmp_expr(c, operator.ge if inclusive else operator.gt, lo, dtype, col) + hi_expr = _cmp_expr(c, operator.le if inclusive else operator.lt, hi, dtype, col) if lo_expr is not None and hi_expr is not None: return lo_expr & hi_expr @@ -337,9 +417,12 @@ def filter_by_dict_polars(df: "PolarsFrameT", filter_dict: "Optional[Dict[str, A def filter_expr_by_dict_polars(df: "Union[pl.DataFrame, pl.LazyFrame]", filter_dict: "Optional[Dict[str, Any]]") -> "Optional[pl.Expr]": """Build the combined boolean ``pl.Expr`` filter_by_dict_polars would apply, or None for an empty/absent filter dict. ``df`` supplies the schema for column/dtype - resolution only — callers may apply the expr to a LazyFrame over the same schema - (the fused connected-join lane), with identical semantics incl. the same typed - error/NIE contract for unsupported shapes.""" + resolution, plus one row-count carve-out: an EMPTY eager ``pl.DataFrame`` (height 0) + skips the scalar-equality typed-error/temporal-parse block, so it can return a plain + ``==`` expr where a LazyFrame over the same schema raises GFQLSchemaError(E302). + Otherwise callers may apply the expr to a LazyFrame over the same schema (the fused + connected-join lane), with identical semantics incl. the same typed error/NIE contract + for unsupported shapes.""" import polars as pl if not filter_dict: @@ -354,7 +437,12 @@ def _dtype_of(name: str) -> "Optional[pl.DataType]": return _schema_memo[0].get(name) for col, val in filter_dict.items(): - resolved_col, resolved_val = resolve_filter_column(df, col, val) + resolved = resolve_filter_column_or_absent(df, col, val) + if resolved is None: + if not absent_column_matches(val): + return pl.lit(False) # absent column is all-null; 3VL never matches (#1916) + continue + resolved_col, resolved_val = resolved if isinstance(resolved_val, ASTPredicate): if _is_cross_type_predicate(df, resolved_col, resolved_val): # numeric-vs-string comparison -> polars ComputeError; decline (NIE). @@ -435,6 +523,14 @@ def _dtype_of(name: str) -> "Optional[pl.DataType]": column_type=str(_eq_dtype), suggestion=f'Use a string value like {col}="value"', ) + if isinstance(resolved_val, str) and _dtype_is_temporal(_eq_dtype): + # Raw temporal `col == 'str'` raises at collect; parse-or-typed-error instead. + exprs.append( + _temporal_str_cmp_expr( + resolved_col, pl.col(resolved_col), operator.eq, resolved_val, _eq_dtype + ) + ) + continue exprs.append(pl.col(resolved_col) == resolved_val) if not exprs: diff --git a/graphistry/compute/gfql/lazy/engine/polars/projection.py b/graphistry/compute/gfql/lazy/engine/polars/projection.py index 93a20e7ff5..f9d9ccb947 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/projection.py +++ b/graphistry/compute/gfql/lazy/engine/polars/projection.py @@ -5,17 +5,21 @@ differential parity vs pandas is the release gate. The #1650 default (``structured=True``) FLATTENS whole-entity ``RETURN n`` to ``{output}.{field}`` columns natively for ANY dtype (float/temporal/nested just become columns, no rendering). Legacy display-string rendering -(``structured=False``) is native only for single-entity int/string/bool nodes (boolean -``label__*`` flags included); float/temporal/nested entity text, multi-entity, edges, and -exotic expressions raise NotImplementedError. +(``structured=False``) is native for int/string/bool node entities, including multi-node +binding rows (boolean ``label__*`` flags included); float/temporal/nested entity text, edge +entities, and exotic expressions raise NotImplementedError. """ from __future__ import annotations -from typing import TYPE_CHECKING, Any, Dict, List, Optional +import typing +from dataclasses import dataclass + +from typing_extensions import Literal, TypedDict from graphistry.Plottable import Plottable +from graphistry.compute.gfql.cypher.projection_columns import alias_field_sources -if TYPE_CHECKING: +if typing.TYPE_CHECKING: import polars as pl from graphistry.compute.gfql.cypher.lowering import ResultProjectionPlan @@ -28,6 +32,19 @@ ) +class _PolarsWholeRowProjectionMeta(TypedDict): + table: Literal["nodes", "edges"] + alias: str + id_column: str + ids: pl.Series + + +@dataclass(frozen=True) +class _AliasView: + frame: pl.DataFrame + columns: typing.Mapping[str, str] + + def _has_temporal_constructor_text(rows_df: pl.DataFrame, col: str) -> bool: """True if a String property column holds Cypher temporal-constructor text (``date({...})``, ``datetime({...})``, …). The TCK graph builder stores temporal properties as these strings; @@ -49,7 +66,7 @@ def _has_temporal_constructor_text(rows_df: pl.DataFrame, col: str) -> bool: return False -def _native_scalar_text_expr(col: str, dtype: Any) -> Optional[Any]: +def _native_scalar_text_expr(col: str, dtype: pl.DataType) -> typing.Optional[pl.Expr]: """Per-dtype cypher value rendering as a polars expression, or None to bail. Matches the pandas entity renderer for safe scalars: ints raw, bools lowercased, strings single-quoted with ``\\``→``\\\\`` then ``'``→``\\'``. Floats (scientific/NaN repr diverges from pandas), @@ -66,18 +83,35 @@ def _native_scalar_text_expr(col: str, dtype: Any) -> Optional[Any]: return None -def _native_node_entity_text_expr(rows_df: Any, alias: str, exclude: Any) -> Optional[Any]: - """Native ``(:Label {prop: val, ...})`` node entity text; ``None`` → caller raises.""" +def _alias_view_polars(rows_df: pl.DataFrame, alias: str) -> typing.Optional[_AliasView]: import polars as pl - cols = list(rows_df.columns) - if alias not in cols: + field_sources = alias_field_sources(rows_df.columns, alias) + if field_sources is None: return None - single_entity_untyped_rows = ( - not any(str(c).startswith(f"{alias}.") for c in cols) and "type" not in cols + if all(field == source for field, source in field_sources.items()): + return _AliasView(frame=rows_df, columns=field_sources) + frame = rows_df.select( + [pl.col(source).alias(field) for field, source in field_sources.items()] ) - if not single_entity_untyped_rows: + return _AliasView(frame=frame, columns=field_sources) + + +def _native_node_entity_text_expr( + view: _AliasView, alias: str, exclude: typing.Sequence[str] +) -> typing.Optional[pl.Expr]: + """Render a native node entity expression from one alias view.""" + import polars as pl + + rows_df = view.frame + cols = list(rows_df.columns) + has_node_entity_shape = alias in cols and "type" not in cols + if not has_node_entity_shape: return None + + def _c(field: str) -> pl.Expr: + return pl.col(view.columns.get(field, field)) + from .dtypes import is_int schema = rows_df.schema excluded = set(str(c) for c in (exclude or ())) @@ -93,17 +127,17 @@ def _native_node_entity_text_expr(rows_df: Any, alias: str, exclude: Any) -> Opt return None # non-boolean label flags -> defer (NIE) labels = ( pl.concat_str([ - pl.when(pl.col(c).fill_null(False)).then(pl.lit(":" + label_name)).otherwise(pl.lit("")) + pl.when(_c(c).fill_null(False)).then(pl.lit(":" + label_name)).otherwise(pl.lit("")) for c, label_name in label_cols ], separator="") if label_cols else pl.lit("") ) segments = [] for col in prop_cols: - val = _native_scalar_text_expr(col, schema[col]) + val = _native_scalar_text_expr(view.columns.get(col, col), schema[col]) if val is None: return None - segments.append(pl.when(pl.col(col).is_null()).then(None).otherwise(pl.lit(f"{col}: ") + val)) + segments.append(pl.when(_c(col).is_null()).then(None).otherwise(pl.lit(f"{col}: ") + val)) if not segments: rendered = pl.lit("(") + labels + pl.lit(")") else: @@ -115,51 +149,44 @@ def _native_node_entity_text_expr(rows_df: Any, alias: str, exclude: Any) -> Opt # Nullify absent (OPTIONAL-MATCH miss) rows — alias marker is null there and an absent # entity must render null, not "()" (mirrors pandas _nullify_missing_alias_rows); a real # property-less node keeps "()". - return pl.when(pl.col(alias).is_null()).then(None).otherwise(rendered) + return pl.when(_c(alias).is_null()).then(None).otherwise(rendered) -def _flat_entity_exprs_polars(rows_df: pl.DataFrame, projection: ResultProjectionPlan, source_alias: str, output_name: str, id_column: Optional[str]) -> Optional[List[pl.Expr]]: - """Structured (flattened) whole-entity projection (#1650), polars edition. Mirrors pandas - ``_flat_entity_columns`` exactly (same field selection + ordering via the shared - ``_flat_entity_field_names``): one ``pl.col(field).alias("{output}.{field}")`` per field. - Single-entity only (None on multi-entity prefixed columns or absent fields). Works for ANY - dtype (float/temporal/nested just become columns), covering cases entity-text defers.""" +def _flat_entity_exprs_polars( + view: _AliasView, + projection: ResultProjectionPlan, + source_alias: str, + output_name: str, + id_column: typing.Optional[str], +) -> typing.Optional[typing.Sequence[pl.Expr]]: + """Flatten one alias view into projected Polars expressions.""" import polars as pl from dataclasses import replace from graphistry.compute.gfql.cypher.result_postprocess import _flat_entity_field_names - cols = list(rows_df.columns) - if source_alias not in cols: - return None - if any(str(c).startswith(f"{source_alias}.") for c in cols): - return None # multi-entity binding -> defer (NIE), matches the text path source_projection = projection if source_alias == projection.alias else replace(projection, alias=source_alias) - fields = _flat_entity_field_names(rows_df, source_projection, id_column) + fields = _flat_entity_field_names(view.frame, source_projection, id_column) if not fields: return None # synthesized absent entity -> caller falls back to text out = [] for field in fields: - if field not in cols: + src = view.columns.get(field) + if src is None: return None - out.append(pl.col(field).alias(f"{output_name}.{field}")) + out.append(pl.col(src).alias(f"{output_name}.{field}")) return out def _record_entity_meta( - entity_meta: Dict[str, Dict[str, Any]], - rows_df: pl.DataFrame, + entity_meta: typing.MutableMapping[str, _PolarsWholeRowProjectionMeta], + view: _AliasView, projection: ResultProjectionPlan, source_alias: str, output_name: str, - id_column: Optional[str], + id_column: typing.Optional[str], ) -> None: - """Record whole-entity projection metadata for one column, mirroring the pandas projector. - - ``_try_native_projection`` reaches this only in the single-entity branch (flat exprs and the - entity-text path both decline multi-entity prefixed columns), so ``rows_df`` is the aligned - source frame and ``rows_df[id_column]`` is the carried alias's id column, row-aligned with the - projected output. Snapshot (``.clone()``) the id column so downstream reentry recovery never - aliases a later-mutated working frame (see #1356).""" + """Record row-aligned identity metadata for one whole-entity output.""" + rows_df = view.frame if id_column is None or id_column not in rows_df.columns: # pragma: no cover - defensive: node re-entry always carries the id column return entity_meta[output_name] = { @@ -170,38 +197,44 @@ def _record_entity_meta( } -def _try_native_projection(result: Plottable, rows_df: pl.DataFrame, projection: ResultProjectionPlan, structured: bool) -> Optional[Plottable]: +def _try_native_projection( + result: Plottable, + rows_df: pl.DataFrame, + projection: ResultProjectionPlan, + structured: bool, +) -> typing.Optional[Plottable]: """Native projection for property/expr columns already in the polars row table + structured- flat or entity-text whole-entity returns; None → caller raises NIE.""" import polars as pl - exprs = [] - # Whole-entity projection metadata side-channel (#1273 WITH->MATCH re-entry): mirror the - # pandas projector (result_postprocess._apply_result_projection_pandas), which records the - # carried alias's id column so the bounded-reentry executor can recover carried node - # identities. Without it a WITH-projected node alias feeding a trailing MATCH declines. - entity_meta: Dict[str, Dict[str, Any]] = {} + exprs: typing.List[pl.Expr] = [] + entity_meta: typing.MutableMapping[str, _PolarsWholeRowProjectionMeta] = {} id_column = result._node + primary = _alias_view_polars(rows_df, projection.alias) + primary_columns = primary.columns if primary is not None else {} for column in projection.columns: if column.kind == "whole_row": if projection.table != "nodes": return None # edge entity rendering -> defer (NIE) source_alias = column.source_name or projection.alias + view = _alias_view_polars(rows_df, source_alias) + if view is None: + return None if structured: - # #1650 default: flatten to {output}.{field} (near-free, any dtype); - # text fallback only for synthesized-absent rows. - flat = _flat_entity_exprs_polars(rows_df, projection, source_alias, column.output_name, id_column) + flat = _flat_entity_exprs_polars(view, projection, source_alias, column.output_name, id_column) if flat is not None: exprs.extend(flat) - _record_entity_meta(entity_meta, rows_df, projection, source_alias, column.output_name, id_column) + _record_entity_meta(entity_meta, view, projection, source_alias, column.output_name, id_column) continue - ent = _native_node_entity_text_expr(rows_df, source_alias, projection.exclude_columns) + ent = _native_node_entity_text_expr(view, source_alias, projection.exclude_columns) if ent is None: return None exprs.append(ent.alias(column.output_name)) - _record_entity_meta(entity_meta, rows_df, projection, source_alias, column.output_name, id_column) + _record_entity_meta(entity_meta, view, projection, source_alias, column.output_name, id_column) continue src = column.source_name + if src is not None: + src = primary_columns.get(src, src) if src is None or src not in rows_df.columns: return None # expression needing evaluation / missing -> defer (NIE) dtype = rows_df.schema[src] @@ -235,8 +268,8 @@ def apply_result_projection_polars( ``structured=True`` (#1650 default): flatten whole-entity returns to ``{output}.{field}`` columns (any dtype, near-free). ``structured=False``: legacy Cypher display string, native - for int/string/bool single-entity nodes with boolean ``label__*`` flags. Multi-entity - bindings, edge entity-text, and (text mode) float/temporal/nested columns are not yet + for int/string/bool node entities, including multi-node binding rows, with boolean + ``label__*`` flags. Edge entity-text and (text mode) float/temporal/nested columns are not yet native → raise rather than secretly run the pandas renderer. """ rows_df = result._nodes @@ -245,7 +278,7 @@ def apply_result_projection_polars( return native raise NotImplementedError( "polars engine does not yet natively render this cypher result projection " - "(whole-entity RETURN over float/temporal/nested/multi-entity columns); " + "(unsupported node entity text, edge entities, or exotic expressions); " "use engine='pandas' or engine='cudf' for this query " "(no silent fallback; parity-or-error by design)" ) diff --git a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py index 5a2ff5027a..a2ab69d5e5 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py +++ b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py @@ -41,6 +41,8 @@ from graphistry.compute.gfql.agg_types import ( GFQL_NUMERIC_ONLY_AGGREGATIONS, numeric_agg_all_null_value, + polars_all_null_agg_literal, + polars_conform_agg_dtype, polars_non_numeric_agg_dtype, raise_non_numeric_aggregation, ) @@ -220,9 +222,9 @@ def _lower_function(node: FunctionCall, columns: Sequence[str]) -> Optional[pl.E if name == "size" and len(args) == 1: # size(x): #chars (String) or #elements (List) — different polars ops, so gate by output # dtype. str.len_chars == pandas str.len (code points); list.len parity; null/empty - # preserved — parity-verified. Numeric/Categorical/unknown decline (NIE): pandas size() - # over a non-sequence Series returns the ROW COUNT (quirk we refuse to replicate), and - # Categorical .str raises in polars only. + # preserved — parity-verified. Numeric/Categorical/unknown decline (NIE), matching the + # pandas/cuDF kernel, which declines size() over a non-sequence Series; Categorical .str + # raises in polars only. dt = _expr_output_dtype(args[0]) if dt == pl.String: return args[0].str.len_chars() @@ -1394,7 +1396,7 @@ def _agg_expr(func: str, expr: Optional[str], columns: Sequence[str], alias: str import polars as pl func = func.lower() if func == "count" and (expr is None or expr == "*"): - return pl.len().alias(alias) + return polars_conform_agg_dtype(pl.len(), func, None, alias) if not isinstance(expr, str) or expr not in columns: return None col = pl.col(expr) @@ -1422,7 +1424,7 @@ def _agg_expr(func: str, expr: Optional[str], columns: Sequence[str], alias: str if dtype == pl.Null: # all-null by construction: `sum`/`mean` are unsupported on `null` dtype in polars, # while cypher says 0 / null. - return pl.lit(numeric_agg_all_null_value(func)).alias(alias) + return polars_all_null_agg_literal(func, alias) dtype_label = polars_non_numeric_agg_dtype(dtype) if dtype_label is not None: # An ALL-NULL column carries no type evidence, so it is never a type error: cypher @@ -1430,14 +1432,14 @@ def _agg_expr(func: str, expr: Optional[str], columns: Sequence[str], alias: str # and pandas already did (an all-None pandas object column arrives here typed # `String`). Both would otherwise raise -- `sum`/`mean` are unsupported on `str`. if is_all_null is not None and is_all_null(expr): - return pl.lit(numeric_agg_all_null_value(func)).alias(alias) + return polars_all_null_agg_literal(func, alias) # Raise, don't return None: None is an NIE-decline that falls back to the pandas # kernel, which would then ANSWER the same wrong-typed query. raise_non_numeric_aggregation(func, expr, dtype_label, alias) if func == "count": - return col.count().alias(alias) + return polars_conform_agg_dtype(col.count(), func, dtype, alias) if func == "sum": - return col.sum().alias(alias) + return polars_conform_agg_dtype(col.sum(), func, dtype, alias) if func in ("avg", "mean"): return col.mean().alias(alias) if func == "min": @@ -1447,7 +1449,7 @@ def _agg_expr(func: str, expr: Optional[str], columns: Sequence[str], alias: str if func == "count_distinct": # count(DISTINCT x) drops nulls (pandas nunique(dropna=True)); polars n_unique() counts # null, so drop_nulls first. - return col.drop_nulls().n_unique().alias(alias) + return polars_conform_agg_dtype(col.drop_nulls().n_unique(), func, dtype, alias) if func == "collect": # collect(x) drops nulls, keeps within-group row order (pandas row/pipeline.py:4552-4582: # ~isna() then agg(list)). Inside group_by(maintain_order=True).agg a multi-valued expr @@ -1649,15 +1651,14 @@ def _cartesian_node_bindings_polars( # L4 pushdown twin of pandas `_gfql_cartesian_node_bindings_row_table` matched = _apply_alias_prefilters_polars(matched, alias, alias_prefilters) # honoured, never dropped (#1804) cols = matched.collect_schema().names() - # prop_cols excludes node_id and any real column named == alias: the pandas - # node execute() leaks a boolean FLAG into a column named ``alias`` - # (shadowing a same-named real property), which the lookup frame surfaces - # as ``alias.alias = True``. Reproduce that exactly. + # prop_cols excludes node_id and any real column named == alias; that column is + # emitted once below as ``alias.alias``: the real user values when the column + # exists (unshadow parity with pandas), else the flag ``True``. prop_cols = [c for c in cols if c != node_id and c != alias] exprs = [ pl.col(node_id).alias(alias), pl.col(node_id).alias(f"{alias}.{node_id}"), - pl.lit(True).alias(f"{alias}.{alias}"), + (pl.col(alias) if alias in cols else pl.lit(True)).alias(f"{alias}.{alias}"), ] exprs.extend(pl.col(c).alias(f"{alias}.{c}") for c in prop_cols) per_alias.append(matched.select(exprs)) @@ -1953,11 +1954,9 @@ def _names(lf: pl.LazyFrame) -> List[str]: seed_nodes = seed_nodes.join(seed_ids_lf, on=node_id, how="semi") # L4 pushdown twin of pandas `_gfql_connected_bindings_state`'s seed prefilter seed_nodes = _apply_alias_prefilters_polars(seed_nodes, first_op._name, alias_prefilters) # honoured, never dropped (#1804) - # The whole generic builder works in LazyFrames (`nodes_lf` / `edges_lf` above); - # `filter_by_dict_polars` is frame-polymorphic at runtime but declares the eager - # type, so pin the path bag lazy here instead of leaving every downstream lazy - # op to fight an eager inference. - state: pl.LazyFrame = seed_nodes.select(pl.col(node_id).alias(WALK_CURRENT_COL)) # type: ignore[assignment] + state = seed_nodes.select( + pl.col(node_id).alias(WALK_CURRENT_COL) + ).unique(subset=[WALK_CURRENT_COL], maintain_order=True) alias_frames: Dict[str, pl.LazyFrame] = {} node_aliases: List[str] = [] first_alias = first_op._name diff --git a/graphistry/compute/gfql/rollout.py b/graphistry/compute/gfql/rollout.py index 67cca064f6..b7623d87a3 100644 --- a/graphistry/compute/gfql/rollout.py +++ b/graphistry/compute/gfql/rollout.py @@ -7,16 +7,22 @@ are compatibility values for callers that still inspect the helper and do not restore loose binder behavior. -Precedence (most specific wins): +.. warning:: + + ``GRAPHISTRY_GFQL_STRICT_SCHEMA`` **does not affect query behavior.** + Setting it to ``0``, ``1``, or leaving it unset all produce the same + result: an absent label still raises ``GFQLSchemaError``. Nothing in the + execution path consults this module -- every symbol below is re-exported + but never called by product code. The precedence chain described here is + the shape a strictness setting WOULD take if one were wired up; it is not + a description of current behavior. Whether to serve absent labels loosely + is an open design question. + +Precedence the helpers implement for their own consumers (most specific wins): 1. Explicit caller parameter (e.g. ``FrontendBinder.bind(strict_name_resolution=True)``) 2. Catalog-level metadata flag (e.g. ``GraphSchemaCatalog.metadata['strict']``) 3. Process-wide env default (e.g. ``GRAPHISTRY_GFQL_STRICT_SCHEMA``) 4. Historical loose default for helper-only consumers - -The env tier remains default-off at the helper layer; binder execution no -longer treats that default as permission for loose compatibility paths. - -Production binder callers no longer use this helper as a loose/strict gate. """ from __future__ import annotations @@ -52,7 +58,7 @@ def env_bool(name: str, default: bool = False) -> bool: def strict_schema_env_default() -> bool: - """Return the env-default for strict schema mode (default off).""" + """Report the env var's value. Execution does not consult it -- see module warning.""" return env_bool(STRICT_SCHEMA_ENV, default=False) diff --git a/graphistry/compute/gfql/row/entity_props.py b/graphistry/compute/gfql/row/entity_props.py index eb921c09c5..6e33e87a61 100644 --- a/graphistry/compute/gfql/row/entity_props.py +++ b/graphistry/compute/gfql/row/entity_props.py @@ -143,6 +143,9 @@ def _nullify_missing_alias_rows(df: DataFrameT, alias_col: str, rendered: Series def _all_non_null_match(mask: SeriesT, non_null: SeriesT) -> bool: if not hasattr(mask, "where"): return False + # all-null proves nothing; vacuous truth rendered NULLs as constructor zero-values + if hasattr(non_null, "any") and not bool(non_null.any()): + return False return bool(mask.where(non_null, True).all()) diff --git a/graphistry/compute/gfql/row/frame_ops.py b/graphistry/compute/gfql/row/frame_ops.py index d850bccce2..c034430ab7 100644 --- a/graphistry/compute/gfql/row/frame_ops.py +++ b/graphistry/compute/gfql/row/frame_ops.py @@ -61,6 +61,64 @@ def _alias_true_mask(table_df: Any, source: str) -> Any: return mask.astype(bool) +def _restore_alias_shadowed_user_column( + ctx: RowPipelineCtx, table_df: "DataFrameT", table: Optional[str], source: str +) -> "DataFrameT": + """An alias named like a user column (``MATCH (name:P) RETURN name.name``) has that + column overwritten by the alias marker upstream, so ``source.source`` read back the + marker. Re-key the user's values from the base frame under an internal restore + column the projection resolves, keeping the boolean marker intact for every other + read. No-op when the alias shadows nothing, the base column is itself a boolean + marker (an intermediate dispatch graph), or rows cannot be re-keyed.""" + from graphistry.compute.gfql.identifiers import shadow_restore_column + + base_graph = ctx._gfql_rows_base_graph if ctx._gfql_rows_base_graph is not None else ctx._g + base_frame = None if base_graph is None else ( + base_graph._nodes if table == "nodes" else base_graph._edges + ) + if base_frame is None or source not in base_frame.columns: + return table_df + if _is_polars(table_df): + import polars as pl + base_is_marker = base_frame.schema.get(source) == pl.Boolean + else: + base_is_marker = str(getattr(base_frame[source], "dtype", "")).startswith("bool") + if base_is_marker: + return table_df + key = base_graph._node if table == "nodes" else base_graph._edge # type: ignore[union-attr] + if _is_polars(table_df): + if ( + key is not None and key != source + and key in table_df.columns and key in base_frame.columns + and base_frame[key].n_unique() == len(base_frame) + ): + orig_cols = list(table_df.columns) + return table_df.drop(source).join( + base_frame.select([key, source]), on=key, how="left" + ).select(orig_cols) + return table_df + restore_col = shadow_restore_column(source) + base_index = getattr(base_frame, "index", None) + if base_index is not None and bool(base_index.is_unique): + # guarded .loc proves index-subset alignment (cuDF Index.isin disagrees with pandas) + try: + restored = base_frame[source].loc[table_df.index] + except (KeyError, IndexError, TypeError): + restored = None + if restored is not None and len(restored) == len(table_df): + out = table_df.copy() + out[restore_col] = restored + return out + if ( + key is not None and key != source + and key in table_df.columns and key in base_frame.columns + and bool(base_frame[key].is_unique) + ): + renamed = base_frame[[key, source]].rename(columns={source: restore_col}) + return table_df.merge(renamed, on=key, how="left") + return table_df + + def row_table(ctx: RowPipelineCtx, table_df: Any) -> "Plottable": """Return a plottable that treats ``table_df`` as the active row table.""" from graphistry.compute.gfql.index.handoff import clear_handoff, read_handoff @@ -228,11 +286,13 @@ def rows( # the polars frame would otherwise widen the variable's type and break the pandas # ``.loc`` branch below (``is_polars_df`` is a TypeGuard, so it does not narrow the # negative branch back to pandas). Same call, same argument, same result. - return row_table(ctx, table_df.filter(pl.col(source).fill_null(False).cast(pl.Boolean))) + return row_table(ctx, _restore_alias_shadowed_user_column( + ctx, table_df.filter(pl.col(source).fill_null(False).cast(pl.Boolean)), table, source)) # unreachable for polars (returned above), but the guard on the ``.copy()`` branch # further up leaves the polars arm in this variable's type: TypeGuard narrows only # the positive branch, so ``not _is_polars(...)`` cannot narrow back to pandas. table_df = table_df.loc[_alias_true_mask(table_df, source)] # type: ignore[union-attr] + table_df = _restore_alias_shadowed_user_column(ctx, table_df, table, source) return row_table(ctx, table_df) diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py index 66420fb54c..6ffd512d9b 100644 --- a/graphistry/compute/gfql/row/pipeline.py +++ b/graphistry/compute/gfql/row/pipeline.py @@ -22,7 +22,7 @@ from graphistry.compute.gfql.call.support import AggSpec from graphistry.compute.gfql.row import frame_ops as row_frame_ops from graphistry.compute.gfql.row.prefilter import AliasPrefilters -from graphistry.compute.typing import DataFrameT +from graphistry.compute.typing import DataFrameT, SeriesT from graphistry.utils.json import JSONVal from graphistry.compute.gfql.row.order_expr import ( extract_temporal_duration_sort_ast, @@ -32,6 +32,8 @@ from graphistry.compute.gfql.agg_types import ( GFQL_NUMERIC_ONLY_AGGREGATIONS, numeric_agg_all_null_value, + pandas_agg_kernel_null_fill, + pandas_conform_agg_dtype, pandas_dtype_is_numeric_for_agg, pandas_non_numeric_agg_dtype, pandas_object_series_is_bool_like, @@ -81,6 +83,7 @@ WALK_PREV_COL, WALK_TO_COL, is_shortest_path_hops_column, + shadow_restore_column, trail_column_name, ) from graphistry.compute.util import generate_safe_column_name @@ -949,6 +952,29 @@ def _gfql_eval_list_comparison_op( out = (~out.astype("boolean")).where(~out.isna(), pd.NA) return out.reset_index(drop=True) + @staticmethod + def _gfql_report_absent_property(name: str) -> None: + """Route an absent row-expression property through the shared strictness + resolution: raise under ``strict``, warn once under ``warn``.""" + from graphistry.compute.gfql.strictness import ( + absent_name_is_lenient, + is_internal_plumbing_name, + ) + + if is_internal_plumbing_name(name): + return + if absent_name_is_lenient(name, kind="property", context="row table"): + return + from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError + + raise GFQLSchemaError( + ErrorCode.E301, + f'Property "{name}" does not exist in row table', + field=name, + value=name, + suggestion='Pass strict="warn" (default) to resolve absent properties to null', + ) + def _gfql_eval_expr_ast(self, table_df: Any, node: Any) -> Tuple[bool, Any]: parser_bundle = _gfql_expr_runtime_parser_bundle() if parser_bundle is None: @@ -1081,6 +1107,11 @@ def _gfql_eval_expr_ast(self, table_df: Any, node: Any) -> Tuple[bool, Any]: if isinstance(node, PropertyAccessExpr): if isinstance(node.value, Identifier): alias_name = node.value.name + if alias_name == node.property: + restore_col = shadow_restore_column(alias_name) + if restore_col in table_df.columns: + # the alias marker overwrote this same-named user column; rows() re-keyed it + return True, table_df[restore_col] if "." not in alias_name and RowPipelineMixin._gfql_has_bindings_alias_prefix(table_df, alias_name): if node.property == NODE_IDENTITY_COLUMN: node_id = self._gfql_node_id_column() @@ -1091,6 +1122,7 @@ def _gfql_eval_expr_ast(self, table_df: Any, node: Any) -> Tuple[bool, Any]: binding_col = f"{alias_name}.{node.property}" if binding_col in table_df.columns: return True, table_df[binding_col] + self._gfql_report_absent_property(binding_col) return True, self._gfql_broadcast_scalar(table_df, pd.NA) has_bound_graph_table = ( (self._node is not None and self._node in table_df.columns) @@ -1114,6 +1146,8 @@ def _gfql_eval_expr_ast(self, table_df: Any, node: Any) -> Tuple[bool, Any]: if hasattr(prop_value, "where"): prop_value = self._gfql_mask_fill(prop_value, alias_mask != True, None) # noqa: E712 return True, prop_value + if node.property not in table_df.columns: + self._gfql_report_absent_property(f"{alias_name}.{node.property}") prop_value = ( table_df[node.property] if node.property in table_df.columns @@ -1575,8 +1609,14 @@ def _is_arith_node(operand_node: Any) -> bool: # hygiene-ok: explicit-any -- AS if hasattr(inner, "astype"): try: return True, series_sequence_len(inner) - except Exception: - pass + except Exception as exc: + # Never fall through to len(series): that is the frame's height. + if not RowPipelineMixin._gfql_series_holds_no_typed_cell(inner): + raise ValueError( + "unsupported row expression: size() requires list/string input" + ) from exc + if len(inner) > 0: + return True, self._gfql_broadcast_scalar(table_df, None) try: return True, len(inner) except Exception: @@ -1908,7 +1948,11 @@ def _floor_series(s: Any) -> Any: list_null_mask = self._gfql_null_mask(base, base[list_col]) try: total_series = series_sequence_len(base[list_col]) - except Exception: + except Exception as exc: + if not RowPipelineMixin._gfql_series_holds_no_typed_cell(base[list_col]): + raise ValueError( + f"unsupported row expression: {str(node.fn).lower()}() requires list/string input" + ) from exc total_series = self._gfql_broadcast_scalar(base, pd.NA) if hasattr(total_series, "where"): total_series = total_series.where(~list_null_mask, 0).fillna(0) @@ -1992,7 +2036,11 @@ def _floor_series(s: Any) -> Any: null_mask = self._gfql_null_mask(base, base[list_col]) try: lengths = series_sequence_len(base[list_col]) - except Exception: + except Exception as exc: + if not RowPipelineMixin._gfql_series_holds_no_typed_cell(base[list_col]): + raise ValueError( + "unsupported row expression: list comprehension requires list/string input" + ) from exc lengths = self._gfql_broadcast_scalar(base, pd.NA) if hasattr(lengths, "fillna"): lengths = lengths.fillna(0) @@ -2391,6 +2439,22 @@ def _coerce_duration_property(value_text: str) -> Any: f"unsupported row expression: property access requires a graph element alias, entity value, or map in {expr!r}" ) + @staticmethod + def _gfql_series_holds_no_typed_cell(series: SeriesT) -> bool: + """No non-null cell exists, so the element type is unknown rather than wrong. + + An empty column (a zero-row intermediate) or an all-null one carries no evidence + that its values are not sequences, so a sequence op over it must not be refused on + dtype alone — an empty ``collect()`` is still a list. + """ + if len(series) == 0: + return True + isna = getattr(series, "isna", None) + if isna is None: + return False + null_mask = isna() + return hasattr(null_mask, "all") and bool(null_mask.all()) + @staticmethod def _gfql_series_is_list_like(series: Any) -> bool: if not hasattr(series, "isna") or not hasattr(series, "astype"): @@ -3993,7 +4057,12 @@ def _gfql_connected_bindings_state( first_nodes = self._gfql_apply_alias_prefilter( first_nodes, first_alias, alias_prefilters ) - state_df = first_nodes[[node_id_col]].copy().rename(columns={node_id_col: WALK_CURRENT_COL}) + state_df = ( + first_nodes[[node_id_col]] + .drop_duplicates(subset=[node_id_col], keep="first") + .copy() + .rename(columns={node_id_col: WALK_CURRENT_COL}) + ) alias_frames: Dict[str, DataFrameT] = {} if isinstance(first_alias, str): state_df[first_alias] = state_df[WALK_CURRENT_COL] @@ -4606,7 +4675,12 @@ def _gfql_cartesian_node_bindings_row_table( ) if isinstance(alias, str): - frame = self._gfql_node_alias_lookup_frame(matched_nodes, str(node_id), alias) + # Same marker shadowing as the connected path, keyed on the node id. + lookup_source = self._gfql_unshadow_alias_marker_column( + matched_nodes, alias, base_nodes, str(node_id) + ) + assert lookup_source is not None # non-None in, non-None out + frame = self._gfql_node_alias_lookup_frame(lookup_source, str(node_id), alias) else: anon_col = RowPipelineMixin._gfql_fresh_col_name(matched_nodes.columns, f"__gfql_binding_node_{idx}__") frame = matched_nodes[[node_id]].copy().rename(columns={node_id: anon_col}) @@ -5576,6 +5650,13 @@ def _build_grouped(group_df: Any) -> Any: if func == "sum" and pandas_object_series_is_bool_like(table_df[expr_col]): # The object-dtype kernel returns a lone-row group as the raw bool. agg_df = agg_df.assign(**{alias: pd.to_numeric(agg_df[alias])}) # bool sums as int (#1821) + null_fill = pandas_agg_kernel_null_fill(func, table_df[expr_col]) + if null_fill is not None: + # cypher sum() never answers null; cuDF's grouped sum does (agg_types.py) + agg_df = agg_df.assign(**{alias: agg_df[alias].fillna(null_fill)}) + agg_df = agg_df.assign(**{alias: pandas_conform_agg_dtype( + agg_df[alias], func, + str(table_df[expr_col].dtype).lower() in {"bool", "boolean"})}) out_df = out_df.merge(agg_df, on=key_cols, how="left", sort=False) if func in {"collect", "collect_distinct"}: diff --git a/graphistry/compute/gfql/same_path/df_utils.py b/graphistry/compute/gfql/same_path/df_utils.py index a934b0ef03..ad0ce929bc 100644 --- a/graphistry/compute/gfql/same_path/df_utils.py +++ b/graphistry/compute/gfql/same_path/df_utils.py @@ -1,7 +1,7 @@ import operator -from typing import Any +from typing import Any, Tuple -from graphistry.compute.typing import DataFrameT +from graphistry.compute.typing import DataFrameT, SeriesT from graphistry.compute.dataframe import ( ineq_eval_pairs, project_node_attrs, @@ -31,12 +31,33 @@ } +def _align_mixed_tz_datetimes(series_left: SeriesT, series_right: SeriesT) -> Tuple[SeriesT, SeriesT]: + """Normalize a tz-aware/tz-naive datetime pair onto UTC-naive: GFQL reads naive + datetimes as UTC (as the row pipeline's ``_native_epoch_ticks`` does), and the + raw pandas compare of the mixed pair raises.""" + left_dtype = getattr(series_left, "dtype", None) + right_dtype = getattr(series_right, "dtype", None) + if getattr(left_dtype, "kind", None) != "M" or getattr(right_dtype, "kind", None) != "M": + return series_left, series_right + left_tz = getattr(left_dtype, "tz", None) + right_tz = getattr(right_dtype, "tz", None) + if (left_tz is None) == (right_tz is None): + return series_left, series_right + try: + if left_tz is not None: + return series_left.dt.tz_convert("UTC").dt.tz_localize(None), series_right + return series_left, series_right.dt.tz_convert("UTC").dt.tz_localize(None) + except (AttributeError, TypeError): # pragma: no cover - engine without tz_convert + return series_left, series_right + + def evaluate_clause(series_left: Any, op: str, series_right: Any, *, null_safe: bool = False) -> Any: fn = _OPS.get(op) if fn is None: if null_safe: return (series_left.notna() & series_right.notna()) & False return False + series_left, series_right = _align_mixed_tz_datetimes(series_left, series_right) if not null_safe: return fn(series_left, series_right) valid = series_left.notna() & series_right.notna() diff --git a/graphistry/compute/gfql/strictness.py b/graphistry/compute/gfql/strictness.py new file mode 100644 index 0000000000..ddec8cdbaa --- /dev/null +++ b/graphistry/compute/gfql/strictness.py @@ -0,0 +1,255 @@ +"""Shared GFQL strictness resolution for absent labels/properties. + +One resolution, consulted by the validator AND every executor, so the two can +never drift: ``strict`` raises, ``warn`` warns once per absent name per call, +``quiet`` is silent. Under ``warn``/``quiet`` an absent name resolves to null, +which is openCypher. +""" + +from __future__ import annotations + +import warnings +from contextlib import contextmanager +from contextvars import ContextVar, Token +from dataclasses import dataclass, field +from typing import Any, FrozenSet, Iterator, Mapping, Optional, Set, Tuple + +from typing_extensions import Literal + +from graphistry.Plottable import Plottable + + +StrictLevel = Literal["strict", "warn", "quiet"] +StrictInput = Any # hygiene-ok: explicit-any -- public param accepts bool | StrictLevel | None + +STRICT_LEVELS: Tuple[str, ...] = ("strict", "warn", "quiet") + +#: What kind of name went missing, for the diagnostic text. +AbsentNameKind = Literal["label", "column", "property", "name"] + +#: Level used when neither the caller nor a bound schema selects one. +DEFAULT_STRICT_LEVEL: StrictLevel = "warn" + +#: Level applied at runtime sites reached OUTSIDE a GFQL execution scope +#: (e.g. a direct ``g.filter_nodes_by_dict``): unchanged, raising behavior. +UNSCOPED_STRICT_LEVEL: StrictLevel = "strict" + + + +def normalize_strict_level(value: StrictInput) -> Optional[StrictLevel]: + """``None`` (unset), ``True``->strict, ``False``->quiet, or a level name.""" + if value is None: + return None + if isinstance(value, bool): + return "strict" if value else "quiet" + if isinstance(value, str) and value in STRICT_LEVELS: + return value # type: ignore[return-value] + raise ValueError( + f"strict must be None, a bool, or one of {STRICT_LEVELS}; got {value!r}" + ) + + +def _declared_level(value: StrictInput) -> Optional[StrictLevel]: + """``normalize_strict_level`` for a value read OFF a bound schema. + + An unrecognized value is treated as unset rather than rejected: only the + caller's own ``strict=`` argument is worth failing loudly over, and duck-typed + schema objects must not turn every query into a TypeError. + """ + try: + return normalize_strict_level(value) + except ValueError: + return None + + +def strict_level_to_bool(level: StrictLevel) -> bool: + """Legacy boolean view for callers that only know strict-vs-loose.""" + return level == "strict" + + +def schema_declared_names(g: Optional[Plottable]) -> Optional[FrozenSet[str]]: + """Every name a bound ``GraphSchema`` declares, or ``None`` when none is bound. + + A name outside this set is a typo even under ``warn``/``quiet``; a name inside + it that this instance lacks is the narrow-subgraph case and is served. + """ + if g is None: + return None + schema = getattr(g, "_gfql_schema", None) + if schema is None: + return None + + names: Set[str] = set() + for type_attr in ("node_types", "edge_types"): + entries = getattr(schema, type_attr, ()) + if not isinstance(entries, (list, tuple)): + continue # duck-typed/partial schema object: nothing declared to judge against + for entry in entries: + properties = getattr(entry, "properties", None) + for name in properties if isinstance(properties, Mapping) else (): + names.add(str(name)) + entry_name = getattr(entry, "name", None) + if isinstance(entry_name, str): + names.add(entry_name) + names.add(f"label__{entry_name}") + for label in _string_members(getattr(entry, "labels", ())): + names.add(label) + names.add(f"label__{label}") + for column in _string_members(getattr(entry, "columns", ())): + names.add(column) + for column_attr in ("node_id_column", "edge_source_column", "edge_destination_column"): + bound_column = getattr(schema, column_attr, None) + if isinstance(bound_column, str): + names.add(bound_column) + if not names: + # A schema object that declares no names cannot disambiguate anything. + return None + return frozenset(names) + + +def _string_members(value: StrictInput) -> Tuple[str, ...]: + if isinstance(value, (list, tuple, set, frozenset)): + return tuple(str(item) for item in value) + return () + + +def resolve_strict_level(g: Optional[Plottable], *, strict: StrictInput = None) -> StrictLevel: + """explicit param -> schema.strict -> schema.metadata['strict'] -> default.""" + explicit = normalize_strict_level(strict) + if explicit is not None: + return explicit + schema = getattr(g, "_gfql_schema", None) if g is not None else None + if schema is not None: + declared = _declared_level(getattr(schema, "strict", None)) + if declared is not None: + return declared + metadata = getattr(schema, "metadata", None) + if isinstance(metadata, Mapping) and "strict" in metadata: + declared = _declared_level(metadata["strict"]) + if declared is not None: + return declared + return DEFAULT_STRICT_LEVEL + + +@dataclass +class _StrictnessScope: + level: StrictLevel + declared: Optional[FrozenSet[str]] = None + warned: Set[str] = field(default_factory=set) + + +_SCOPE: ContextVar[Optional[_StrictnessScope]] = ContextVar("gfql_strictness_scope", default=None) + + +@contextmanager +def strictness_scope( + level: StrictLevel, + *, + declared: Optional[FrozenSet[str]] = None, +) -> Iterator[_StrictnessScope]: + """Publish the resolved level to the runtime sites for one GFQL call. + + Nested calls reuse the outer scope so warn-once stays once per user call. + """ + existing = _SCOPE.get() + if existing is not None: + yield existing + return + scope = _StrictnessScope(level=level, declared=declared) + token: Token[Optional[_StrictnessScope]] = _SCOPE.set(scope) + try: + yield scope + finally: + _SCOPE.reset(token) + + +def current_strict_level() -> StrictLevel: + scope = _SCOPE.get() + return scope.level if scope is not None else UNSCOPED_STRICT_LEVEL + + +def is_internal_plumbing_name(name: str) -> bool: + """Synthetic row-pipeline columns (alias markers, re-entry keys, label flags). + + Never user-authored, so their absence is never a user-facing diagnostic. + """ + bare = name.rsplit(".", 1)[-1] + return bare.startswith("__") or bare.startswith("label__") + + +#: Columns cypher label matching resolves onto; structural, never declared properties. +LABEL_CARRIER_COLUMNS = frozenset({"type", "labels"}) + + +def name_is_schema_typo(name: str) -> bool: + """True when a bound schema exists and does not declare ``name``.""" + scope = _SCOPE.get() + if scope is None or scope.declared is None: + return False + if name in LABEL_CARRIER_COLUMNS or is_internal_plumbing_name(name): + return False + bare = name.rsplit(".", 1)[-1].rsplit(":", 1)[-1] + return name not in scope.declared and bare not in scope.declared + + +def absent_name_is_lenient( + name: str, + *, + kind: AbsentNameKind = "column", + context: Optional[str] = None, +) -> bool: + """Whether ``name``'s absence should resolve to null instead of raising. + + Returns ``True`` under ``warn`` (after warning once per distinct name per + call) and ``quiet``; ``False`` under ``strict`` and for a name a bound schema + does not declare, leaving the caller's own error to fire. + """ + if name_is_schema_typo(name): + return False + level = current_strict_level() + if level == "strict": + return False + if level == "warn": + scope = _SCOPE.get() + key = f"{kind}:{name}" + if scope is None or key not in scope.warned: + if scope is not None: + scope.warned.add(key) + where = f" in {context}" if context else "" + warnings.warn( + f'GFQL: {kind} "{name}" is absent{where}; it resolves to null ' + f"(openCypher). Pass strict=True to make this an error, or " + f'strict="quiet" to silence this warning.', + UserWarning, + stacklevel=3, + ) + return True + + +def absent_filter_key_is_lenient( + col: str, + val: Any, # hygiene-ok: explicit-any -- filter values are heterogeneous by contract + *, + context: Optional[str] = None, +) -> bool: + """``absent_name_is_lenient`` for a filter-dict key, reported as the LABEL it + came from when cypher lowered ``(n:X)`` to ``label__X: True``. + + Shared by the preflight and the runtime so both warn under the same key and a + single absent label warns once, not twice. + """ + if col.startswith("label__") and val is True: + return absent_name_is_lenient(col[len("label__"):], kind="label", context=context) + if col in LABEL_CARRIER_COLUMNS and isinstance(val, str): + return absent_name_is_lenient(val, kind="label", context=context) + return absent_name_is_lenient(col, kind="column", context=context) + + +def absent_column_matches(value: Any) -> bool: # hygiene-ok: explicit-any -- filter values are heterogeneous by contract + """3VL verdict for a filter against an all-null (absent) column. + + Every comparison against null is null (no match); only ``IS NULL`` is true. + """ + from graphistry.compute.predicates.comparison import IsNA + + return isinstance(value, IsNA) diff --git a/graphistry/compute/gfql/temporal/constructors.py b/graphistry/compute/gfql/temporal/constructors.py index 8c10461e8b..95192df418 100644 --- a/graphistry/compute/gfql/temporal/constructors.py +++ b/graphistry/compute/gfql/temporal/constructors.py @@ -862,7 +862,7 @@ def _decimal_value(key: str) -> Decimal: days_combined = ( _decimal_value("weeks") * 7 + _decimal_value("days") - + months_frac * Decimal("30.436875") + + months_frac * _AVERAGE_DAYS_PER_MONTH ) days_int = int(days_combined) days_frac = days_combined - Decimal(days_int) @@ -1068,5 +1068,8 @@ def normalize_temporal_constructor_text(text: str) -> Optional[str]: _split_zone_name, py_timedelta_from_offset, ) -from graphistry.compute.gfql.temporal.durations import _DURATION_TOKEN_RE # noqa: E402 +from graphistry.compute.gfql.temporal.durations import ( # noqa: E402 + _AVERAGE_DAYS_PER_MONTH, + _DURATION_TOKEN_RE, +) from graphistry.compute.gfql.temporal.truncation import _zone_compatible_local_datetime_text # noqa: E402 diff --git a/graphistry/compute/gfql/temporal/durations.py b/graphistry/compute/gfql/temporal/durations.py index 1c8d6dc239..62deaf8fae 100644 --- a/graphistry/compute/gfql/temporal/durations.py +++ b/graphistry/compute/gfql/temporal/durations.py @@ -76,6 +76,11 @@ def parse_temporal_sort_duration_components(text: str) -> Optional[tuple[int, in _NANOS_PER_DAY = 24 * 60 * 60 * 1_000_000_000 +_AVERAGE_NANOS_PER_MONTH = 2_629_746 * 1_000_000_000 +"""The only length a fractional month has: 365.2425 / 12 days, i.e. 30.436875 days.""" + +_AVERAGE_DAYS_PER_MONTH = Decimal(_AVERAGE_NANOS_PER_MONTH) / Decimal(_NANOS_PER_DAY) + def parse_duration_calendar_components(text: str) -> Optional[tuple[int, int, int]]: """``(months, days, time_nanoseconds)`` for an ISO-8601 duration literal, or None. diff --git a/graphistry/compute/gfql/temporal/folding.py b/graphistry/compute/gfql/temporal/folding.py index 58b1b3fe94..f4e8cb5edc 100644 --- a/graphistry/compute/gfql/temporal/folding.py +++ b/graphistry/compute/gfql/temporal/folding.py @@ -14,15 +14,21 @@ _rebuild_expr_node, ) from graphistry.compute.gfql.temporal.durations import ( + _AVERAGE_NANOS_PER_MONTH, _NANOS_PER_DAY, _fold_duration_function_call, format_duration_calendar_components, parse_duration_calendar_components, ) +from graphistry.compute.gfql.language_defs import ( + GFQL_COMPARISON_BINARY_OPS, + GFQL_INEQUALITY_EQUALITY_COMPARISON_BINARY_OPS, +) from graphistry.compute.gfql.temporal.rendering import _render_temporal_arg from graphistry.compute.gfql.temporal.truncation import _fold_temporal_truncate_call from graphistry.compute.gfql.temporal.values import ( _TemporalValue, + _days_from_civil, _days_in_month, _format_localdatetime_parts, _format_localtime_parts, @@ -130,21 +136,96 @@ def _shift_temporal_value(value: _TemporalValue, months: int, days: int, time_na return rendered -def _scale_duration(components: tuple[int, int, int], factor: float, divide: bool) -> Optional[str]: - """Scale each duration group IN PLACE: the time group never migrates into days - (PT18H * 2 is PT36H, and date + PT36H is a no-op while date + P1DT12H is not); - only a fractional day result spills into time (P1D / 2 is PT12H).""" +def _scale_duration(components: tuple[int, int, int], factor: float, divide: bool) -> str: + """Scale each duration group IN PLACE, then cascade DOWNWARD whatever no longer fits + that group: a fractional month becomes days at the average month of 30.436875 days, + and a fractional day becomes time (P1M / 2 is P15DT5H14M33S, P1D / 2 is PT12H). + Nothing ever migrates UP -- the time group stays put (PT18H * 2 is PT36H, and date + + PT36H is a no-op while date + P1DT12H is not) and days are never re-absorbed into + months, whose length varies. Each group TRUNCATES toward zero and hands the exact + remainder down, so negatives mirror their positive twin and PT2S / 3 is + PT0.666666666S, not ...667S. A result that is whole in month-space keeps its months + (P2M / 2 is P1M), so the average month is used only where a month must actually be + split -- and there scaling stops round-tripping: (P1M / 2) * 2 is P30DT10H29M6S.""" months, days, time_nanos = components scaled_months = (months / factor) if divide else (months * factor) - if scaled_months != int(scaled_months): - return None - scaled_days = (days / factor) if divide else (days * factor) + whole_months = int(scaled_months) + month_spill_days = _AVERAGE_NANOS_PER_MONTH * (scaled_months - whole_months) / _NANOS_PER_DAY + scaled_days = ((days / factor) if divide else (days * factor)) + month_spill_days whole_days = int(scaled_days) - day_spill_nanos = (scaled_days - whole_days) * _NANOS_PER_DAY + day_spill_nanos = _NANOS_PER_DAY * (scaled_days - whole_days) scaled_time = (time_nanos / factor) if divide else (time_nanos * factor) return format_duration_calendar_components( - int(scaled_months), whole_days, int(round(scaled_time + day_spill_nanos)) + whole_months, whole_days, int(scaled_time + day_spill_nanos) + ) + + +_TZ_OFFSET_PREFIX_RE = re.compile(r"^(Z|[+-]\d{2}:\d{2}(?::\d{2})?)") + +_NANOS_PER_SECOND = 1_000_000_000 + + +def _temporal_instant_key(value: _TemporalValue) -> Optional[tuple[str, int]]: + """(temporal kind, instant nanoseconds) for a parsed temporal literal. + + Zoned kinds normalize to UTC ("compared on a global timeline"); a bare + ``[zone]`` suffix with no resolvable offset returns None (no fold).""" + offset_nanos = 0 + if value.kind in {"datetime", "time"}: + match = _TZ_OFFSET_PREFIX_RE.match(value.tz_suffix or "") + if match is None: + return None + token = match.group(1) + if token != "Z": + sign = -1 if token[0] == "-" else 1 + seconds = int(token[1:3]) * 3600 + int(token[4:6]) * 60 + if len(token) > 6: + seconds += int(token[7:9]) + offset_nanos = sign * seconds * _NANOS_PER_SECOND + days = 0 + if value.date_value is not None: + days = _days_from_civil(value.date_value.year, value.date_value.month, value.date_value.day) + civil_nanos = ( + days * _NANOS_PER_DAY + + (value.hour * 3600 + value.minute * 60 + value.second) * _NANOS_PER_SECOND + + value.nanosecond ) + return (value.kind, civil_nanos - offset_nanos) + + +def _fold_temporal_comparison(node: BinaryOp) -> Optional[Literal]: + """Constant-fold `` ``. + + openCypher CIP2016-06-14: zoned values compare on the UTC global timeline + (same instant under different offsets IS equal), while values of different + temporal types are never equal (`=` false) and are incomparable for + ordering (`<` etc. null). Both engines otherwise diverge here — pandas + instant-compared across types, polars compared rendered text.""" + cmp_fn = GFQL_COMPARISON_BINARY_OPS.get(str(node.op)) + if cmp_fn is None: + return None + if not (isinstance(node.left, Literal) and isinstance(node.right, Literal)): + return None + if not (isinstance(node.left.value, str) and isinstance(node.right.value, str)): + return None + try: + left_value = _parse_temporal_value(node.left.value) + right_value = _parse_temporal_value(node.right.value) + except ValueError: + return None + if left_value is None or right_value is None: + return None + left_key = _temporal_instant_key(left_value) + right_key = _temporal_instant_key(right_value) + if left_key is None or right_key is None: + return None + if left_key[0] != right_key[0]: + if str(node.op) == "=": + return Literal(False) + if str(node.op) in GFQL_INEQUALITY_EQUALITY_COMPARISON_BINARY_OPS: + return Literal(True) + return Literal(None) + return Literal(bool(cmp_fn(left_key[1], right_key[1]))) def _fold_temporal_arithmetic(node: BinaryOp) -> Optional[Literal]: @@ -184,14 +265,12 @@ def _number_of(value: object) -> Optional[float]: # Multiplying by zero is PT0S; only DIVISION by zero declines. if factor is None or (factor == 0 and op == "/"): return None - scaled = _scale_duration(left_duration, factor, divide=(op == "/")) - return None if scaled is None else Literal(scaled) + return Literal(_scale_duration(left_duration, factor, divide=(op == "/"))) if right_duration is not None and left_duration is None and op == "*": factor = _number_of(left_value) if factor is None: return None - scaled = _scale_duration(right_duration, factor, divide=False) - return None if scaled is None else Literal(scaled) + return Literal(_scale_duration(right_duration, factor, divide=False)) return None sign = -1 if op == "-" else 1 @@ -328,6 +407,9 @@ def _fold(inner: ExprNode) -> ExprNode: arithmetic = _fold_temporal_arithmetic(rebuilt) if arithmetic is not None: return arithmetic + comparison = _fold_temporal_comparison(rebuilt) + if comparison is not None: + return comparison return rebuilt return _fold(node) diff --git a/graphistry/compute/gfql_fast_paths.py b/graphistry/compute/gfql_fast_paths.py index 1e830a9a5e..c67b594c46 100644 --- a/graphistry/compute/gfql_fast_paths.py +++ b/graphistry/compute/gfql_fast_paths.py @@ -21,7 +21,7 @@ from graphistry.compute.typing import ArrayLike, ArrayNamespace from graphistry.Engine import Engine, EngineAbstract, POLARS_ENGINES, df_concat, df_cons, df_to_engine, df_unique, resolve_engine from graphistry.util import setup_logger -from .ast import ASTObject, ASTLet, ASTNode, ASTEdge, ASTCall +from .ast import ASTObject, ASTLet, ASTNode, ASTEdge, ASTCall, serialize_binding_ops from .chain import Chain, chain as chain_impl from .gfql.query_types import GFQLQuery from .chain_let import chain_let as chain_let_impl @@ -45,9 +45,12 @@ from graphistry.compute.gfql.agg_types import ( GFQL_NUMERIC_ONLY_AGGREGATIONS, numeric_agg_all_null_value, + pandas_agg_kernel_null_fill, pandas_dtype_is_numeric_for_agg, pandas_non_numeric_agg_dtype, pandas_object_series_is_bool_like, + polars_all_null_agg_literal, + polars_conform_agg_dtype, polars_non_numeric_agg_dtype, raise_non_numeric_aggregation, ) @@ -1931,7 +1934,8 @@ def _low_cardinality_pure_count_plan( is a decline away from zero. The two formulations are VALUE-IDENTICAL wherever this admits -- same key rows, same - counts, same ``UInt32`` count dtype, same treatment of null / NaN / empty-input keys -- + counts, same INTEGER count dtype (both conformed off agg_types), same treatment of + null / NaN / empty-input keys -- and the caller's gate has already made the following ``sort`` TOTAL over the output rows, so neither formulation's internal row order can reach the answer. Choosing between them is a routing decision, not a semantic one. @@ -1992,7 +1996,9 @@ def _low_cardinality_pure_count_plan( # ``name=`` (polars >= 1.0, and the declared floor is 1.29) keeps the count column out # of a rename, so a group key literally named ``count`` is served rather than crashing. - return work_lf.select(pl.col(group_key).value_counts(name=out_alias)).unnest(group_key) + counts = work_lf.select(pl.col(group_key).value_counts(name=out_alias)).unnest(group_key) + return counts.with_columns( + polars_conform_agg_dtype(pl.col(out_alias), "count", None, out_alias)) def _single_hop_grouped_aggregate_fused_polars( @@ -2118,16 +2124,19 @@ def _single_hop_grouped_aggregate_fused_polars( or polars_non_numeric_agg_dtype(agg_dtype) is not None ): return None + result_dtype = prop_dtypes.get(expr_col) if expr_col is not None else None if func == "count" and expr_col is None: - agg_exprs.append(pl.len().alias(out_alias)) + agg_exprs.append(polars_conform_agg_dtype(pl.len(), func, None, out_alias)) elif expr_col is None: return None elif func == "count": - agg_exprs.append(pl.col(expr_col).count().alias(out_alias)) + agg_exprs.append( + polars_conform_agg_dtype(pl.col(expr_col).count(), func, result_dtype, out_alias)) elif func == "avg": agg_exprs.append(pl.col(expr_col).mean().alias(out_alias)) elif func == "sum": - agg_exprs.append(pl.col(expr_col).sum().alias(out_alias)) + agg_exprs.append( + polars_conform_agg_dtype(pl.col(expr_col).sum(), func, result_dtype, out_alias)) elif func == "min": agg_exprs.append(pl.col(expr_col).min().alias(out_alias)) elif func == "max": @@ -2256,7 +2265,11 @@ def _execute_single_hop_grouped_aggregate_fast_path( chain: Chain, *, engine: Union[EngineAbstract, str], + reentry_start_nodes: Optional[DataFrameT] = None, ) -> Optional[Plottable]: + if reentry_start_nodes is not None: + # seed comes from filter_dicts alone; engaging would widen a carried WITH..MATCH seed + return None ops = list(chain.chain) if len(ops) not in (3, 4, 5) or not all(isinstance(op, ASTCall) for op in ops): return None @@ -2504,18 +2517,21 @@ def join_props_polars(work_df: Any, alias: str, node_df: Any, edge_col: str) -> and work.height > 0 and work[expr_alias].null_count() == work.height ): - agg_exprs.append(pl.lit(numeric_agg_all_null_value(func)).alias(alias)) + agg_exprs.append(polars_all_null_agg_literal(func, alias)) continue if dtype_label is not None: raise_non_numeric_aggregation(func, expr_alias, dtype_label, alias) + agg_dtype = work_schema.get(expr_alias) if expr_alias is not None else None if func == "count" and (expr_alias is None or with_items[expr_alias][1] is None): - agg_exprs.append(pl.len().alias(alias)) + agg_exprs.append(polars_conform_agg_dtype(pl.len(), func, None, alias)) elif func == "count" and expr_alias is not None: - agg_exprs.append(pl.col(expr_alias).count().alias(alias)) + agg_exprs.append( + polars_conform_agg_dtype(pl.col(expr_alias).count(), func, agg_dtype, alias)) elif func == "avg" and expr_alias is not None: agg_exprs.append(pl.col(expr_alias).mean().alias(alias)) elif func == "sum" and expr_alias is not None: - agg_exprs.append(pl.col(expr_alias).sum().alias(alias)) + agg_exprs.append( + polars_conform_agg_dtype(pl.col(expr_alias).sum(), func, agg_dtype, alias)) elif func == "min" and expr_alias is not None: agg_exprs.append(pl.col(expr_alias).min().alias(alias)) elif func == "max" and expr_alias is not None: @@ -2599,6 +2615,10 @@ def join_props_df(work_df: DataFrameT, alias: str, node_df: DataFrameT, edge_col if pandas_object_series_is_bool_like(work[expr_alias]): # Twin of the row-pipeline group_by sum(bool) numeric retype. agg_df = agg_df.assign(**{alias: pd.to_numeric(agg_df[alias])}) # bool sums as int (#1821) + null_fill = pandas_agg_kernel_null_fill(func, work[expr_alias]) + if null_fill is not None: + # Twin of the row-pipeline sum() null repair: cypher sum() never answers null. + agg_df = agg_df.assign(**{alias: agg_df[alias].fillna(null_fill)}) elif func == "min" and expr_alias is not None: agg_df = grouped[expr_alias].min().reset_index(name=alias) elif func == "max" and expr_alias is not None: @@ -3019,7 +3039,11 @@ def _execute_two_hop_count_fast_path( chain: Chain, *, engine: Union[EngineAbstract, str], + reentry_start_nodes: Optional[DataFrameT] = None, ) -> Optional[Plottable]: + if reentry_start_nodes is not None: + # same seed-blindness as the grouped-aggregate path above + return None alias = _two_hop_count_alias(chain) if alias is None: return None @@ -3276,6 +3300,30 @@ def _note(role: str, column: str, part: Optional[Tuple[str, "PartitionValue"]], return out +#: Join key for the seeded typed-hop bag expansion; never a user column. +_SEEDED_BAG_KEY = "__gfql_seeded_bag_key__" + + +def _seeded_typed_hop_bag_rows( + dst_rows: DataFrameT, edges: DataFrameT, *, to_col: str, node: str, is_polars: bool, +) -> DataFrameT: + """One destination-node row per matched edge (openCypher bag), not the node set. + + ``dst_rows`` is deduped by ``node`` and ``edges`` already drops the dangling + ones, so this re-expands exactly the multiplicity the dedup removed.""" + # Equal heights means the dedup removed nothing (the two sides cover each other). + if len(edges) == len(dst_rows): + return dst_rows + if is_polars: + import polars as pl + keys = edges.select(pl.col(to_col).alias(_SEEDED_BAG_KEY)) # type: ignore[union-attr] + keyed = dst_rows.with_columns(pl.col(node).alias(_SEEDED_BAG_KEY)) # type: ignore[union-attr] + return keys.join(keyed, on=_SEEDED_BAG_KEY, how="inner").drop(_SEEDED_BAG_KEY) # type: ignore[no-any-return] + keys = edges[[to_col]].rename(columns={to_col: _SEEDED_BAG_KEY}).reset_index(drop=True) # type: ignore[union-attr] + joined = keys.merge(dst_rows, left_on=_SEEDED_BAG_KEY, right_on=node, how="inner") + return joined.drop(columns=[_SEEDED_BAG_KEY]).reset_index(drop=True) # type: ignore[no-any-return] + + def _execute_seeded_typed_hop_fast_path( base_graph: Plottable, compiled_query: CompiledCypherQuery, @@ -3373,7 +3421,16 @@ def _execute_seeded_typed_hop_fast_path( # source node (n0) — the forward seeded shape MATCH (m {id})-[:T]->(p) RETURN p. # Other alias/seed placements (e.g. reverse patterns where the seed is on the # RETURN node) fall back to the full path. - return_alias = projection.alias if projection is not None else str((call.params or {}).get("source", "")) + call_params = call.params or {} + # `binding_ops` is the same seeded shape lowered to one row per matched EDGE. + binding_ops = call_params.get("binding_ops") + bag_rows = isinstance(binding_ops, list) + if bag_rows: + if serialize_binding_ops(ops[:3]) != binding_ops: + return None + return_alias = projection.alias if projection is not None else (n2._name or "") + else: + return_alias = projection.alias if projection is not None else str(call_params.get("source", "")) if n2._name != return_alias: return None select_items: Optional[list] = None @@ -3460,6 +3517,13 @@ def _execute_seeded_typed_hop_fast_path( hop_details=[{"hop": 1}] if index_ctx is not None else None, ) p_rows, _edges = dst_res + if bag_rows: + p_rows = _seeded_typed_hop_bag_rows( + p_rows, _edges, + to_col=dst if direction == "forward" else src, node=node, is_polars=is_polars, + ) + if projection is not None and len(p_rows) == 0: + return None if select_items is not None: # Lean property projection (IS5 shape): the deduped destination rows carry # the raw property columns — rename/select directly, same values the @@ -3476,11 +3540,13 @@ def _execute_seeded_typed_hop_fast_path( # dtypes like nullable Int64/StringDtype, categoricals) declines to the # full path rather than risk a silent dtype divergence. import numpy as np - # The upcast above is a PANDAS pivot artifact. cuDF's rows-pivot keeps - # the source dtypes (verified: int64 stays int64, bool stays bool), so - # applying the pandas casts there would diverge from its own canonical - # path. The dtype-class decline guard still applies to both. - is_cudf_rows = "cudf" in type(p_rows).__module__ + # The upcast above is a PANDAS rows-pivot artifact, so it applies only where + # the canonical path IS that pivot: cuDF's pivot keeps the source dtypes, and + # an INDEXED bag lowering is served by the indexed connected-bindings kernel, + # which never pivots (int64/bool, as polars and cuDF already answer). + # The dtype-class decline guard still applies to all of them. + canonical_keeps_source_dtypes = ( + "cudf" in type(p_rows).__module__ or (bag_rows and index_ctx is not None)) casts: Dict[str, str] = {} for out_name, prop in select_items: if prop == node: @@ -3491,10 +3557,10 @@ def _execute_seeded_typed_hop_fast_path( if not isinstance(d, np.dtype): return None if d == np.dtype(bool): - if not is_cudf_rows: + if not canonical_keeps_source_dtypes: casts[out_name] = "object" elif d.kind in "iuf": - if not is_cudf_rows: + if not canonical_keeps_source_dtypes: casts[out_name] = "float64" elif d.kind != "O": return None diff --git a/graphistry/compute/gfql_unified.py b/graphistry/compute/gfql_unified.py index 82517d843a..de6b17aa03 100644 --- a/graphistry/compute/gfql_unified.py +++ b/graphistry/compute/gfql_unified.py @@ -7,7 +7,7 @@ import threading import pandas as pd from types import MappingProxyType -from typing import Any, Callable, Dict, List, Literal, Mapping, Optional, Sequence, Set, Tuple, TYPE_CHECKING, Union, cast +from typing import Any, Callable, Dict, List, Literal, Mapping, Optional, Sequence, Set, Tuple, TYPE_CHECKING, TypeVar, Union, cast from graphistry.Plottable import Plottable from graphistry.Engine import Engine, EngineAbstract, POLARS_ENGINES, df_concat, df_cons, df_to_engine, df_unique, is_polars_df, is_series_like, resolve_engine, series_to_pylist from graphistry.util import setup_logger @@ -61,6 +61,7 @@ compiled_query_scalar_reentry_state as _compiled_query_scalar_reentry_state, freeform_broadcast_row_to_nodes as _freeform_broadcast_row_to_nodes, reentry_validation_error as _reentry_validation_error, + restrict_connected_join_rows_to_reentry_seed as _restrict_connected_join_rows_to_reentry_seed, union_scalar_reentry_results as _union_scalar_reentry_results, ) from graphistry.compute.gfql.cypher.call_procedures import CompiledCypherProcedureCall, execute_cypher_call @@ -94,6 +95,7 @@ from graphistry.compute.gfql.identifiers import EDGE_INDEX_BASE from graphistry.compute.validate.validate_schema import validate_chain_schema from graphistry.compute.gfql_validate import gfql_validate as gfql_preflight_validate +from graphistry.compute.gfql.strictness import StrictInput, StrictLevel from graphistry.otel import otel_traced, otel_detail_enabled logger = setup_logger(__name__) @@ -744,6 +746,8 @@ def _apply_connected_match_join( engine: Union[EngineAbstract, str], policy: Optional[PolicyDict], context: ExecutionContext, + start_nodes: Optional[DataFrameT] = None, + reentry_alias: Optional[str] = None, ) -> Plottable: from graphistry.compute.ast import ASTCall, ASTNode as _ASTNode, serialize_binding_ops @@ -757,6 +761,11 @@ def _apply_connected_match_join( # recomputes instead of returning a stale cached answer (BLOCKER 1). cache_store: Dict[str, Any] = {} + for pattern_chain in plan.pattern_chains: + _reject_node_alias_shadowing_id_binding( + base_graph, pattern_chain, include_edge_endpoint_aliases=True + ) + trail_identity_col = _trail_edge_identity_col(base_graph) trail_arms = _connected_join_trail_arms(plan, identity_col=trail_identity_col) arms_may_share_an_edge = trail_arms is not None @@ -769,10 +778,14 @@ def _apply_connected_match_join( base_graph, engine=requested_engine, identity_col=trail_identity_col ) - # Both two-star fast paths emit the raw arm product, so they serve disjoint arms only. + # Both two-star fast paths serve only disjoint, unseeded arms (their seeds are filter_dicts-only) fast_grouped_count = ( - None if arms_may_share_an_edge - else _connected_join_two_star_fast_grouped_count(base_graph, plan, engine=requested_engine, cache_store=cache_store) + None if arms_may_share_an_edge or start_nodes is not None + else _run_fast_path_on_requested_target( + engine, + lambda: _connected_join_two_star_fast_grouped_count( + base_graph, plan, engine=requested_engine, cache_store=cache_store), + )[0] ) if fast_grouped_count is not None: out = base_graph.bind() @@ -781,8 +794,12 @@ def _apply_connected_match_join( return out fast_rows = ( - None if arms_may_share_an_edge - else _connected_join_two_star_fast_rows(base_graph, plan, engine=requested_engine, cache_store=cache_store) + None if arms_may_share_an_edge or start_nodes is not None + else _run_fast_path_on_requested_target( + engine, + lambda: _connected_join_two_star_fast_rows( + base_graph, plan, engine=requested_engine, cache_store=cache_store), + )[0] ) if fast_rows is not None: if len(fast_rows) == 0: @@ -888,6 +905,14 @@ def _apply_connected_match_join( else joined_rows.drop(columns=drop_columns)) joined_rows = _joined_hidden_scalar_columns(joined_rows) joined_rows = _joined_alias_columns(joined_rows) + if start_nodes is not None: + # the arms above re-matched from the whole graph; narrow to the carried seeds + joined_rows = _restrict_connected_join_rows_to_reentry_seed( + joined_rows, + start_nodes=start_nodes, + reentry_alias=reentry_alias, + node_col=node_col, + ) joined_plottable = base_graph.bind() joined_plottable._nodes = joined_rows joined_plottable._edges = df_ctor() @@ -1165,7 +1190,12 @@ def _concat_union_branch_rows( concat: Callable[..., DataFrameT], ) -> DataFrameT: """Row-concat UNION branch frames without letting a branch's dtype rewrite another's values.""" - frames = list(row_frames) + # UNION aligns columns by NAME (Neo4j); the output keeps the first branch's order. + first_columns = list(row_frames[0].columns) + frames = [ + frame if list(frame.columns) == first_columns else frame[first_columns] + for frame in row_frames + ] non_empty = [frame for frame in frames if len(frame) > 0] if non_empty and len(non_empty) != len(frames): # A 0-row branch contributes no rows, and its dtype must not drag the union supertype. @@ -1355,12 +1385,48 @@ def _policied_auto_serves_via_pandas_until_the_polars_route_emits_hooks( ) -def _fast_path_execution_target_ignoring_requested_engine( +def _fast_path_execution_target( engine: Union[EngineAbstract, Engine, str], ) -> "ExecutionTarget": - """Not GPU until every fast-path arm is GPU-or-decline (#1824).""" + """Lazy-collect target for a Cypher fast path: GPU only when ``polars-gpu`` was requested. + + Comparing the requested value IS the "did the caller ask for GPU" test, and needs no + graph: ``resolve_engine`` never produces ``Engine.POLARS_GPU`` from ``AUTO``, and the + AUTO cuDF route that does target GPU re-enters ``gfql`` with the engine already pinned + to ``polars-gpu``, so it arrives here as an explicit request. A request for GPU runs on + GPU or raises; it is never quietly served on CPU. + """ from graphistry.compute.gfql.lazy import ExecutionTarget - return ExecutionTarget.CPU + requested = engine.value if isinstance(engine, (Engine, EngineAbstract)) else engine + return ExecutionTarget.GPU if requested == Engine.POLARS_GPU.value else ExecutionTarget.CPU + + +_FastPathOut = TypeVar("_FastPathOut") + + +def _run_fast_path_on_requested_target( + engine: Union[EngineAbstract, Engine, str], + run: Callable[[], Optional[_FastPathOut]], +) -> Tuple[Optional[_FastPathOut], str]: + """Run one Cypher fast path with its lazy collects on the requested engine's target. + + ONE seam for every fast-path arm so the arms cannot drift onto different targets. Returns + ``(result, reason)``; a ``None`` result is a decline the caller falls back from. On the GPU + target a non-GPU-executable plan surfaces as ``NotImplementedError`` (``lazy._gpu_raise``) + and is converted to a decline, so the caller's generic route re-runs the shape on the SAME + GPU target -- which itself is GPU-or-raise, never a silent CPU answer. On the CPU target a + ``NotImplementedError`` is a real bug and propagates. + """ + from graphistry.compute.gfql.lazy import ExecutionTarget, target_mode + target = _fast_path_execution_target(engine) + try: + with target_mode(target): + out = run() + except NotImplementedError: + if target != ExecutionTarget.GPU: + raise + return None, "declined; plan not GPU-executable, generic route answers" + return out, ("served" if out is not None else "declined; caller falls back") def _execute_compiled_query_via_physical_plan( @@ -1384,6 +1450,11 @@ def _execute_compiled_query_via_physical_plan( engine=engine, policy=policy, context=context, + start_nodes=start_nodes, + reentry_alias=( + None if compiled_query.reentry_plan is None + else compiled_query.reentry_plan.reentry_alias_name + ), ) if connected_optional_match is not None: @@ -1400,33 +1471,25 @@ def _execute_compiled_query_via_physical_plan( # this is where the decision is consumed, it is one place instead of N return # paths, and it cannot be bypassed the way patching a directly-imported name is. from graphistry.compute.gfql.index.api import record_fast_path_decision - from graphistry.compute.gfql.lazy import ExecutionTarget, target_mode - _fp_target = _fast_path_execution_target_ignoring_requested_engine(engine) _FastPathName = Literal["single_hop_grouped_aggregate", "two_hop_count", "seeded_typed_hop"] def _try_fast(path_name: _FastPathName, run: Callable[[], Optional[Plottable]]) -> Optional[Plottable]: - try: - with target_mode(_fp_target): - out = run() - reason = "served" if out is not None else "declined; caller falls back" - except NotImplementedError: - if _fp_target != ExecutionTarget.GPU: - raise - out = None - reason = "declined; plan not GPU-executable, chain route answers" + out, reason = _run_fast_path_on_requested_target(engine, run) record_fast_path_decision( path=path_name, engine=engine, served=out is not None, reason=reason) return out fast_grouped = _try_fast( "single_hop_grouped_aggregate", - lambda: _execute_single_hop_grouped_aggregate_fast_path(base_graph, compiled_query.chain, engine=engine)) + lambda: _execute_single_hop_grouped_aggregate_fast_path( + base_graph, compiled_query.chain, engine=engine, reentry_start_nodes=start_nodes)) if fast_grouped is not None: return fast_grouped fast_count = _try_fast( "two_hop_count", - lambda: _execute_two_hop_count_fast_path(base_graph, compiled_query.chain, engine=engine)) + lambda: _execute_two_hop_count_fast_path( + base_graph, compiled_query.chain, engine=engine, reentry_start_nodes=start_nodes)) if fast_count is not None: return fast_count fast_hop = _try_fast( @@ -1519,6 +1582,9 @@ def _execute_compiled_query_chain_non_union( compiled_query=compiled_query, engine=engine, ) + _reject_node_alias_shadowing_id_binding( + base_graph, compiled_query.chain, include_edge_endpoint_aliases=True + ) # #1712: a bounded-reentry main chain that is a binding-ops row pipeline # (rows(binding_ops) -> group_by -> ...) must seed its first alias from the @@ -2330,7 +2396,8 @@ def gfql(self: Plottable, language: Optional[Literal["cypher", "gremlin"]] = None, params: Optional[CypherParams] = None, validate: bool = False, - shortest_path_backend: str = "auto") -> Plottable: + shortest_path_backend: str = "auto", + strict: StrictInput = None) -> Plottable: """ Execute a GFQL query - either a chain or a DAG @@ -2349,9 +2416,39 @@ def gfql(self: Plottable, ``"igraph"`` (require igraph, raise if missing), ``"cugraph"`` (require cugraph, raise if missing), or ``"bfs"`` (always use DataFrame BFS). ``"auto"`` tries cugraph on CUDF engine, igraph on pandas, falls back to BFS silently. + :param strict: Absent-label/property strictness: ``"strict"`` raises, ``"warn"`` + (default) warns once per absent name and resolves it to null (openCypher), + ``"quiet"`` resolves silently. ``True``/``False`` map to ``"strict"``/``"quiet"``. + ``None`` consults ``bind(schema=...)``, then the ``"warn"`` default. :returns: Resulting Plottable :rtype: Plottable """ + from graphistry.compute.gfql.strictness import ( + resolve_strict_level, schema_declared_names, strictness_scope) + + with strictness_scope( + resolve_strict_level(self, strict=strict), declared=schema_declared_names(self) + ) as _strictness: + return _gfql_with_strictness( + self, query, engine=engine, output=output, policy=policy, where=where, + language=language, params=params, validate=validate, + shortest_path_backend=shortest_path_backend, strict=_strictness.level) + + +def _gfql_with_strictness( + self: Plottable, + query: GFQLQuery, + *, + engine: Union[EngineAbstract, str], + output: Optional[str], + policy: Optional[Dict[str, PolicyFunction]], + where: Optional[Sequence[WhereComparison]], + language: Optional[Literal["cypher", "gremlin"]], + params: Optional[CypherParams], + validate: bool, + shortest_path_backend: str, + strict: StrictLevel, +) -> Plottable: if _policied_auto_serves_via_pandas_until_the_polars_route_emits_hooks(engine, policy, self): engine = Engine.PANDAS.value @@ -2376,9 +2473,8 @@ def gfql(self: Plottable, shortest_path_backend=shortest_path_backend, ) - # engine inference, cuDF arm (owner-directed policy addition, 2026-08-02; supersedes the - # earlier "AUTO never selects polars-gpu" doctrine for THIS arm only): when every bound - # frame is cuDF AND the cudf-polars GPU target is GENUINELY usable (probed once per + # engine inference, cuDF arm: when every bound frame is cuDF AND the cudf-polars + # GPU target is GENUINELY usable (probed once per # process — polars imports, cudf + cudf_polars installed, and a real GPU collect # succeeds; see lazy.polars_gpu_available), prefer the native lazy polars engine on its # GPU execution target over the legacy CUDF path. Both serve cudf->cudf: inputs cross @@ -2489,7 +2585,7 @@ def gfql(self: Plottable, where=where_param, language=language, params=params, - strict=True, + strict=strict, schema=True, collect_all=False, ) @@ -2660,7 +2756,9 @@ def gfql(self: Plottable, context.policy_depth = policy_depth -def _reject_node_alias_shadowing_id_binding(g: Plottable, chain_obj: Chain) -> None: +def _reject_node_alias_shadowing_id_binding( + g: Plottable, chain_obj: Chain, *, include_edge_endpoint_aliases: bool = False +) -> None: """Typed decline for a node alias named after the node-ID binding column. The alias marker is stamped as `` = True``, so an alias equal to the node-id @@ -2669,10 +2767,12 @@ def _reject_node_alias_shadowing_id_binding(g: Plottable, chain_obj: Chain) -> N polars answered ``True``. Neither is a usable result; decline the same way on both. """ node_id = getattr(g, "_node", None) - if not isinstance(node_id, str): - return + endpoint_cols = { + col for col in (getattr(g, "_source", None), getattr(g, "_destination", None)) + if isinstance(col, str) + } for op in chain_obj.chain: - if isinstance(op, ASTNode) and getattr(op, "_name", None) == node_id: + if isinstance(node_id, str) and isinstance(op, ASTNode) and getattr(op, "_name", None) == node_id: raise GFQLValidationError( ErrorCode.E108, "A node alias cannot be named after the node-ID binding column", @@ -2683,6 +2783,23 @@ def _reject_node_alias_shadowing_id_binding(g: Plottable, chain_obj: Chain) -> N f"overwrite the node-ID binding. Rename the alias." ), ) + # Cypher-only decline; raw GFQL chains keep their documented overwrite parity. + if ( + include_edge_endpoint_aliases + and isinstance(op, ASTEdge) + and getattr(op, "_name", None) in endpoint_cols + ): + raise GFQLValidationError( + ErrorCode.E108, + "An edge alias cannot be named after an edge endpoint binding column", + field="chain.name", + value=getattr(op, "_name", None), + suggestion=( + "The alias flag is materialized as a column named like the edge " + "source/destination binding, which would overwrite the endpoints. " + "Rename the alias." + ), + ) def _chain_dispatch( diff --git a/graphistry/compute/gfql_validate.py b/graphistry/compute/gfql_validate.py index 76b94bd60e..723d5fb91e 100644 --- a/graphistry/compute/gfql_validate.py +++ b/graphistry/compute/gfql_validate.py @@ -19,12 +19,20 @@ from graphistry.compute.gfql.frontends.cypher.binder import FrontendBinder from graphistry.compute.gfql.ir.compilation import GraphSchemaCatalog, PlanContext from graphistry.compute.gfql.query_types import GFQLQuery +from graphistry.compute.gfql.strictness import ( + StrictInput, + absent_name_is_lenient, + resolve_strict_level, + schema_declared_names, + strict_level_to_bool, + strictness_scope, +) from graphistry.compute.gfql.same_path_types import ( WhereComparison, normalize_where_entries, parse_where_json, ) -from graphistry.compute.validate.validate_schema import validate_chain_schema +from graphistry.compute.validate.validate_schema import validate_chain_schema, validate_graph_shape GFQLValidationQuery = GFQLQuery @@ -122,18 +130,18 @@ def _build_schema_catalog(g: Plottable, *, strict: Optional[bool]) -> GraphSchem ) -def _resolve_strict_mode(g: Plottable, *, strict: Optional[bool]) -> bool: - if strict is not None: - return bool(strict) - bound_schema = getattr(g, "_gfql_schema", None) - if bound_schema is not None: - schema_strict = getattr(bound_schema, "strict", None) - if schema_strict is not None: - return bool(schema_strict) - metadata = getattr(bound_schema, "metadata", None) - if isinstance(metadata, Mapping) and "strict" in metadata: - return bool(metadata["strict"]) - return True +def _resolve_strict_mode(g: Plottable, *, strict: StrictInput) -> bool: + """Legacy boolean view of the shared level resolution.""" + return strict_level_to_bool(resolve_strict_level(g, strict=strict)) + + +def _bind_names(g: Plottable, parsed: Any) -> None: # hygiene-ok: explicit-any -- parsed is the cypher parser's AST root + """Strict name resolution against the bound schema, or this instance's columns.""" + FrontendBinder().bind( + parsed, + PlanContext(catalog=_build_schema_catalog(g, strict=True)), + strict_name_resolution=True, + ) def _validate_cypher( @@ -141,21 +149,37 @@ def _validate_cypher( query: str, *, params: Optional[Mapping[str, Any]], - strict: Optional[bool], + strict: StrictInput, + schema: bool = True, ) -> Dict[str, Any]: parsed = parse_cypher(query) - strict_mode = _resolve_strict_mode(g, strict=strict) - if strict_mode: - strict_ctx = PlanContext(catalog=_build_schema_catalog(g, strict=strict)) - FrontendBinder().bind(parsed, strict_ctx, strict_name_resolution=True) + level = resolve_strict_level(g, strict=strict) + declared = schema_declared_names(g) + can_judge_names = schema or declared is not None # schema=False holds no frames; a declared schema is names without data + if level != "quiet" and can_judge_names: + try: + _bind_names(g, parsed) + except GFQLValidationError as exc: + ctx: Mapping[str, object] = exc.context or {} + name = str(ctx.get("value") or ctx.get("field") or "") + with strictness_scope(level, declared=declared): + if not absent_name_is_lenient(name, kind="name", context="query"): + raise compiled = compile_cypher_query(parsed, params=params) compiled_kind: Literal["query", "union", "graph"] = "query" if isinstance(compiled, CompiledCypherUnionQuery): compiled_kind = "union" + compiled_chains = [branch.chain for branch in compiled.branches] elif isinstance(compiled, CompiledCypherGraphQuery): compiled_kind = "graph" + compiled_chains = [compiled.chain] + [b.chain for b in compiled.graph_bindings] else: compiled = cast(CompiledCypherQuery, compiled) + compiled_chains = [compiled.chain] + if schema: + # schema=False callers (e.g. remote execution) hold no local frames to judge + for compiled_chain in compiled_chains: + validate_graph_shape(g, compiled_chain.chain, collect_all=False) # runtime declines these too (#1889) return { "ok": True, "query_type": "chain", @@ -378,7 +402,7 @@ def gfql_validate( where: Optional[Sequence[WhereComparison]] = None, language: Optional[Literal["cypher", "gremlin"]] = None, params: Optional[Mapping[str, Any]] = None, - strict: Optional[bool] = None, + strict: StrictInput = None, collect_all: bool = False, schema: bool = True, ) -> Dict[str, Any]: @@ -386,6 +410,10 @@ def gfql_validate( Raises structured GFQL exceptions on validation failures and never dispatches query execution operators. + + :param strict: Absent-name strictness -- ``"strict"`` raises, ``"warn"`` (default) + warns once per absent name, ``"quiet"`` is silent. ``True``/``False`` map to + ``"strict"``/``"quiet"``; ``None`` consults ``bind(schema=...)`` then the default. """ try: if isinstance(query, str): @@ -401,7 +429,7 @@ def gfql_validate( suggestion="Use language='cypher' for now; Gremlin string compilation is not implemented yet.", language="gfql", ) - return _validate_cypher(g, query, params=params, strict=strict) + return _validate_cypher(g, query, params=params, strict=strict, schema=schema) if language is not None: raise ValueError("language is only supported when query is a string") diff --git a/graphistry/compute/hop.py b/graphistry/compute/hop.py index 17494ed7cb..ba5db818b3 100644 --- a/graphistry/compute/hop.py +++ b/graphistry/compute/hop.py @@ -15,6 +15,7 @@ from .filter_by_dict import filter_by_dict from graphistry.Engine import safe_merge from .typing import DataFrameT, DomainT, SeriesT +from .endpoint_utils import drop_null_endpoint_edges from .dataframe_utils import column_frame, column_values from .util import generate_safe_column_name @@ -430,6 +431,8 @@ def _domain_union(left: Optional[DomainT], right: Optional[DomainT]) -> Optional if EDGE_ID not in edges_indexed.columns: raise ValueError(f"Edge binding column '{EDGE_ID}' (from g._edge='{g2._edge}') not found in edges. Available columns: {list(edges_indexed.columns)}") + edges_indexed = drop_null_endpoint_edges(edges_indexed, source_col, destination_col) + def resolve_label_col(requested: Optional[str], df, default_base: str) -> Optional[str]: if requested is None: return generate_safe_column_name(default_base, df, prefix='__gfqlhop_', suffix='__') @@ -866,18 +869,20 @@ def _build_pairs(src_col: str, dst_col: str) -> DataFrameT: edge_records_with_endpoints[edge_hop_col] == hop_level ] + # An edge at >= min_hops ends a qualifying walk itself; only sub-min levels feed one. + level_is_goal = hop_level >= resolved_min_hops # fixes #1944 new_node_hops = None if direction == 'forward': - reaching_edges = hop_edges[hop_edges[g2._destination].isin(current_targets)] + reaching_edges = hop_edges if level_is_goal else hop_edges[hop_edges[g2._destination].isin(current_targets)] new_source_series = reaching_edges[g2._source] new_node_hops = reaching_edges[[g2._source]].rename(columns={g2._source: node_col}) elif direction == 'reverse': - reaching_edges = hop_edges[hop_edges[g2._source].isin(current_targets)] + reaching_edges = hop_edges if level_is_goal else hop_edges[hop_edges[g2._source].isin(current_targets)] new_source_series = reaching_edges[g2._destination] new_node_hops = reaching_edges[[g2._destination]].rename(columns={g2._destination: node_col}) else: - reaching_fwd = hop_edges[hop_edges[g2._destination].isin(current_targets)] - reaching_rev = hop_edges[hop_edges[g2._source].isin(current_targets)] + reaching_fwd = hop_edges if level_is_goal else hop_edges[hop_edges[g2._destination].isin(current_targets)] + reaching_rev = hop_edges if level_is_goal else hop_edges[hop_edges[g2._source].isin(current_targets)] reaching_edges = concat([reaching_fwd, reaching_rev], ignore_index=True, sort=False).drop_duplicates(subset=[EDGE_ID]) new_source_series = concat([ reaching_fwd[g2._source], diff --git a/graphistry/compute/predicates/str.py b/graphistry/compute/predicates/str.py index 10a547c9eb..307ef6152f 100644 --- a/graphistry/compute/predicates/str.py +++ b/graphistry/compute/predicates/str.py @@ -13,6 +13,9 @@ def _series_supports_str_ops(s: Any) -> bool: A numeric, temporal, or boolean column does NOT: pandas and cuDF both raise on ``s.str`` attribute access for those dtypes. Used to make the string predicates value-safe instead of surfacing an opaque ``AttributeError: Can only use .str accessor with string values!``. + + ENGINE-DIVERGENT for categorical-of-str: pandas exposes ``.str`` on it, cuDF does not. + Callers must go through ``_str_ops_series`` so the two engines answer the same. """ try: s.str # the accessor validates dtype on attribute access @@ -21,6 +24,41 @@ def _series_supports_str_ops(s: Any) -> bool: return True +def _categories_are_strings(s: SeriesT) -> bool: + """True iff ``s`` is a categorical whose CATEGORIES are strings. + + Only string categories may be decategorized for ``.str``: a numeric/temporal categorical + would have to be stringified, and that rendering diverges pandas<->cuDF. + """ + import pandas.api.types as pd_types + cats = getattr(getattr(s, 'dtype', None), 'categories', None) + if cats is None: + return False + cat_dtype = getattr(cats, 'dtype', None) + if cat_dtype is None: + return False + return bool(pd_types.is_string_dtype(cat_dtype)) or bool(pd_types.is_object_dtype(cat_dtype)) + + +def _str_ops_series(s: SeriesT) -> Optional[SeriesT]: + """``s`` rendered so ``.str`` works, or None when the column is not string-valued. + + A categorical-of-str is string-VALUED on every engine, but only pandas lends it a ``.str`` + accessor; on cuDF the raw accessor raises, which previously routed the whole column into the + non-string (null/False) result — a silently wrong answer instead of pandas' rows. Decoding the + codes back to their string categories is exact and null-preserving on both engines. + """ + if _series_supports_str_ops(s): + return s + if not _categories_are_strings(s): + return None + try: + decoded = s.astype(str) + except Exception: + return None + return decoded if _series_supports_str_ops(decoded) else None + + def _nonstring_null_result(s: Any, na: Optional[bool]) -> Any: """Result of a string predicate applied to a NON-string column. @@ -129,8 +167,10 @@ def __init__( self.regex = regex def __call__(self, s: SeriesT) -> SeriesT: - if not _series_supports_str_ops(s): + s_str = _str_ops_series(s) + if s_str is None: return _nonstring_null_result(s, self.na) + s = s_str is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ # workaround cuDF not supporting 'case' and 'na' parameters @@ -300,8 +340,10 @@ def _compute_result(self, s: SeriesT, is_cudf: bool) -> SeriesT: return self._match_boundary(s, self.pat) def __call__(self, s: SeriesT) -> SeriesT: - if not _series_supports_str_ops(s): + s_str = _str_ops_series(s) + if s_str is None: return _nonstring_null_result(s, self.na) + s = s_str is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ result = self._compute_result(s, is_cudf) if is_cudf: @@ -423,8 +465,10 @@ def _compute_result(self, s: SeriesT, is_cudf: bool) -> SeriesT: raise NotImplementedError def __call__(self, s: SeriesT) -> SeriesT: - if not _series_supports_str_ops(s): + s_str = _str_ops_series(s) + if s_str is None: return _nonstring_null_result(s, self.na) + s = s_str is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ result = self._compute_result(s, is_cudf) if is_cudf: @@ -652,7 +696,8 @@ def predicate(s: Any) -> Any: raise NotImplementedError() def __call__(self, s: SeriesT) -> SeriesT: - return cast(SeriesT, type(self).predicate(s)) + s_str = _str_ops_series(s) + return cast(SeriesT, type(self).predicate(s if s_str is None else s_str)) class IsNumeric(_CallablePredicate): diff --git a/graphistry/compute/python_remote.py b/graphistry/compute/python_remote.py index 62ab627d04..96db8c8d07 100644 --- a/graphistry/compute/python_remote.py +++ b/graphistry/compute/python_remote.py @@ -5,15 +5,45 @@ import zipfile from typing_extensions import Literal import ast +import textwrap import pandas as pd import requests -from graphistry.Engine import Engine, EngineAbstractType, resolve_input_engine +from graphistry.Engine import EngineAbstractType from graphistry.Plottable import Plottable -from graphistry.models.compute.chain_remote import FormatType, OutputTypeAll, OutputTypeDf +from graphistry.compute.remote_df_io import ( + require_supported_frame_library, + resolve_csv_reader, + resolve_remote_engine, + validate_csv_import_args) +from graphistry.compute.remote_response import ( + decode_json_body, + decode_json_result, + error_document_error, + raise_for_remote_error, + require_json_result_keys, + select_zip_member, +) +from graphistry.models.compute.chain_remote import DFImportArgs, FormatType, OutputTypeAll, OutputTypeDf from graphistry.otel import inject_trace_headers +def normalize_task_code(code: Union[str, Callable[..., object]]) -> str: + """Normalize a callable or source string to a parseable top-level ``def task`` source.""" + + if callable(code): + code_str = inspect.getsource(code) + old_name = code.__name__ + if old_name != "task": + code_str = code_str.replace(f"def {old_name}", "def task", 1) + code = code_str + + assert code is not None and isinstance(code, str), f"Expected code to be a string, received type: {type(code)}" + + # Source from a nested def, or written as an indented literal, does not parse as-is. + return textwrap.dedent(code) + + def validate_python_str(code: str) -> bool: """Validate Python code string. @@ -42,7 +72,8 @@ def python_remote_generic( output_type: Optional[OutputTypeAll] = 'json', engine: EngineAbstractType = 'auto', run_label: Optional[str] = None, - validate: bool = True + validate: bool = True, + df_import_args: Optional[DFImportArgs] = None, ) -> Union[Plottable, pd.DataFrame, Any]: """Remotely run Python code on a remote dataset. @@ -57,7 +88,7 @@ def python_remote_generic( :param dataset_id: Optional dataset_id. If not provided, will fallback to self._dataset_id. If not defined, will upload current data, store that dataset_id, and run code against that. :type dataset_id: Optional[str] - :param format: What format to fetch results. Defaults to 'json'. We recommend a columnar format such as parquet. + :param format: What format to fetch results. Defaults to 'json'. We recommend a columnar format such as parquet. ``'csv'`` is untyped on the wire: the client re-infers dtypes and can rewrite values, so it warns and serves. Pass ``df_import_args`` to control the reader. :type format: Optional[FormatType] :param output_type: What shape of output to fetch. Defaults to 'json'. Options include 'nodes', 'edges', 'all' (both), 'table', 'shape', and 'json'. @@ -72,6 +103,9 @@ def python_remote_generic( :param validate: Whether to locally test code, and if uploading data, the data. Default true. :type validate: bool + :param df_import_args: Reader kwargs the client applies when decoding a ``format='csv'`` response. Optional; without it csv dtypes are re-inferred from text, which can rewrite values (``'007'`` -> ``7.0``) and break the returned graph's own node/edge id join. The warning names each lossy axis your kwargs do not govern, and clears only once they govern both: dtype inference (``dtype``/``converters``) and NA substitution (``keep_default_na``/``na_values``/``na_filter``/``converters``). Prefer ``format='parquet'``, which is faithful and needs no reader args. + :type df_import_args: Optional[Dict[str, Any]] + **Example: Upload data and count the results** :: @@ -97,13 +131,13 @@ def task(g: Plottable) -> Dict[str, Any]: print(f'num_edges: {num_edges}') """ - if callable(code): - if code.__name__ != "task": - code_str = inspect.getsource(code) - old_name = code.__name__ - code = code_str.replace(f"def {old_name}", "def task", 1) + code = normalize_task_code(code) - assert code is not None and isinstance(code, str), f"Expected code to be a string, received type: {type(code)}" + validate_csv_import_args(df_import_args, "python_remote") + frame_lib = require_supported_frame_library(self._nodes, self._edges, "python_remote") + engine_str = resolve_remote_engine(engine, self, "python_remote").value + + assert format in ["json", "csv", "parquet"], f"format should be 'json', 'csv', or 'parquet', got: {format}" if validate: if not validate_python_str(code): @@ -122,16 +156,6 @@ def task(g: Plottable) -> Dict[str, Any]: if not dataset_id: raise ValueError("Missing dataset_id; either pass in, or call on g2=g1.plot(render='g') in api=3 mode ahead of time") - - assert format in ["json", "csv", "parquet"], f"format should be 'json', 'csv', or 'parquet', got: {format}" - - # Resolve engine: auto -> pandas/cudf based on graph DataFrame type - engine_resolved = resolve_input_engine(engine, self) - if engine_resolved not in [Engine.PANDAS, Engine.CUDF]: - raise ValueError(f"Remote Python execution only supports 'pandas' or 'cudf' engines (or 'auto' which resolves to one of them). " - f"Got engine='{engine}' which resolved to '{engine_resolved.value}'. " - f"Dask engines are not supported for remote execution.") - engine_str = engine_resolved.value # TODO remove auto-indent when server updated # workaround parsing bug by indenting each line by 4 spaces @@ -156,37 +180,25 @@ def task(g: Plottable) -> Dict[str, Any]: response = requests.post(url, headers=headers, json=request_body, verify=self.session.certificate_validation) - # Enhanced error handling for GFQL validation errors - if not response.ok: - try: - # Try to parse JSON error response for more details - if response.headers.get('content-type', '').startswith('application/json'): - error_data = response.json() - error_msg = error_data.get('error', str(error_data)) - raise ValueError(f"GFQL remote operation failed: {error_msg} (HTTP {response.status_code})") - except ValueError: - # Re-raise ValueError (which includes our custom message) - raise - except Exception: - # Fall back to default error handling for other JSON parsing errors - pass - response.raise_for_status() - - if self._edges is None or isinstance(self._edges, pd.DataFrame): - df_cons = pd.DataFrame - read_csv = pd.read_csv - read_parquet = pd.read_parquet - elif 'cudf.core.dataframe' in str(getmodule(self._edges)): + raise_for_remote_error(response, "Remote Python operation") + + # Library was resolved pre-request; reuse it so the two cannot drift. + if frame_lib == "cudf": import cudf df_cons = cudf.DataFrame read_csv = cudf.read_csv read_parquet = cudf.read_parquet else: - raise ValueError(f"Unknown self._edges type, expected cudf/pandas DataFrame: {type(self._edges)}") + df_cons = pd.DataFrame + read_csv = pd.read_csv + read_parquet = pd.read_parquet + + if format == "csv": + read_csv = resolve_csv_reader(read_csv, df_import_args, "python_remote") if output_type == "shape": if format == "json": - return pd.DataFrame(response.json()) + return pd.DataFrame(decode_json_result(response, "Remote Python operation")) elif format == "csv": return read_csv(BytesIO(response.content)) elif format == "parquet": @@ -196,42 +208,28 @@ def task(g: Plottable) -> Dict[str, Any]: elif output_type == "all" and format in ["csv", "parquet"]: zip_buffer = BytesIO(response.content) try: - with zipfile.ZipFile(zip_buffer, "r") as zip_ref: - nodes_file = [f for f in zip_ref.namelist() if "nodes" in f][0] - edges_file = [f for f in zip_ref.namelist() if "edges" in f][0] + zip_ref_cm = zipfile.ZipFile(zip_buffer, "r") + except zipfile.BadZipFile as e: + raise error_document_error(response, "Remote Python operation", "a zip archive") from e + with zip_ref_cm as zip_ref: + names = zip_ref.namelist() + nodes_file = select_zip_member(names, "nodes", "Remote Python operation") + edges_file = select_zip_member(names, "edges", "Remote Python operation") - nodes_data = zip_ref.read(nodes_file) - edges_data = zip_ref.read(edges_file) + nodes_data = zip_ref.read(nodes_file) + edges_data = zip_ref.read(edges_file) - if len(nodes_data) > 0: - nodes_df = read_parquet(BytesIO(nodes_data)) if format == "parquet" else read_csv(BytesIO(nodes_data)) - else: - nodes_df = df_cons() + if len(nodes_data) > 0: + nodes_df = read_parquet(BytesIO(nodes_data)) if format == "parquet" else read_csv(BytesIO(nodes_data)) + else: + nodes_df = df_cons() - if len(edges_data) > 0: - edges_df = read_parquet(BytesIO(edges_data)) if format == "parquet" else read_csv(BytesIO(edges_data)) - else: - edges_df = df_cons() + if len(edges_data) > 0: + edges_df = read_parquet(BytesIO(edges_data)) if format == "parquet" else read_csv(BytesIO(edges_data)) + else: + edges_df = df_cons() - return self.edges(edges_df).nodes(nodes_df) - except zipfile.BadZipFile as e: - # Handle case where response is not a zip file (e.g., error response) - try: - # Try to parse as JSON error response - if response.headers.get('content-type', '').startswith('application/json'): - error_data = response.json() - error_msg = error_data.get('error', str(error_data)) - raise ValueError(f"GFQL remote operation failed: {error_msg} (Expected zip file but got JSON error)") - else: - # Try to decode as text for better error context - try: - error_text = response.content.decode('utf-8')[:500] # First 500 chars - raise ValueError(f"GFQL remote operation failed: Expected zip file but received: {error_text}") - except UnicodeDecodeError: - raise ValueError(f"GFQL remote operation failed: Expected zip file but received invalid data (HTTP {response.status_code})") - except Exception: - # Fallback: re-raise original BadZipFile with more context - raise ValueError(f"GFQL remote operation failed: {str(e)} - Response may be an error message instead of expected zip file") + return self.edges(edges_df).nodes(nodes_df) elif output_type in ["nodes", "edges", "table"] and format in ["csv", "parquet"]: data = BytesIO(response.content) if len(response.content) > 0: @@ -249,8 +247,12 @@ def task(g: Plottable) -> Dict[str, Any]: elif output_type == "table": return df elif format == "json": - o = response.json() + if output_type == "json": + # A task's own return value is the result here, error-shaped documents included. + return decode_json_body(response, "Remote Python operation") + o = decode_json_result(response, "Remote Python operation") if output_type == "all": + o = require_json_result_keys(o, ['nodes', 'edges'], response, "Remote Python operation") return self.edges(df_cons(o['edges'])).nodes(df_cons(o['nodes'])) elif output_type == "nodes": out = self.nodes(df_cons(o)) @@ -262,8 +264,6 @@ def task(g: Plottable) -> Dict[str, Any]: return out elif output_type == "table": return df_cons(o) - elif output_type == "json": - return o else: raise ValueError(f"JSON format read with unexpected output_type: {output_type}") else: @@ -281,7 +281,8 @@ def python_remote_g( output_type: Optional[OutputTypeAll] = 'all', engine: EngineAbstractType = 'auto', run_label: Optional[str] = None, - validate: bool = True + validate: bool = True, + df_import_args: Optional[DFImportArgs] = None, ) -> Plottable: """Remotely run Python code on a remote dataset that returns a Plottable @@ -296,7 +297,7 @@ def python_remote_g( :param dataset_id: Optional dataset_id. If not provided, will fallback to self._dataset_id. If not defined, will upload current data, store that dataset_id, and run code against that. :type dataset_id: Optional[str] - :param format: What format to fetch results. Defaults to 'parquet'. + :param format: What format to fetch results. Defaults to 'parquet'. ``'csv'`` is untyped on the wire: the client re-infers dtypes and can rewrite values, so it warns and serves. Pass ``df_import_args`` to control the reader. :type format: Optional[FormatType] :param output_type: What shape of output to fetch. Defaults to 'all'. Options include 'nodes', 'edges', 'all' (both). For other variants, see python_remote_shape and python_remote_json. @@ -311,6 +312,9 @@ def python_remote_g( :param validate: Whether to locally test code, and if uploading data, the data. Default true. :type validate: bool + :param df_import_args: Reader kwargs the client applies when decoding a ``format='csv'`` response. Optional; without it csv dtypes are re-inferred from text, which can rewrite values and break the returned graph's own node/edge id join. The warning names each lossy axis your kwargs do not govern, and clears only once they govern both: dtype inference (``dtype``/``converters``) and NA substitution (``keep_default_na``/``na_values``/``na_filter``/``converters``). Prefer ``format='parquet'``, which is faithful and needs no reader args. + :type df_import_args: Optional[Dict[str, Any]] + **Example: Upload data and count the results** :: @@ -345,7 +349,8 @@ def task(g: Plottable) -> Dict[str, Any]: output_type=output_type, engine=engine, run_label=run_label, - validate=validate + validate=validate, + df_import_args=df_import_args, ) assert isinstance(out, Plottable), f"Expected Plottable, got: {type(out)}" @@ -362,7 +367,8 @@ def python_remote_table( output_type: Optional[OutputTypeDf] = 'table', engine: EngineAbstractType = 'auto', run_label: Optional[str] = None, - validate: bool = True + validate: bool = True, + df_import_args: Optional[DFImportArgs] = None, ) -> pd.DataFrame: """Remotely run Python code on a remote dataset that returns a table @@ -377,7 +383,7 @@ def python_remote_table( :param dataset_id: Optional dataset_id. If not provided, will fallback to self._dataset_id. If not defined, will upload current data, store that dataset_id, and run code against that. :type dataset_id: Optional[str] - :param format: What format to fetch results. Defaults to 'parquet'. + :param format: What format to fetch results. Defaults to 'parquet'. ``'csv'`` is untyped on the wire: the client re-infers dtypes and can rewrite values, so it warns and serves. Pass ``df_import_args`` to control the reader. :type format: Optional[FormatType] :param output_type: What shape of output to fetch. Defaults to 'table'. Options include 'table', 'nodes', and 'edges'. @@ -392,6 +398,9 @@ def python_remote_table( :param validate: Whether to locally test code, and if uploading data, the data. Default true. :type validate: bool + :param df_import_args: Reader kwargs the client applies when decoding a ``format='csv'`` response. Optional; without it csv dtypes are re-inferred from text, which can rewrite values and break the returned graph's own node/edge id join. The warning names each lossy axis your kwargs do not govern, and clears only once they govern both: dtype inference (``dtype``/``converters``) and NA substitution (``keep_default_na``/``na_values``/``na_filter``/``converters``). Prefer ``format='parquet'``, which is faithful and needs no reader args. + :type df_import_args: Optional[Dict[str, Any]] + **Example: Upload data and count the results** :: @@ -426,7 +435,8 @@ def task(g: Plottable) -> Dict[str, Any]: output_type=output_type, engine=engine, run_label=run_label, - validate=validate + validate=validate, + df_import_args=df_import_args, ) assert isinstance(out, pd.DataFrame), f"Expected pd.DataFrame, got: {type(out)}" diff --git a/graphistry/compute/remote_df_io.py b/graphistry/compute/remote_df_io.py new file mode 100644 index 0000000000..0d64644b6e --- /dev/null +++ b/graphistry/compute/remote_df_io.py @@ -0,0 +1,182 @@ +"""Client-side decoding policy for remote GFQL / remote Python results. + +CSV is an untyped wire format: the server writes text and the client cannot +recover the original dtypes from it, so a bare reader re-infers them. Callers +are warned unless their reader kwargs govern both lossy axes -- dtype inference +and NA substitution -- which are independent: ``dtype=str`` still turns ``'NA'`` +into ``NaN``, and ``keep_default_na=False`` still turns ``'007'`` into ``7``. +``parquet`` carries an Arrow schema and is the faithful default. +""" +from inspect import getmodule +import typing +import warnings +from typing import BinaryIO, Callable, Optional +from typing_extensions import Literal + +from graphistry.Engine import Engine, EngineAbstract, EngineAbstractType, resolve_input_engine +from graphistry.Plottable import Plottable +from graphistry.compute.exceptions import ErrorCode, GFQLRemoteError +from graphistry.compute.typing import DataFrameT +from graphistry.models.compute.chain_remote import DFImportArgs + +RemoteAPIName = Literal["gfql_remote", "python_remote"] + + + +CSV_DTYPE_KWARGS = frozenset({'converters', 'dtype'}) +CSV_NA_KWARGS = frozenset({'converters', 'keep_default_na', 'na_filter', 'na_values'}) + +CSV_DTYPE_AXIS_WARNING = ( + "dtype inference is left to the reader, which retypes text ('007' -> 7.0, '08' -> 8.0); " + "govern it with a " + " or ".join(sorted(CSV_DTYPE_KWARGS)) + " reader kwarg" +) +CSV_NA_AXIS_WARNING = ( + "NA substitution is left to the reader, which blanks the pandas NA vocabulary " + "('NA'/''/'null' -> NaN); govern it with a " + + ", ".join(sorted(CSV_NA_KWARGS)) + " reader kwarg" +) +CSV_LOSSY_REMEDY = ( + "format='parquet' (the default) carries an Arrow schema and is faithful. " + "For a faithful csv read pass df_import_args, e.g. " + "df_import_args={'dtype': str, 'keep_default_na': False, 'na_values': []}." +) + + +def _frame_type_name(df: Optional[DataFrameT]) -> str: + if df is None: + return "None" + return f"{type(df).__module__.split('.')[0]}.{type(df).__name__}" + + +def _is_pandas_like(df: Optional[DataFrameT]) -> bool: + import pandas as pd + return df is None or isinstance(df, pd.DataFrame) or 'unittest.mock' in str(type(df)) + + +def resolve_remote_engine( + engine: EngineAbstractType, + graph: Plottable, + api_name: RemoteAPIName, +) -> Engine: + """Resolve a supported remote engine before auth, upload, or transport. + + :param engine: Requested engine or ``auto``. + :param graph: Graph used to resolve ``auto``. + :param api_name: Public entry point named in the error message. + :return: A pandas or cudf engine. + :raises GFQLRemoteError: When the resolved engine is not supported remotely. + """ + resolved = resolve_input_engine(engine, graph) + if resolved in (Engine.PANDAS, Engine.CUDF): + return resolved + + requested = engine.value if isinstance(engine, EngineAbstract) else engine + raise GFQLRemoteError( + ErrorCode.E405, + f"{api_name}: remote execution supports only 'pandas' and 'cudf' engines; " + f"requested {requested!r}, which resolved to {resolved.value!r}.", + field="engine", + value=requested, + suggestion="Use engine='pandas', engine='cudf', or engine='auto' with supported frames.", + ) + + +def require_supported_frame_library( + nodes: Optional[DataFrameT], edges: Optional[DataFrameT], api_name: RemoteAPIName +) -> str: + """Resolve which DataFrame library backs a remote call, before any request is sent. + + :param nodes: The graph's node frame, or ``None``. + :param edges: The graph's edge frame, or ``None``. + :param api_name: Public entry point named in the error message. + :return: ``"cudf"`` or ``"pandas"``. + :raises GFQLRemoteError: When either frame is some other library (e.g. polars). + """ + if any('cudf.core.dataframe' in str(getmodule(df)) for df in (nodes, edges) if df is not None): + return "cudf" + if _is_pandas_like(nodes) and _is_pandas_like(edges): + return "pandas" + raise GFQLRemoteError( + ErrorCode.E404, + f"{api_name}: remote execution supports pandas and cudf frames; got " + f"nodes={_frame_type_name(nodes)}, edges={_frame_type_name(edges)}. " + f"Convert with .to_pandas() before calling, or run this query locally.", + ) + + +def validate_csv_import_args( + df_import_args: Optional[DFImportArgs], + api_name: RemoteAPIName, +) -> None: + """Reject a malformed ``df_import_args`` before any request is sent. + + Type validation only: supplying nothing is legitimate and is handled at decode. + + :param df_import_args: Caller-supplied reader kwargs, or ``None``. + :param api_name: Public entry point named in the error message. + :raises GFQLRemoteError: When supplied but not a dict. + """ + if df_import_args is not None and not isinstance(df_import_args, dict): + raise GFQLRemoteError( + ErrorCode.E403, + f"{api_name}: df_import_args must be a dict of reader kwargs, got: {type(df_import_args)}", + field="df_import_args", + ) + + +def ungoverned_csv_axes(df_import_args: Optional[DFImportArgs]) -> typing.List[str]: + """Name the lossy csv axes the caller's reader kwargs do not govern. + + :param df_import_args: Caller-supplied reader kwargs, or ``None``. + :return: Zero, one, or two axis descriptions; empty means the read is under caller control. + """ + keys = set(df_import_args or {}) + axes: typing.List[str] = [] + if not (keys & CSV_DTYPE_KWARGS): + axes.append(CSV_DTYPE_AXIS_WARNING) + if not (keys & CSV_NA_KWARGS): + axes.append(CSV_NA_AXIS_WARNING) + return axes + + +def resolve_csv_import_args( + df_import_args: Optional[DFImportArgs], + api_name: RemoteAPIName, +) -> DFImportArgs: + """Resolve csv reader kwargs, warning per lossy axis the caller left to inference. + + :param df_import_args: Caller-supplied reader kwargs; ``None`` means none supplied. + :param api_name: Public entry point named in the message. + :return: Reader kwargs to apply. + :raises GFQLRemoteError: When ``df_import_args`` is supplied but is not a dict. + """ + validate_csv_import_args(df_import_args, api_name) + axes = ungoverned_csv_axes(df_import_args) + if axes: + warnings.warn( + f"{api_name}: format='csv' is untyped on the wire and this read is not fully " + f"under your control: {'; '.join(axes)}. {CSV_LOSSY_REMEDY}", + UserWarning, + stacklevel=3, + ) + return {} if df_import_args is None else df_import_args + + +def resolve_csv_reader( + read_csv: Callable[..., DataFrameT], + df_import_args: Optional[DFImportArgs], + api_name: RemoteAPIName, +) -> Callable[[BinaryIO], DataFrameT]: + """Bind a csv reader that applies the caller's explicit reader kwargs. + + :param read_csv: Engine-specific reader (``pandas.read_csv`` or ``cudf.read_csv``). + :param df_import_args: Caller-supplied reader kwargs; ``None`` means no opt-in. + :param api_name: Public entry point named in the error message. + :return: Callable taking a buffer and returning a DataFrame. + """ + args = resolve_csv_import_args(df_import_args, api_name) + + def read(buf: BinaryIO) -> DataFrameT: + return read_csv(buf, **args) + + return read diff --git a/graphistry/compute/remote_response.py b/graphistry/compute/remote_response.py new file mode 100644 index 0000000000..96b3178886 --- /dev/null +++ b/graphistry/compute/remote_response.py @@ -0,0 +1,252 @@ +"""Shared client-side error surfacing for remote GFQL / remote Python calls. + +Every failure a user can hit through the public remote APIs is raised as a typed +GFQL error carrying the HTTP status and the server's own message; no ``requests`` +or ``zipfile`` exception reaches the caller. +""" +from pathlib import PurePosixPath +from typing import Any, Dict, List, Optional, Sequence +from typing_extensions import Literal + +import requests + +from graphistry.Plottable import Plottable +from graphistry.compute.exceptions import ErrorCode, GFQLRemoteError, GFQLSchemaError + + +ZipMemberKind = Literal['nodes', 'edges'] + +_BODY_CHARS = 500 + + +def _body_text(response: requests.Response) -> str: + try: + return response.text[:_BODY_CHARS] + except Exception: + return f"<{len(response.content)} undecodable bytes>" + + +def parse_json_body(response: requests.Response) -> Optional[Any]: # hygiene-ok: explicit-any -- an arbitrary server JSON document + """Decode a response body as JSON, or ``None`` when it is not JSON at all. + + :param response: The HTTP response to decode. + :return: The decoded document, or ``None`` when the body does not decode. + """ + if not response.headers.get('content-type', '').startswith('application/json'): + return None + try: + return response.json() + # requests' JSONDecodeError subclasses ValueError, so a ValueError arm cannot be the fallback. + except Exception: + return None + + +def server_error_message(response: requests.Response) -> Optional[str]: + """Extract the server's own error text from a JSON body, when it carries one. + + :param response: The HTTP response to inspect. + :return: The server's message, or ``None`` when the body is not a JSON error document. + """ + body = parse_json_body(response) + if isinstance(body, dict) and 'error' in body: + return str(body['error']) + return None + + +def json_body_is_error(body: Any) -> bool: # hygiene-ok: explicit-any -- an arbitrary server JSON document + """Whether a decoded 200-response body is an error document rather than a result.""" + return isinstance(body, dict) and 'error' in body + + +def raise_for_remote_error(response: requests.Response, api_name: str) -> None: + """Raise a typed error for a non-2xx response, preferring the server's message. + + :param response: The failed HTTP response. + :param api_name: Public entry point named in the error message. + :raises GFQLRemoteError: Always, when ``response`` is not ok. + """ + if response.ok: + return + server_msg = server_error_message(response) + detail = server_msg if server_msg is not None else _body_text(response) + raise GFQLRemoteError( + ErrorCode.E401, + f"{api_name} failed (HTTP {response.status_code}): {detail}", + status_code=response.status_code, + server_message=server_msg, + ) + + +def error_document_error( + response: requests.Response, + api_name: str, + expected: str, +) -> GFQLRemoteError: + """Build (do not raise) a typed error for a 200 response that is not the expected payload. + + :param response: The HTTP response whose body was not usable. + :param api_name: Public entry point named in the error message. + :param expected: What the client expected to decode, e.g. ``"a zip archive"``. + :return: The error to raise at the call site. + """ + server_msg = server_error_message(response) + status = f"HTTP {response.status_code}" + if server_msg is not None: + message = f"{api_name} failed ({status}): {server_msg}" + else: + message = f"{api_name} failed ({status}): expected {expected}, got: {_body_text(response)}" + return GFQLRemoteError( + ErrorCode.E402, + message, + status_code=response.status_code, + server_message=server_msg, + ) + + +def decode_json_body(response: requests.Response, api_name: str) -> Any: # hygiene-ok: explicit-any -- an arbitrary server JSON document + """Decode a success response as JSON, raising typed instead of leaking a decoder error. + + :param response: The HTTP response to decode. + :param api_name: Public entry point named in the error message. + :return: The decoded document, whatever its shape. + :raises GFQLRemoteError: When the body does not decode as JSON. + """ + try: + return response.json() + # requests' JSONDecodeError subclasses ValueError, so a ValueError arm cannot be the fallback. + except Exception as e: + raise error_document_error(response, api_name, "a JSON result") from e + + +def decode_json_result(response: requests.Response, api_name: str) -> Any: # hygiene-ok: explicit-any -- an arbitrary server JSON document + """Decode a success response as a graph/table result, refusing error documents. + + :param response: The HTTP response to decode. + :param api_name: Public entry point named in the error message. + :return: The decoded result document. + :raises GFQLRemoteError: When the body does not decode, or is an error document. + """ + body = decode_json_body(response, api_name) + if json_body_is_error(body): + raise error_document_error(response, api_name, "a JSON result") + return body + + +def select_zip_member(names: Sequence[str], kind: ZipMemberKind, api_name: str) -> str: + """Pick the zip member holding the ``kind`` table. + + Policy, in order: a member whose stem is exactly ``kind`` wins; failing that, a + member whose name mentions ``kind`` and NOT the other kind is accepted only when + it is the sole such candidate. A compound name mentioning both (``nodes_and_edges``) + is never bound to either table -- which table it holds is unknowable from the name. + Anything else declines. + + :param names: Member names in the archive. + :param kind: ``"nodes"`` or ``"edges"``. + :param api_name: Public entry point named in the error message. + :return: The selected member name. + :raises GFQLRemoteError: When no member matches, or the match is ambiguous. + """ + other = 'edges' if kind == 'nodes' else 'nodes' + + exact = [nm for nm in names if PurePosixPath(nm).stem == kind] + if len(exact) == 1: + return exact[0] + candidates = exact + if not exact: + mentions = [nm for nm in names if kind in PurePosixPath(nm).name] + # A server may prefix member names; accept a loose match only when it cannot be the other table. + candidates = [nm for nm in mentions if other not in PurePosixPath(nm).name] + if len(candidates) == 1: + return candidates[0] + if not candidates: + if mentions: + raise GFQLRemoteError( + ErrorCode.E402, + f"{api_name} failed: server zip response has no '{kind}' member; " + f"{mentions} name both '{kind}' and '{other}', so which table they hold is undecidable", + member=kind, + members=list(names), + ambiguous=mentions, + ) + raise GFQLRemoteError( + ErrorCode.E402, + f"{api_name} failed: server zip response has no '{kind}' member", + member=kind, + members=list(names), + ) + raise GFQLRemoteError( + ErrorCode.E402, + f"{api_name} failed: server zip response has {len(candidates)} candidate '{kind}' members, cannot pick one", + member=kind, + members=list(names), + candidates=candidates, + ) + + +def require_json_result_keys( + body: Any, # hygiene-ok: explicit-any -- an arbitrary server JSON document + keys: Sequence[str], + response: requests.Response, + api_name: str, +) -> Dict[str, Any]: # hygiene-ok: explicit-any -- an arbitrary server JSON document + """Require a decoded JSON result to be an object carrying ``keys``. + + :param body: The decoded response body. + :param keys: Keys the result must provide. + :param response: The originating response, used for the error message. + :param api_name: Public entry point named in the error message. + :return: The validated body. + :raises GFQLRemoteError: When the body is an error document or misses a key. + """ + if json_body_is_error(body) or not isinstance(body, dict): + raise error_document_error(response, api_name, f"a JSON object with {list(keys)}") + missing = [k for k in keys if k not in body] + if missing: + raise GFQLRemoteError( + ErrorCode.E402, + f"{api_name} failed: server JSON response is missing {missing}", + status_code=response.status_code, + missing=missing, + keys=sorted(str(k) for k in body.keys()), + ) + return body + + +def check_subset_result_bindings( + g: 'Plottable', + node_col_subset: Optional[List[str]], + edge_col_subset: Optional[List[str]], + api_name: str, +) -> None: + """Reject a requested column subset that dropped a column the result graph is bound to. + + Runs on the final result, so server-supplied metadata bindings are the ones checked. + + :param g: The Plottable about to be returned to the caller. + :param node_col_subset: The caller's requested node columns, or ``None``. + :param edge_col_subset: The caller's requested edge columns, or ``None``. + :param api_name: Public entry point named in the error message. + :raises GFQLSchemaError: When a bound column is absent from a returned frame. + """ + checks: List[Any] = [] # hygiene-ok: explicit-any -- heterogeneous (frame, binding name, kwarg name, table) tuples + if node_col_subset is not None: + checks.append((g._nodes, g._node, 'node_col_subset', 'nodes')) + if edge_col_subset is not None: + checks.append((g._edges, g._source, 'edge_col_subset', 'edges')) + checks.append((g._edges, g._destination, 'edge_col_subset', 'edges')) + for df, col, subset_arg, table in checks: + if df is None or col is None: + continue + columns = list(getattr(df, 'columns', [])) + if not columns: + continue + if col not in columns: + raise GFQLSchemaError( + ErrorCode.E301, + f"{api_name} returned {table} without the bound '{col}' column, " + f"so the result graph would be unusable", + field=col, + value=columns, + suggestion=f"Include '{col}' in {subset_arg}, or rebind the result with g.{table}(df, ...)", + ) diff --git a/graphistry/compute/validate/validate_schema.py b/graphistry/compute/validate/validate_schema.py index 6dc3835e08..4ad884e2db 100644 --- a/graphistry/compute/validate/validate_schema.py +++ b/graphistry/compute/validate/validate_schema.py @@ -79,6 +79,35 @@ def trace_chain_schema( return snapshots +def validate_graph_shape( + g: Plottable, + ops: Optional[Union[List[ASTObject], 'Chain']] = None, + collect_all: bool = False, +) -> List[GFQLSchemaError]: + """Check whether the graph shape can answer the query at all. + + Shared by the validator and the executors so both report the same verdict. + """ + errors: List[GFQLSchemaError] = [] + + if g._nodes is None and g._edges is None: + errors.append(GFQLSchemaError( + ErrorCode.E305, + 'Cannot query graph: neither nodes nor edges are bound', + suggestion='Bind data with g.nodes(df, node) and/or g.edges(df, source, destination)' + )) + elif g._edges is None and any(isinstance(op, ASTEdge) for op in _coerce_chain_ops(ops or [])): + errors.append(GFQLSchemaError( + ErrorCode.E304, + 'Cannot traverse edges: graph has no edges bound', + suggestion='Bind edges via g.edges(df, source, destination), or use a node-only pattern' + )) + + if errors and not collect_all: + raise errors[0] + return errors + + def validate_chain_schema( g: Plottable, ops: Union[List[ASTObject], 'Chain'], @@ -105,7 +134,7 @@ def validate_chain_schema( """ chain_ops = _coerce_chain_ops(ops) - errors: List[GFQLSchemaError] = [] + errors: List[GFQLSchemaError] = validate_graph_shape(g, chain_ops, collect_all=collect_all) # Get available columns node_columns = set(g._nodes.columns) if g._nodes is not None else set() @@ -195,12 +224,16 @@ def _validate_filter_dict( collect_all: bool = False ) -> List[GFQLSchemaError]: """Validate filter dictionary against dataframe schema.""" + from graphistry.compute.gfql.strictness import absent_filter_key_is_lenient + errors = [] for col, val in filter_dict.items(): try: try: resolved_col, resolved_val = resolve_filter_column(df, col, val) except GFQLSchemaError: + if absent_filter_key_is_lenient(col, val, context=f"{context} dataframe"): + continue # resolves to null at execution; nothing to type-check error = GFQLSchemaError( ErrorCode.E301, f'Column "{col}" does not exist in {context} dataframe', @@ -215,6 +248,8 @@ def _validate_filter_dict( # Check column exists if resolved_col not in columns: + if absent_filter_key_is_lenient(col, val, context=f"{context} dataframe"): + continue error = GFQLSchemaError( ErrorCode.E301, f'Column "{col}" does not exist in {context} dataframe', diff --git a/graphistry/embed_utils.py b/graphistry/embed_utils.py index 6fc70ce98e..9db4a0c45c 100644 --- a/graphistry/embed_utils.py +++ b/graphistry/embed_utils.py @@ -272,8 +272,8 @@ def embed( embedding_dim : int relation embedding dimension. defaults to 32 use_feat : bool - wether to featurize nodes, if False will produce random embeddings and shape them during training. - Defaults to True + whether to featurize nodes, if False will produce random embeddings and shape them during training. + Defaults to False X : XSymbolic Which columns in the nodes dataframe to featurize. Inherets args from graphistry.featurize(). Defaults to None. @@ -288,13 +288,13 @@ def embed( num_steps : int num_steps. Defaults to 50 lr : float - learning rate. Defaults to 0.002 + learning rate. Defaults to 0.01 inplace : Optional[bool] inplace device : Optional[str] accelarator. Defaults to "cpu" evaluate : bool - Whether to evaluate. Defaults to False. + Whether to evaluate. Defaults to True. Returns ------- diff --git a/graphistry/feature_utils.py b/graphistry/feature_utils.py index b4d2771e1c..fef7b0adef 100644 --- a/graphistry/feature_utils.py +++ b/graphistry/feature_utils.py @@ -2576,13 +2576,14 @@ def featurize( :param encode: encoding for KBinsDiscretizer, can be one of `onehot`, `onehot-dense`, `ordinal`, default 'ordinal' :param strategy: strategy for KBinsDiscretizer, can be one of - `uniform`, `quantile`, `kmeans`, default 'quantile' + `uniform`, `quantile`, `kmeans`, default 'uniform' :param n_quantiles: if use_scaler = "quantile", sets the number of quantiles, default=100 :param output_distribution: if use_scaler="quantile"|"robust", choose from ["normal", "uniform"] :param dbscan: whether to run DBSCAN, default False. :param min_dist: DBSCAN eps parameter, default 0.5. - :param min_samples: DBSCAN min_samples parameter, default 5. + :param min_samples: DBSCAN min_samples parameter, default 1. Note that + min_samples=1 makes every point a core point, so nothing is labeled noise. :param keep_n_decimals: number of decimals to keep :param remove_node_column: whether to remove node column so it is not featurized, default True. diff --git a/graphistry/layout/circle.py b/graphistry/layout/circle.py index 968f024174..aae2d80771 100644 --- a/graphistry/layout/circle.py +++ b/graphistry/layout/circle.py @@ -78,9 +78,10 @@ def circle_layout( Arranges nodes in a circular layout If partition_by and and bounding_box df are provided, do as multiple circles - - Each circle is sorted, by default by degree - + + Node order around each circle is by node id (within partition), always. See the + ``sort_by`` note below. + The ring radius is set to circumscribe the bounding box of the nodes Parameters @@ -98,19 +99,27 @@ def circle_layout( :param point_spacing: The distance between nodes within a ring, along the circumference. Defaults to ring_spacing * 0.1 if not provided. :type point_spacing: Optional[float] - :param partition_by: Column name or list of column names to sort nodes by. Defaults to None, in which case no sorting is applied. + :param partition_by: Column name or list of column names to partition nodes by, laying + out one circle per partition. Defaults to None, in which case a single circle is used. :type partition_by: Optional[Union[str, List[str]]] - :param sort_by: Column name or list of column names to sort nodes by. Defaults to None, in which case sorting is by degree, in-degree, outdegree. + :param sort_by: Currently has NO effect on the layout. Node order around each circle is + always determined by node id (within partition); the sort this parameter performs is + discarded by that ordering before any position is assigned. Passing a column that + does not exist still raises KeyError, and leaving it None additionally attaches + degree columns (``degree``, ``degree_in``, ``degree_out``) to the output nodes. :type sort_by: Optional[Union[str, List[str]]] - :param ascending: Whether to sort ascending or descending. + :param ascending: Currently has NO effect on the layout; consumed only by the discarded + sort described under ``sort_by``. :type ascending: Union[bool, List[bool]] - :param na_position: Where to position NaNs in the sorting order. Defaults to 'last'. + :param na_position: Currently has NO effect on the layout; consumed only by the discarded + sort described under ``sort_by``. Defaults to 'last'. :type na_position: str - :param ignore_index: Whether to ignore the index when sorting. Defaults to True. + :param ignore_index: Currently has NO effect on the layout; consumed only by the discarded + sort described under ``sort_by``. Defaults to True. :type ignore_index: bool :param engine: The engine to use for computations (either 'pandas' or 'cudf'). Defaults to EngineAbstract.AUTO. diff --git a/graphistry/layout/fa2.py b/graphistry/layout/fa2.py index 220e552b69..96d6e4c787 100644 --- a/graphistry/layout/fa2.py +++ b/graphistry/layout/fa2.py @@ -64,17 +64,17 @@ def fa2_layout( """ Applies FA2 layout for connected nodes and circle layout for singleton (edgeless) nodes - Allows optional parameterization of the circle layout, e.g., sort keys + Allows optional parameterization of the circle layout, e.g., ring and point spacing - :param g: The graph object with nodes and edges, in a format compatible with Graphistry's Plottable object. - :type g: graphistry.Plottable.Plottable - :param fa2_params: Optional parameters for customizing the Force-Atlas 2 (FA2) layout, passed through to `fa2_layout`. + :param self: The graph object with nodes and edges, in a format compatible with Graphistry's Plottable object. + :type self: graphistry.Plottable.Plottable + :param fa2_params: Optional parameters for the underlying force-directed layout, forwarded as `params=` to `layout_cugraph('force_atlas2', ...)` on GPU or `layout_igraph('fr', ...)` on CPU. :type fa2_params: Optional[Dict[str, Any]] - :param circle_layout_params: Optional parameters for customizing the circle layout, passed through to `general_circle_layout`. Can include: - - `by`: Column name(s) for sorting nodes (default: 'degree'). - - `ascending`: Boolean(s) to control sorting order. + :param circle_layout_params: Optional parameters for customizing the circle layout, passed through to :func:`graphistry.layout.circle.circle_layout`. Can include: + - `partition_by`: Node column(s) selecting one circle per partition. - `ring_spacing`: Spacing between rings in the circle layout. - `point_spacing`: Spacing between points in each ring. + - `sort_by` / `ascending`: accepted, but currently have no effect on positions (see `circle_layout`). :type circle_layout_params: Optional[Dict[str, Any]] :param singleton_layout: Optional custom layout function for singleton nodes (default: circle_layout). diff --git a/graphistry/layout/gib/partitioned_layout.py b/graphistry/layout/gib/partitioned_layout.py index 91afa5efac..bb20f98699 100644 --- a/graphistry/layout/gib/partitioned_layout.py +++ b/graphistry/layout/gib/partitioned_layout.py @@ -26,11 +26,15 @@ def partitioned_layout( """ :param partition_offsets: {'dx', 'dy', 'x', 'y'} => => float :type partition_offsets: Dict[str, Dict[int, float]] - :param layout_alg: Layout algorithm to be applied if partition_key column does not already exist; GPU defaults to fa2_layout, CPU defaults to igraph fr + :param layout_alg: Layout algorithm used to position nodes within each partition. When + None, the default depends on ``bulk_mode``: under ``bulk_mode=True`` (the default) + both CPU and GPU use fa2_layout; under ``bulk_mode=False`` CPU uses igraph ``fr`` + and GPU uses cugraph ``force_atlas2``. :type layout_alg: Optional[Union[str, Callable[[Plottable], Plottable]]] :param layout_params: Parameters for the layout algorithm :type layout_params: Optional[Dict[str, Any]] - :param partition_key: The partition key; defaults to the layout_alg + :param partition_key: Name of the existing node column holding the partition id; must + already be present on the node table. Defaults to the literal ``'partition'``. :type partition_key: str :param bulk_mode: Whether to apply layout in bulk mode :type bulk_mode: bool @@ -111,7 +115,8 @@ def partitioned_layout( end_communities = timer() # Define end_communities here to track layout time logger.debug('part_layout time: %s s', end_communities - start) - if True and len(singleton_nodes) > 0: + # small-partition fallbacks only apply to non-bulk mode; bulk already positions every node + if not bulk_mode and len(singleton_nodes) > 0: logger.debug('# SINGLETONS: %s', len(singleton_nodes)) start_sing = timer() singletons = singleton_nodes.assign( @@ -123,7 +128,7 @@ def partitioned_layout( end_sing = timer() logger.debug('singleton groups (%s): %s s', len(singletons), end_sing - start_sing) - if True and len(pair_nodes) > 0: + if not bulk_mode and len(pair_nodes) > 0: logger.debug('# PAIRS: %s', len(pair_nodes)) start_pair = timer() pairs_indexed = pair_nodes.reset_index() @@ -136,14 +141,14 @@ def partitioned_layout( logger.debug('pairs groups (%s): %s s', len(pairs), end_pair - start_pair) #FIXME: how to make safe? - if True and len(edgeless_nodes) > 0: + if not bulk_mode and len(edgeless_nodes) > 0: logger.debug('# EDGELESS: %s', len(edgeless_nodes)) start_e = timer() edgeless = edgeless_nodes # FIXME: Sorted grid vs random if engine == Engine.PANDAS: edgeless['x'] = pd.Series(np.random.default_rng().uniform(0., 1., size=len(edgeless)), dtype='float32') - edgeless['x'] = pd.Series(np.random.default_rng().uniform(0., 1., size=len(edgeless)), dtype='float32') + edgeless['y'] = pd.Series(np.random.default_rng().uniform(0., 1., size=len(edgeless)), dtype='float32') elif engine == Engine.CUDF: import cudf, cupy as cp edgeless['x'] = cudf.Series(cp.random.rand(len(edgeless), dtype=cp.float32)) @@ -157,30 +162,14 @@ def partitioned_layout( combined_nodes = df_concat(engine)(node_partitions, ignore_index=True, sort=False) # FA unnconnected nodes, though circle would autoplace - updates = {} - if engine == Engine.PANDAS: - if combined_nodes.x.isna().any(): - logger.debug('filling layout-returned NAs as random: %s xs', combined_nodes.x.isna().sum()) - assert combined_nodes.x.isna().sum() == 0 - updates['x'] = pd.Series(np.random.default_rng().uniform(0., 1., size=len(combined_nodes)), dtype='float32') - if combined_nodes.y.isna().any(): - logger.debug('filling layout-returned NAs as random: %s ys', combined_nodes.y.isna().sum()) - assert combined_nodes.y.isna().sum() == 0 - updates['y'] = pd.Series(np.random.default_rng().uniform(0., 1., size=len(combined_nodes)), dtype='float32') - elif engine == Engine.CUDF: - import cudf, cupy as cp - if combined_nodes.x.isna().any(): - logger.debug('filling layout-returned NAs as random: %s xs', combined_nodes.x.isna().sum()) - assert combined_nodes.x.isna().sum() == 0 - updates['x'] = cudf.Series(cp.random.rand(len(combined_nodes), 1, dtype=cp.float32)) - if combined_nodes.y.isna().any(): - logger.debug('filling layout-returned NAs as random: %s ys', combined_nodes.y.isna().sum()) - assert combined_nodes.y.isna().sum() == 0 - updates['y'] = cudf.Series(cp.random.rand(len(combined_nodes), 1, dtype=cp.float32)) - else: - raise ValueError('Unknown engine, expected Pandas or CuDF') - if len(updates.keys()) > 0: - combined_nodes = combined_nodes.fillna(updates) + for axis in ['x', 'y']: + na_count = combined_nodes[axis].isna().sum() + if na_count > 0: + logger.debug('filling layout-returned NAs as random: %s %ss', na_count, axis) + fill = df_cons(engine)({ + axis: np.random.default_rng().uniform(0., 1., size=len(combined_nodes)).astype('float32') + })[axis] + combined_nodes[axis] = combined_nodes[axis].fillna(fill) node_stats = combined_nodes.groupby(partition_key).agg({ 'x': ['max', 'min'], diff --git a/graphistry/models/compute/chain_remote.py b/graphistry/models/compute/chain_remote.py index 725f6ceedd..d798816dc8 100644 --- a/graphistry/models/compute/chain_remote.py +++ b/graphistry/models/compute/chain_remote.py @@ -1,4 +1,4 @@ -from typing import Set, Union +from typing import Any, Dict, Set, Union from typing_extensions import Literal @@ -16,3 +16,7 @@ OutputTypeAll = Union[OutputTypeGraph, OutputTypeDf, OutputTypeJson] output_types_all = output_types_graph.union(output_types_df).union(output_types_json) + + +# Reader kwargs forwarded to the client-side csv reader; values are heterogeneous by nature +DFImportArgs = Dict[str, Any] diff --git a/graphistry/pygraphistry.py b/graphistry/pygraphistry.py index 6416069369..1190217758 100644 --- a/graphistry/pygraphistry.py +++ b/graphistry/pygraphistry.py @@ -122,7 +122,7 @@ def _is_authenticated(self, value: bool) -> None: def authenticate(self) -> None: """Authenticate via already provided configuration. This is called once automatically per session when uploading and rendering a visualization. - If token_refresh_ms > 0 (defaults to 10min), this starts an automatic refresh loop. + The JWT token is refreshed on plot() calls; there is no background refresh loop. Note that a manual .login() is still required every 24hr by default. """ @@ -146,7 +146,7 @@ def not_implemented_thunk() -> str: relogin: Callable[[], str] = not_implemented_thunk # Will be updated after class initialization def login(self, username: str, password: str, org_name: Optional[str] = None, fail_silent: bool = False) -> str: - """Authenticate and set token for reuse (api=3). If token_refresh_ms (default: 10min), auto-refreshes token. + """Authenticate and set token for reuse (api=3). The token is refreshed on plot() calls. By default, must be reinvoked within 24hr. Note: Hub keeps a separate “active organization” slot (defaulting to the personal org) that powers @@ -183,7 +183,7 @@ def relogin(): return token def pkey_login(self, personal_key_id: str, personal_key_secret: str, org_name: Optional[str] = None, fail_silent: bool = False) -> str: - """Authenticate with personal key/secret and set token for reuse (api=3). If token_refresh_ms (default: 10min), auto-refreshes token. + """Authenticate with personal key/secret and set token for reuse (api=3). The token is refreshed on plot() calls. By default, must be reinvoked within 24hr.""" if self.session.store_token_creds_in_memory: @@ -640,8 +640,6 @@ def register( :type bolt: Union[dict, Any] :param protocol: Protocol used to contact visualization server, defaults to "https". :type protocol: Optional[str] - :param token_refresh_ms: Ignored for now; JWT token auto-refreshed on plot() calls. - :type token_refresh_ms: int :param store_token_creds_in_memory: Store username/password in-memory for JWT token refreshes (Token-originated have a hard limit, so always-on requires creds somewhere) :type store_token_creds_in_memory: Optional[bool] :param client_protocol_hostname: Override protocol and host shown in browser. Defaults to protocol/server or envvar GRAPHISTRY_CLIENT_PROTOCOL_HOSTNAME. @@ -940,7 +938,7 @@ def hypergraph(self, and the renderable result Plotter. Hypergraphs reveal relationships between rows and between column values. This transform is useful for lists of events, samples, relationships, and other structured high-dimensional data. - Specify local compute engine by passing `engine='pandas'`, 'cudf', 'dask', 'dask_cudf' (default: 'pandas'). + Specify local compute engine by passing `engine='pandas'`, 'cudf', 'dask', 'dask_cudf' (default: 'auto', which selects the engine from the input dataframe type). If events are not in that engine's format, they will be converted into it. The transform creates a node for every unique value in the entity_types columns (default: all columns). @@ -1462,7 +1460,7 @@ def encode_point_color(self, :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :returns: Plotter @@ -1536,7 +1534,7 @@ def encode_edge_color(self, :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :returns: Plotter @@ -1577,7 +1575,7 @@ def encode_point_size(self, :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :returns: Plotter @@ -1659,7 +1657,7 @@ def encode_point_icon(self, :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :param as_text: Values should instead be treated as raw strings, instead of icons and images. (Default False.) @@ -1744,7 +1742,7 @@ def encode_edge_icon(self, :param for_default: Use encoding for when no user override is set. Default on. :type for_default: Optional[bool] - :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default on. + :param for_current: Use encoding as currently active. Clearing the active encoding resets it to default, which may be different. Default off. :type for_current: Optional[bool] :param as_text: Values should instead be treated as raw strings, instead of icons and images. (Default False.) diff --git a/graphistry/tests/compute/gfql/coverage_baselines/ci-pandas-py3.12.json b/graphistry/tests/compute/gfql/coverage_baselines/ci-pandas-py3.12.json index 523cefb3cd..7031cfa993 100644 --- a/graphistry/tests/compute/gfql/coverage_baselines/ci-pandas-py3.12.json +++ b/graphistry/tests/compute/gfql/coverage_baselines/ci-pandas-py3.12.json @@ -33,6 +33,7 @@ "graphistry/compute/gfql/cypher/procedures/__init__.py": 100.0, "graphistry/compute/gfql/cypher/procedures/common.py": 31.37, "graphistry/compute/gfql/cypher/procedures/networkx.py": 37.7, + "graphistry/compute/gfql/cypher/projection_columns.py": 92.31, "graphistry/compute/gfql/cypher/projection_planning.py": 80.78, "graphistry/compute/gfql/cypher/reentry/__init__.py": 100.0, "graphistry/compute/gfql/cypher/reentry/carried_outputs.py": 94.17, diff --git a/graphistry/tests/compute/gfql/cypher/test_binder.py b/graphistry/tests/compute/gfql/cypher/test_binder.py index d28439a520..25361bc832 100644 --- a/graphistry/tests/compute/gfql/cypher/test_binder.py +++ b/graphistry/tests/compute/gfql/cypher/test_binder.py @@ -839,7 +839,7 @@ def test_binder_strict_schema_checks_expr_tree_where_property_refs() -> None: def test_binder_strict_schema_rejects_missing_relationship_property_in_match_pattern() -> None: ctx = _strict_catalog_ctx( node_columns=["id", "label__Person"], - edge_columns=["src", "dst", "since"], + edge_columns=["src", "dst", "type", "since"], ) with pytest.raises(GFQLValidationError) as exc_info: FrontendBinder().bind( @@ -853,3 +853,24 @@ def test_binder_strict_schema_rejects_missing_relationship_property_in_match_pat assert "weight" in err.context["value"] assert "disable strict mode" not in err.context["suggestion"] assert err.context["suggestion"] == "Use properties that exist in edge schema columns or extend the schema catalog." + + +def test_binder_strict_schema_empty_declared_relationship_catalog_rejects_type() -> None: + ctx = PlanContext( + catalog=GraphSchemaCatalog.from_schema_parts( + node_columns=["id"], + edge_columns=["src", "dst", "type"], + metadata={"strict": True, "edge_types": ()}, + ) + ) + with pytest.raises(GFQLValidationError) as exc_info: + FrontendBinder().bind( + parse_cypher("MATCH (a)-[r:NOPE]->(b) RETURN r"), + ctx, + strict_name_resolution=True, + ) + err = exc_info.value + assert err.code == ErrorCode.E301 + assert err.context["relationship_type"] == "NOPE" + assert err.context["available_relationship_types"] == () + assert err.context["field"].endswith(".types") diff --git a/graphistry/tests/compute/gfql/cypher/test_binding_seed_identity.py b/graphistry/tests/compute/gfql/cypher/test_binding_seed_identity.py new file mode 100644 index 0000000000..3311bac2dc --- /dev/null +++ b/graphistry/tests/compute/gfql/cypher/test_binding_seed_identity.py @@ -0,0 +1,84 @@ +"""Generic binding rows use node ids as identities.""" +from __future__ import annotations + +import typing + +import pandas as pd +import pytest +from typing_extensions import Literal + +import graphistry +from graphistry.Engine import Engine, df_to_engine +from graphistry.Plottable import Plottable +from graphistry.tests.compute.gfql.engagement import assert_fast_path +from graphistry.tests.compute.gfql.polars_test_utils import engine_skip_reason, to_pandas_any + + +_GFQLEngine = Literal["pandas", "polars", "cudf", "polars-gpu"] +_ENGINES: typing.Tuple[_GFQLEngine, ...] = ("pandas", "polars", "cudf", "polars-gpu") + + +def _bind(nodes: pd.DataFrame, edges: pd.DataFrame, engine: _GFQLEngine) -> Plottable: + resolved_engine = Engine(engine) + return graphistry.nodes(df_to_engine(nodes, resolved_engine), "id").edges( + df_to_engine(edges, resolved_engine), "s", "d" + ) + + +def _smoke(engine: _GFQLEngine) -> Plottable: + nodes = pd.DataFrame({"id": [1, 2]}) + edges = pd.DataFrame({"s": [1], "d": [2]}) + return _bind(nodes, edges, engine).gfql("MATCH (n) RETURN n.id AS id", engine=engine) + + +def _require_engine(engine: _GFQLEngine) -> None: + skip_reason = engine_skip_reason(engine, lambda: _smoke(engine)) + if skip_reason is not None: + pytest.skip(skip_reason) + + +def _run_generic(engine: _GFQLEngine, *, parallel_edge: bool) -> typing.Mapping[str, int]: + nodes = pd.DataFrame({ + "id": [1, 2, 3, 4, 5, 6, 7, 8, 1, 2], + "kind": ["P", "P", "P", "P", "C", "C", "C", "C", "P", "P"], + "city": [None, None, None, None, "LA", "NY", "SF", "LA", None, None], + }) + edges = pd.DataFrame({ + "s": [1, 2, 3, 4, 1, 2, 3, 4, 1], + "d": [5, 5, 6, 7, 8, 6, 8, 8, 5], + }) + if parallel_edge: + edges = pd.concat( + [edges, pd.DataFrame({"s": [1], "d": [5]})], + ignore_index=True, + ) + + _require_engine(engine) + graph = _bind(nodes, edges, engine) + query = ( + "MATCH (p {kind:'P'})-->(c {kind:'C'}) " + "RETURN c.city AS city, count(*) AS n ORDER BY city ASC SKIP 0" + ) + assert_fast_path( + graph, query, "single_hop_grouped_aggregate", served=False, engine=engine + ) + result_frame = graph.gfql(query, engine=engine)._nodes + pandas_frame = to_pandas_any(result_frame) + assert isinstance(pandas_frame, pd.DataFrame) + return {str(row.city): int(row.n) for row in pandas_frame.itertuples(index=False)} + + +@pytest.mark.parametrize("engine", _ENGINES) +@pytest.mark.parametrize( + "parallel_edge,expected", + [ + (False, {"LA": 6, "NY": 2, "SF": 1}), + (True, {"LA": 7, "NY": 2, "SF": 1}), + ], +) +def test_generic_binding_seed_ids_are_identities( + engine: _GFQLEngine, + parallel_edge: bool, + expected: typing.Mapping[str, int], +) -> None: + assert _run_generic(engine, parallel_edge=parallel_edge) == expected diff --git a/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_fused_polars.py b/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_fused_polars.py index 2b6eb53903..03b9cd193a 100644 --- a/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_fused_polars.py +++ b/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_fused_polars.py @@ -42,6 +42,7 @@ from __future__ import annotations import itertools +import typing from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple import numpy as np @@ -333,6 +334,20 @@ def probe(*args: Any, **kwargs: Any) -> Any: return calls +_PolarsEngine = typing.Literal["polars", "polars-gpu"] + + +def _require_fused_service( + engine: _PolarsEngine, calls: Sequence[bool], context: str +) -> None: + observed = list(calls) + if engine == "polars-gpu" and observed == [False]: + pytest.xfail( + "#1997: cudf-polars cannot execute this fused plan; GPU fallback values passed" + ) + assert observed == [True], f"{context}: fused lane must serve on {engine}; got {observed}" + + def _force_eager(monkeypatch: pytest.MonkeyPatch) -> None: """Pin the eager twin as the oracle arm for the differential.""" monkeypatch.setattr( @@ -370,7 +385,7 @@ def probe(*args: Any, **kwargs: Any) -> Any: @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) @pytest.mark.parametrize("label,query", _SERVED_SHAPES, ids=[s[0] for s in _SERVED_SHAPES]) def test_grouped_aggregate_fused_polars_serves_total_order_shapes( - engine: str, label: str, query: str, monkeypatch: pytest.MonkeyPatch + engine: _PolarsEngine, label: str, query: str, monkeypatch: pytest.MonkeyPatch ) -> None: """Every shape whose ORDER BY names all group keys is SERVED, and answers what pandas answers -- row order included.""" @@ -381,8 +396,8 @@ def test_grouped_aggregate_fused_polars_serves_total_order_shapes( calls = _probe_fused(monkeypatch) result = _records(graph.gfql(query, engine=engine)) - assert calls == [True], f"{label}: fused lane must serve on {engine}" assert result == oracle, f"{label}: fused lane diverged from the pandas oracle" + _require_fused_service(engine, calls, label) @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) @@ -446,7 +461,7 @@ def test_grouped_aggregate_fused_polars_is_never_reached_by_dataframe_engines( @pytest.mark.parametrize("graph_name", sorted(_GRAPHS)) @pytest.mark.parametrize("label,query", _SERVED_SHAPES, ids=[s[0] for s in _SERVED_SHAPES]) def test_grouped_aggregate_fused_polars_matches_eager_twin_and_pandas( - engine: str, graph_name: str, label: str, query: str, monkeypatch: pytest.MonkeyPatch + engine: _PolarsEngine, graph_name: str, label: str, query: str, monkeypatch: pytest.MonkeyPatch ) -> None: """DIFFERENTIAL: fused == eager twin == pandas oracle, ORDER-SENSITIVELY, and the fused lane really ran (otherwise the comparison is vacuous). @@ -463,10 +478,10 @@ def test_grouped_aggregate_fused_polars_matches_eager_twin_and_pandas( calls = _probe_fused(monkeypatch) fused = _records(_graph(engine, nodes, edges).gfql(query, engine=engine)) - assert calls == [True], f"{graph_name}/{label}: lane did not serve -- differential vacuous" assert fused == eager, f"{graph_name}/{label}: fused lane diverged from the eager twin" if graph_name not in ("dup_node_rows", "dup_start_node_rows"): assert fused == oracle, f"{graph_name}/{label}: fused lane diverged from pandas" + _require_fused_service(engine, calls, f"{graph_name}/{label}") @pytest.mark.parametrize("graph_name,query,fused_rows,pandas_rows", [ @@ -692,7 +707,7 @@ def test_grouped_aggregate_fused_polars_supports_min_and_max_through_the_ast_sur @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) def test_grouped_aggregate_fused_polars_node_key_named_like_the_source_column( - engine: str, monkeypatch: pytest.MonkeyPatch + engine: _PolarsEngine, monkeypatch: pytest.MonkeyPatch ) -> None: """The node key may share its name with the edge SOURCE column -- that name then appears on both sides of the semi-join and of the property lookup.""" @@ -705,13 +720,13 @@ def test_grouped_aggregate_fused_polars_node_key_named_like_the_source_column( result = _records( _graph(engine, nodes, edges, node_key="s").gfql(Q_COUNT_STAR, engine=engine)) - assert calls == [True] assert result == oracle + _require_fused_service(engine, calls, "node key/source-column collision") @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) def test_grouped_aggregate_fused_polars_empty_match_returns_no_groups( - engine: str, monkeypatch: pytest.MonkeyPatch + engine: _PolarsEngine, monkeypatch: pytest.MonkeyPatch ) -> None: """An empty match produces no GROUPS (unlike a bare ``count(*)``, which openCypher counts as 0 over no rows -- that shape is served by a different fast path).""" @@ -721,8 +736,8 @@ def test_grouped_aggregate_fused_polars_empty_match_returns_no_groups( calls = _probe_fused(monkeypatch) result = _records(_graph(engine, nodes, edges).gfql(Q_COUNT_STAR, engine=engine)) - assert calls == [True] assert result == oracle == (["city", "n"], []) + _require_fused_service(engine, calls, "empty match") def _null_group_key_data() -> Tuple[pd.DataFrame, pd.DataFrame]: @@ -739,7 +754,7 @@ def _null_group_key_data() -> Tuple[pd.DataFrame, pd.DataFrame]: ("MATCH (p)-[{rel:'L'}]->(c) RETURN c.city AS city, count(*) AS n ORDER BY city DESC", None), ]) def test_grouped_aggregate_fused_polars_places_nulls_the_opencypher_way( - engine: str, query: str, expected_first_city: Optional[str], + engine: _PolarsEngine, query: str, expected_first_city: Optional[str], monkeypatch: pytest.MonkeyPatch ) -> None: """openCypher orders NULL as the LARGEST value: last on ASC, first on DESC. polars @@ -751,14 +766,14 @@ def test_grouped_aggregate_fused_polars_places_nulls_the_opencypher_way( calls = _probe_fused(monkeypatch) result = _records(_graph(engine, nodes, edges).gfql(query, engine=engine)) - assert calls == [True] assert result == oracle assert result[1][0]["city"] == expected_first_city + _require_fused_service(engine, calls, "null group-key ordering") @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) def test_grouped_aggregate_fused_polars_null_aggregate_value_ordering( - engine: str, monkeypatch: pytest.MonkeyPatch + engine: _PolarsEngine, monkeypatch: pytest.MonkeyPatch ) -> None: """Same null-largest rule on the AGGREGATE column, where the null comes from averaging an all-null group -- and with LIMIT, so getting it wrong returns a different ROW.""" @@ -770,8 +785,8 @@ def test_grouped_aggregate_fused_polars_null_aggregate_value_ordering( calls = _probe_fused(monkeypatch) result = _records(_graph(engine, nodes, edges).gfql(query, engine=engine)) - assert calls == [True] assert result == oracle + _require_fused_service(engine, calls, "null aggregate ordering") # ------------------------------------------------------- the benchmark shapes themselves @@ -812,7 +827,7 @@ def _gb_shaped_graph() -> Tuple[pd.DataFrame, pd.DataFrame]: @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) @pytest.mark.parametrize("label,query", _GB_SHAPES, ids=[s[0] for s in _GB_SHAPES]) def test_grouped_aggregate_fused_polars_serves_the_graph_benchmark_shapes( - engine: str, label: str, query: str, monkeypatch: pytest.MonkeyPatch + engine: _PolarsEngine, label: str, query: str, monkeypatch: pytest.MonkeyPatch ) -> None: """STRUCTURAL LOCK-IN for the three benchmark cells this lane exists to move: q1, q3 and q4 must be SERVED (not merely fast) and must answer what pandas answers.""" @@ -822,8 +837,8 @@ def test_grouped_aggregate_fused_polars_serves_the_graph_benchmark_shapes( calls = _probe_fused(monkeypatch) result = _records(_graph(engine, nodes, edges).gfql(query, engine=engine)) - assert calls == [True], f"{label}: the benchmark shape must be served by the fused lane" assert result == oracle + _require_fused_service(engine, calls, label) @pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) diff --git a/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_lowcard_count.py b/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_lowcard_count.py index adbd472bef..e630e699e5 100644 --- a/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_lowcard_count.py +++ b/graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_lowcard_count.py @@ -34,7 +34,7 @@ """ from __future__ import annotations -from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple +from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Sequence, Tuple import numpy as np import pandas as pd @@ -44,6 +44,9 @@ from graphistry.Plottable import Plottable import graphistry.compute.gfql_fast_paths as gfql_fast_paths_module +if TYPE_CHECKING: + import polars as pl + MAX_GROUPS = gfql_fast_paths_module._LOWCARD_COUNT_MAX_GROUPS MAX_INPUT_ROWS = gfql_fast_paths_module._LOWCARD_COUNT_MAX_INPUT_ROWS @@ -722,6 +725,17 @@ def test_gate_unit_declines_a_non_polars_owner_frame() -> None: }) is None +def _twin_count(alias: str) -> "pl.Expr": + """The group_by formulation EXACTLY as production builds it -- ``pl.len()`` conformed to the + Cypher INTEGER count contract (agg_types.py). A raw ``pl.len()`` here would compare the + value_counts lane against a twin production no longer emits, and the schema equality below is + the whole point: the two lanes must be type-identical, not merely value-identical.""" + import polars as pl + + from graphistry.compute.gfql.agg_types import polars_conform_agg_dtype + return polars_conform_agg_dtype(pl.len(), "count", None, alias) + + def test_gate_unit_serves_a_group_key_literally_named_count() -> None: """``value_counts`` names its output column ``count`` by default, which would collide. The lane passes ``name=`` instead of renaming, so this shape is SERVED, and it must @@ -741,7 +755,7 @@ def test_gate_unit_serves_a_group_key_literally_named_count() -> None: edge_rows=3, ) assert plan is not None - twin = work.group_by(["count"], maintain_order=True).agg(pl.len().alias("n")).collect() + twin = work.group_by(["count"], maintain_order=True).agg(_twin_count("n")).collect() got = plan.collect() assert got.schema == twin.schema assert sorted(got.rows()) == sorted(twin.rows()) @@ -776,7 +790,7 @@ def test_gate_unit_matches_the_group_by_twin_on_awkward_keys( edge_rows=len(values), ) assert plan is not None - twin = work.group_by(["city"], maintain_order=True).agg(pl.len().alias("n")).collect() + twin = work.group_by(["city"], maintain_order=True).agg(_twin_count("n")).collect() got = plan.collect() assert got.schema == twin.schema diff --git a/graphistry/tests/compute/gfql/cypher/test_lowering.py b/graphistry/tests/compute/gfql/cypher/test_lowering.py index fb7a8d4d33..e7fd2cd3b7 100644 --- a/graphistry/tests/compute/gfql/cypher/test_lowering.py +++ b/graphistry/tests/compute/gfql/cypher/test_lowering.py @@ -2428,8 +2428,8 @@ def test_lower_match_query_executes_bracketless_relationship_and_label_where() - ) result = _mk_graph(nodes, edges).gfql(chain) - assert result._nodes[["id", "type", "score"]].to_dict(orient="records") == [ - {"id": "t1", "type": "TextNode", "score": 7} + assert result._nodes[["i.id", "i.type", "i.score"]].to_dict(orient="records") == [ + {"i.id": "t1", "i.type": "TextNode", "i.score": 7} ] @@ -2445,8 +2445,8 @@ def test_lower_match_query_executes_bracketless_relationship_with_labeled_alias_ chain = cypher_to_gfql("MATCH (a)-->(b:Foo) RETURN b") result = _mk_graph(nodes, edges).gfql(chain) - assert result._nodes[["id", "type"]].to_dict(orient="records") == [ - {"id": "b", "type": "Foo"} + assert result._nodes[["b.id", "b.type"]].to_dict(orient="records") == [ + {"b.id": "b", "b.type": "Foo"} ] @@ -6167,8 +6167,8 @@ def test_string_cypher_failfast_rejects_optional_match_null_extension_shapes_wit def test_string_cypher_optional_arm_label_where_serves_edge_projection() -> None: """#1891 regression fix: an optional-arm label WHERE null-extends instead of gating -- matched arm projects the edge, unmatched arm keeps the seed - row with r = null, and a missing label follows the standard - column-not-found contract (same as plain MATCH).""" + row with r = null, and a missing label follows the standard absent-name + contract (same as plain MATCH): strict raises, the warn default null-extends.""" graph = _mk_graph( pd.DataFrame( { @@ -6194,7 +6194,10 @@ def test_string_cypher_optional_arm_label_where_serves_edge_projection() -> None unmatched = graph.gfql("MATCH (n:Single) OPTIONAL MATCH (n)-[r]-(m) WHERE m:Single RETURN r") assert entity_text_records(unmatched, {"r": "edges"}) == [{"r": None}] with pytest.raises(GFQLSchemaError): - graph.gfql("MATCH (n:Single) OPTIONAL MATCH (n)-[r]-(m) WHERE m:NonExistent RETURN r") + graph.gfql("MATCH (n:Single) OPTIONAL MATCH (n)-[r]-(m) WHERE m:NonExistent RETURN r", + strict=True) + absent = graph.gfql("MATCH (n:Single) OPTIONAL MATCH (n)-[r]-(m) WHERE m:NonExistent RETURN r") + assert entity_text_records(absent, {"r": "edges"}) == [{"r": None}] def test_string_cypher_failfast_rejects_graph_backed_unwind_after_with_as_validation_error() -> None: @@ -6400,12 +6403,17 @@ def test_string_cypher_with_unwind_reentry_progresses_past_parser_to_row_scope_b "RETURN foaf" ) - with pytest.raises(GFQLValidationError) as exc_info: - compile_cypher(query) + g = _mk_graph( + pd.DataFrame({"id": ["s1", "b1", "c1"], "label__S": [True, False, False], + "label__B": [False, True, False], "label__C": [False, False, True]}), + pd.DataFrame({"s": ["s1", "b1"], "d": ["b1", "c1"], "type": ["X", "Y"]}), + ) + with pytest.raises(GFQLSchemaError) as exc_info: + g.gfql(query) - assert exc_info.value.code == ErrorCode.E108 - assert "one MATCH source alias at a time" in exc_info.value.message - assert "#1273" in exc_info.value.message + assert exc_info.value.code == ErrorCode.E301 + assert exc_info.value.context.get("field") == "where_rows.root" + assert exc_info.value.context.get("value") == "root" def test_string_cypher_rejects_with_unwind_reentry_when_unwind_source_is_not_collected_alias() -> None: diff --git a/graphistry/tests/compute/gfql/cypher/test_temporal_arithmetic_folding_branches.py b/graphistry/tests/compute/gfql/cypher/test_temporal_arithmetic_folding_branches.py index 7f372ebcbe..38fcd1712b 100644 --- a/graphistry/tests/compute/gfql/cypher/test_temporal_arithmetic_folding_branches.py +++ b/graphistry/tests/compute/gfql/cypher/test_temporal_arithmetic_folding_branches.py @@ -237,15 +237,48 @@ def test_scale_duration_multiplies_every_group() -> None: assert _scale_duration((0, 0, _SECOND_NS), 3.0, divide=True) == "PT0.333333333S" -def test_scale_duration_declines_a_fractional_month_result() -> None: - """Half a month has no fixed length, so the fold must decline rather than round.""" - assert _scale_duration((1, 0, 0), 2.0, divide=True) is None - assert _scale_duration((1, 0, 0), 0.5, divide=False) is None +def test_scale_duration_spills_a_fractional_month_at_the_average_month() -> None: + """Half a month is 15.2184375 days (30.436875 / 2), which then spills into time.""" + assert _scale_duration((1, 0, 0), 2.0, divide=True) == "P15DT5H14M33S" + assert _scale_duration((1, 0, 0), 0.5, divide=False) == "P15DT5H14M33S" + assert _scale_duration((1, 0, 0), 3.0, divide=True) == "P10DT3H29M42S" + + +def test_scale_duration_keeps_whole_months_as_months() -> None: + """The average month is used ONLY where a month must actually be split: a result + that is whole in month-space stays in month-space, exact rather than approximate.""" assert _scale_duration((2, 0, 0), 2.0, divide=True) == "P1M" + assert _scale_duration((12, 0, 0), 2.0, divide=True) == "P6M" + assert _scale_duration((1, 0, 0), 1.0, divide=True) == "P1M" + + +def test_scale_duration_truncates_the_month_group_toward_zero() -> None: + """int() toward zero, so the month remainder carries the sign of the result and a + negative scale mirrors its positive twin instead of spilling a whole extra month.""" + assert _scale_duration((1, 0, 0), -2.0, divide=True) == "P-15DT-5H-14M-33S" + assert _scale_duration((-1, 0, 0), 2.0, divide=True) == "P-15DT-5H-14M-33S" + assert _scale_duration((-1, 0, 0), -2.0, divide=True) == "P15DT5H14M33S" + assert _scale_duration((3, 0, 0), 2.0, divide=True) == "P1M15DT5H14M33S" + assert _scale_duration((-3, 0, 0), 2.0, divide=True) == "P-1M-15DT-5H-14M-33S" + + +def test_scale_duration_does_not_round_trip_through_a_split_month() -> None: + """ACCEPTED, not a bug: the average month is one-way, so halving and doubling a + month lands on days+time, never back on P1M. Do not 'fix' this into month-space.""" + half = _scale_duration((1, 0, 0), 2.0, divide=True) + assert half == "P15DT5H14M33S" + assert _scale_duration(parse_duration_calendar_components(half) or (0, 0, 0), 2.0, divide=False) == ( + "P30DT10H29M6S" + ) -def test_scale_duration_rounds_the_seconds_group_to_whole_nanoseconds() -> None: +def test_scale_duration_truncates_the_seconds_group_to_whole_nanoseconds() -> None: + """Truncation toward zero at every cascade step, so a third of two seconds is + ...666S and not ...667S, and the negative twin truncates the same way.""" assert _scale_duration((0, 0, 1), 3.0, divide=True) == "PT0S" + assert _scale_duration((0, 0, 2 * _SECOND_NS), 3.0, divide=True) == "PT0.666666666S" + assert _scale_duration((0, 0, 2 * _SECOND_NS), -3.0, divide=True) == "PT-0.666666666S" + assert _scale_duration((0, 0, 8 * _SECOND_NS), 9.0, divide=True) == "PT0.888888888S" # =========================================================================== @@ -338,9 +371,26 @@ def test_shift_declines_when_the_shifted_year_leaves_the_representable_range() - ("PT18H", "*", 2, "PT36H"), ("P2DT2H", "*", 2, "P4DT4H"), ("P3D", "*", 1.5, "P4DT12H"), + # ... but a month that has to be SPLIT resolves at the average month of + # 30.436875 days, spilling months -> days -> time rather than declining. + ("P1M", "/", 2, "P15DT5H14M33S"), + ("P1M", "*", 0.5, "P15DT5H14M33S"), + ("P1M", "/", -2, "P-15DT-5H-14M-33S"), + ("P1M", "*", -0.5, "P-15DT-5H-14M-33S"), + ("-P1M", "/", 2, "P-15DT-5H-14M-33S"), + ("P1M2D", "/", 2, "P16DT5H14M33S"), + ("P1Y1M", "/", 2, "P6M15DT5H14M33S"), + ("P1M", "/", 4, "P7DT14H37M16.5S"), + ("P1M", "*", 1.5, "P1M15DT5H14M33S"), + # A month that divides evenly is still exact in month-space. + ("P1M", "/", 1, "P1M"), + ("P2M", "/", 2, "P1M"), + ("P1Y", "/", 2, "P6M"), + ("P1M", "*", 2, "P2M"), # * commutes (2, "*", "P1D", "P2D"), (0.5, "*", "P1D", "PT12H"), + (0.5, "*", "P1M", "P15DT5H14M33S"), ], ) def test_fold_temporal_arithmetic_folds(left: Any, op: str, right: Any, expected: str) -> None: @@ -377,9 +427,6 @@ def test_fold_temporal_arithmetic_folds(left: Any, op: str, right: Any, expected # served as PT0S (pinned in the fold table above) -- declining it left # Python's `str * 0` to emit the empty string. ("P1D", "/", 0), - # A fractional month result has no fixed length. - ("P1M", "*", 0.5), - ("P1M", "/", 2), # The left operand is not a parseable temporal. ("foo", "+", "P1D"), ("P1D", "+", "foo"), @@ -829,3 +876,27 @@ def test_temporal_shift_past_date_max_declines_not_overflowerror(): ]: with pytest.raises(GFQLTypeError): g.gfql(q, engine="pandas") + + +# --- defensive arms of the #1948 comparison fold ------------------------------------- + + +def test_shift_declines_a_dateless_non_time_value(): + from graphistry.compute.gfql.temporal.folding import _shift_temporal_value + from graphistry.compute.gfql.temporal.values import _TemporalValue + value = _TemporalValue(kind="date", date_value=None) + assert _shift_temporal_value(value, months=1, days=0, time_nanos=0) is None + + +def test_fold_comparison_returns_none_on_unparseable_temporal_text(): + from graphistry.compute.gfql.expr_parser import BinaryOp, Literal + from graphistry.compute.gfql.temporal.folding import _fold_temporal_comparison + node = BinaryOp("<", Literal("2020-13-99T99:99"), Literal("2020-01-01")) + assert _fold_temporal_comparison(node) is None + + +def test_rewrite_keeps_unknown_current_temporal_call_verbatim(monkeypatch): + import graphistry.compute.gfql.temporal.folding as folding + monkeypatch.setattr(folding._tt, "_current_temporal_literal", lambda fn, now: None) + text = "n.ts > datetime()" + assert folding.rewrite_temporal_constructors_in_expr(text) == text diff --git a/graphistry/tests/compute/gfql/index/test_exists_pattern_index_agreement.py b/graphistry/tests/compute/gfql/index/test_exists_pattern_index_agreement.py new file mode 100644 index 0000000000..b583f42cd4 --- /dev/null +++ b/graphistry/tests/compute/gfql/index/test_exists_pattern_index_agreement.py @@ -0,0 +1,173 @@ +"""Indexed-vs-scan agreement for the EXISTS / NOT EXISTS pattern-predicate family. + +An index is an optimization: for every pattern shape it must produce the SAME rows as +the un-indexed graph, or decline identically. Never a third answer. + +The matrix is generated (graph shape x pattern x polarity) rather than hand-listed, so +new adjacency shortcuts are held to the whole family and not just the reported input. +""" +import pytest + +import graphistry + +pl = pytest.importorskip("polars") + + +def _graphs(): + """(name, nodes, edges) shapes that stress every way edge-table keys can diverge + from a node-table-intersected answer: self-loops, edge endpoints absent from the + node table, isolated nodes, string ids, duplicate node rows, no edges at all.""" + return [ + ("self_loop_and_isolated", + pl.DataFrame({"id": [0, 1, 2, 3]}), pl.DataFrame({"s": [0, 3], "d": [1, 3]})), + ("endpoint_absent_from_nodes", + pl.DataFrame({"id": [0, 1, 2]}), pl.DataFrame({"s": [0, 9], "d": [9, 9]})), + ("chain_plus_self_loop", + pl.DataFrame({"id": [0, 1, 2, 3, 4, 5]}), pl.DataFrame({"s": [0, 1, 4, 3], "d": [1, 2, 4, 0]})), + ("string_ids", + pl.DataFrame({"id": ["a", "b", "c", "d"]}), pl.DataFrame({"s": ["a", "d"], "d": ["b", "d"]})), + ("duplicate_node_rows", + pl.DataFrame({"id": [0, 0, 1, 2, 3]}), pl.DataFrame({"s": [0, 3], "d": [1, 3]})), + ("every_edge_is_a_self_loop", + pl.DataFrame({"id": [0, 1, 2]}), pl.DataFrame({"s": [0, 1], "d": [0, 1]})), + ("no_edges", + pl.DataFrame({"id": [0, 1, 2]}), + pl.DataFrame({"s": [], "d": []}, schema={"s": pl.Int64, "d": pl.Int64})), + ] + + +PATTERNS = [ + "(n)-->(n)", "(n)<--(n)", "(n)--(n)", + "(n)-->(m)", "(n)<--(m)", "(n)--(m)", + "(n)-->()", "(n)<--()", "(n)--()", + "(n)-->()-->()", "(n)-->(m)-->(n)", + "(n)-->(m) WHERE m <> n", +] + + +def _outcome(g, query): + """('rows', sorted-ids) or ('declined', None) — the two outcomes an index may pick + between; anything else is a third answer and fails the comparison outright.""" + try: + out = g.gfql(query, engine="polars") + except NotImplementedError: + return ("declined", None) + col = out._nodes.columns[0] + return ("rows", tuple(sorted(out._nodes[col].to_list()))) + + +@pytest.mark.parametrize("shape", [s[0] for s in _graphs()]) +@pytest.mark.parametrize("pattern", PATTERNS) +@pytest.mark.parametrize("polarity", ["EXISTS", "NOT EXISTS"]) +def test_indexed_pattern_predicate_matches_the_scan(shape, pattern, polarity): + nodes, edges = next((n, e) for name, n, e in _graphs() if name == shape) + g = graphistry.nodes(nodes, "id").edges(edges, "s", "d") + query = "MATCH (n) WHERE %s { %s } RETURN n.id" % (polarity, pattern) + scan = _outcome(g, query) + indexed = _outcome(g.gfql_index_all(), query) + assert indexed == scan, "indexed %r != scan %r for %s" % (indexed, scan, query) + + +def test_indexed_self_loop_exists_does_not_answer_has_any_out_edge(): + """The reported input: only node 3 self-loops, so 'has an out-edge' (nodes 0 and 3) + is the wrong answer for ``(n)-->(n)``.""" + g = graphistry.nodes(pl.DataFrame({"id": [0, 1, 2, 3]}), "id").edges( + pl.DataFrame({"s": [0, 3], "d": [1, 3]}), "s", "d") + query = "MATCH (n) WHERE EXISTS { (n)-->(n) } RETURN n.id" + with pytest.raises(NotImplementedError): + g.gfql(query, engine="polars") + with pytest.raises(NotImplementedError): + g.gfql_index_all().gfql(query, engine="polars") + + +def test_indexed_self_loop_not_exists_does_not_drop_satisfying_rows(): + """The mirror: ``NOT EXISTS { (n)-->(n) }`` is true of 0, 1 and 2 here, so the + 'has an out-edge' complement (1 and 2) drops a row that satisfies the predicate.""" + g = graphistry.nodes(pl.DataFrame({"id": [0, 1, 2, 3]}), "id").edges( + pl.DataFrame({"s": [0, 3], "d": [1, 3]}), "s", "d") + query = "MATCH (n) WHERE NOT EXISTS { (n)-->(n) } RETURN n.id" + with pytest.raises(NotImplementedError): + g.gfql(query, engine="polars") + with pytest.raises(NotImplementedError): + g.gfql_index_all().gfql(query, engine="polars") + + +def _keys(g, ops, alias="n"): + from graphistry.compute.ast import serialize_binding_ops + from graphistry.compute.gfql.lazy.engine.polars.pattern_apply import _pattern_alias_keys_polars + return _pattern_alias_keys_polars(g, serialize_binding_ops(ops), alias) + + +def test_adjacency_membership_declines_a_repeated_endpoint_alias(): + from graphistry.compute.ast import e_forward, n + g = graphistry.nodes(pl.DataFrame({"id": [0, 1, 2, 3]}), "id").edges( + pl.DataFrame({"s": [0, 3], "d": [1, 3]}), "s", "d").gfql_index_all() + assert _keys(g, [n(name="n"), e_forward(), n(name="n")]) is None + assert _keys(g, [n(name="n"), e_forward(), n(name="m")]) is not None + + +def test_adjacency_membership_excludes_edges_whose_other_end_is_not_a_node(): + """Adjacency keys are edge-derived; the scan additionally requires BOTH endpoints in + the node table. Node 0's only out-edge points at 9, which is not a node, so 0 does + not participate — with or without an index.""" + from graphistry.compute.ast import e_forward, n + ops = [n(name="n"), e_forward(), n(name="m")] + dangling = graphistry.nodes(pl.DataFrame({"id": [0, 1, 2]}), "id").edges( + pl.DataFrame({"s": [0, 9], "d": [9, 9]}), "s", "d") + covered = graphistry.nodes(pl.DataFrame({"id": [0, 1, 9]}), "id").edges( + pl.DataFrame({"s": [0, 9], "d": [9, 9]}), "s", "d") + + def ids(g): + keys = _keys(g, ops) + assert keys is not None + return sorted(keys.get_column("id").to_list()) + + assert ids(dangling.gfql_index_all()) == [] == ids(dangling) + assert ids(covered.gfql_index_all()) == [0, 9] == ids(covered) + + +@pytest.mark.parametrize("nodes,keys,expected", [ + (pl.DataFrame({"id": [0, 1, 2]}), [0, 2], True), + (pl.DataFrame({"id": [0, 1, 2]}), [0, 9], False), + (pl.DataFrame({"id": [0, None, 2]}), [0, 2], False), + (pl.DataFrame({"id": ["a", "b"]}), [0, 1], False), + (pl.LazyFrame({"id": [0, 1, 2]}), [0, 2], False), + (None, [0], False), + (pl.DataFrame({"other": [0, 1]}), [0], False), +]) +def test_node_coverage_check_declines_anything_it_cannot_compare(nodes, keys, expected): + """Every way the node table can fail to answer 'are these ids all nodes?' — absent, + lazy, wrong column, null id, incomparable dtype — must read as NOT covered.""" + import numpy as np + from graphistry.compute.gfql.lazy.engine.polars.pattern_apply import _nodes_cover_keys + + g = graphistry.edges(pl.DataFrame({"s": [0], "d": [1]}), "s", "d") + if nodes is not None: + g = g.nodes(nodes, "id" if "id" in nodes.collect_schema().names() else "other") + assert _nodes_cover_keys(g, "id", np.asarray(keys)) is expected + + +@pytest.mark.parametrize("direction", ["forward", "reverse", "undirected"]) +def test_adjacency_membership_still_answers_the_distinct_alias_shapes(direction, monkeypatch): + """The guards must not silently retire the adjacency route for the shapes it does + answer — otherwise the agreement matrix above proves nothing.""" + from graphistry.compute.ast import e_forward, e_reverse, e_undirected, n + import graphistry.compute.gfql.index.degrees as index_degrees + + edge_op = {"forward": e_forward, "reverse": e_reverse, "undirected": e_undirected}[direction]() + g = graphistry.nodes(pl.DataFrame({"id": [0, 1, 2]}), "id").edges( + pl.DataFrame({"s": [0, 1], "d": [1, 2]}), "s", "d").gfql_index_all() + + seen = [] + orig = index_degrees.adjacency_membership_keys + + def wrapped(registry, direction, edges_df, cols, engine): + seen.append(direction) + return orig(registry, direction, edges_df, cols, engine) + + monkeypatch.setattr(index_degrees, "adjacency_membership_keys", wrapped) + keys = _keys(g, [n(name="n"), edge_op, n(name="m")]) + assert keys is not None + assert seen, "the adjacency route was not consulted at all" + expected = {"forward": [0, 1], "reverse": [1, 2], "undirected": [0, 1, 2]}[direction] + assert sorted(keys.get_column("id").to_list()) == expected diff --git a/graphistry/tests/compute/gfql/index/test_index.py b/graphistry/tests/compute/gfql/index/test_index.py index 9e541740bd..177eb1aed1 100644 --- a/graphistry/tests/compute/gfql/index/test_index.py +++ b/graphistry/tests/compute/gfql/index/test_index.py @@ -1295,16 +1295,40 @@ def node_ids(gg): # test exists for, and it holds on nodes as well as edges. assert node_ids(candidate) == node_ids(whole_column), f"[{shape}] node sets differ" - # Against the scan we compare only the nodes the scan also produces. There is a - # PRE-EXISTING indexed-vs-scan divergence, unrelated to this PR and present identically - # on master 84be35fb: for an undirected to_fixed_point wavefront hop the indexed path - # keeps the SEED in `_nodes` while the scan drops it when the walk never returns to it - # (edges are identical). Asserting equality here would encode that bug as expected; this - # asserts the indexed result is a superset and that any excess is exactly the seed. - seed_ids = set(seeds["id"].to_list() if hasattr(seeds["id"], "to_list") else seeds["id"].tolist()) - extra = node_ids(candidate) - node_ids(scan) - assert not (node_ids(scan) - node_ids(candidate)), f"[{shape}] indexed path LOST nodes" - assert extra <= seed_ids, f"[{shape}] indexed path gained non-seed nodes: {sorted(extra)[:5]}" + +@pytest.mark.parametrize("engine", _cpu_engines()) +@pytest.mark.parametrize("shape", [ + "one_hop", + "two_hop", + "fixed_point", + pytest.param("undirected", marks=pytest.mark.xfail(strict=True, reason=( + "known divergence: for an undirected to_fixed_point wave-front hop the indexed path " + "keeps the SEED in `_nodes` while the scan drops it when the walk never returns to it " + "(edges are identical; 1957 indexed node rows vs 1956 scanned). Strict, so it flips " + "the moment the wave-front seed handling is unified."))), +]) +def test_indexed_wavefront_node_set_matches_the_scan(typed_graph, engine, shape): + """The index must not change WHICH NODES a wave-front hop reports, only how fast it + gets there — the scan is the oracle.""" + from graphistry.Engine import Engine as _E, df_to_engine + + g = typed_graph + if engine == "polars": + g = g.edges(df_to_engine(g._edges, _E.POLARS), "src", "dst").nodes( + df_to_engine(g._nodes, _E.POLARS), "id") + gi = g.gfql_index_all(engine=engine) + seeds = g._nodes[:1] if engine == "pandas" else g._nodes.head(1) + kw = dict(one_hop=dict(hops=1, direction="forward"), + two_hop=dict(hops=2, direction="forward"), + fixed_point=dict(to_fixed_point=True, direction="forward"), + undirected=dict(to_fixed_point=True, direction="undirected"))[shape] + kwargs = dict(return_as_wave_front=True, edge_match={"etype": 1}, engine=engine, **kw) + + def node_ids(gg): + s = gg._nodes["id"] + return set(s.to_list() if hasattr(s, "to_list") else s.tolist()) + + assert node_ids(gi.hop(nodes=seeds, **kwargs)) == node_ids(g.hop(nodes=seeds, **kwargs)) @pytest.mark.parametrize("engine", _cpu_engines()) diff --git a/graphistry/tests/compute/gfql/test_aggregate_type_contract.py b/graphistry/tests/compute/gfql/test_aggregate_type_contract.py index c6d20878dc..67dd3c8269 100644 --- a/graphistry/tests/compute/gfql/test_aggregate_type_contract.py +++ b/graphistry/tests/compute/gfql/test_aggregate_type_contract.py @@ -20,6 +20,12 @@ skipped GPU param states its own boundary in the pytest report rather than passing quietly. Deliberately does NOT use ``available_nonpandas_engines()``: that helper SHRINKS the parametrization silently when a stack is missing, so a lane can vanish without any signal. + +THE BOOLEAN LANE at the bottom of this file pins the documented ``sum``/``avg``-over-BOOLEAN +extension as VALUES AND RETURN TYPES. Values alone were already uniform; the return types were not +(polars answered ``sum``/``count`` with ``UInt32``, pandas and cuDF with ``int64``), and cuDF +answered ``sum`` over a group with no non-null values with NULL where Cypher says 0 -- a VALUE +divergence that only an exercised GPU arm could find. """ import datetime @@ -268,10 +274,13 @@ def test_polars_native_null_dtype_column_follows_the_all_null_contract(engine): edges = pl.DataFrame({"src": [0, 1], "dst": [1, 2]}) assert nodes.schema["nul"] == pl.Null g = graphistry.nodes(nodes, "id").edges(edges, "src", "dst") - got = _run(g, "MATCH (n) RETURN n.grp AS grp, sum(n.nul) AS s, avg(n.nul) AS a ORDER BY grp", - engine) + query = "MATCH (n) RETURN n.grp AS grp, sum(n.nul) AS s, avg(n.nul) AS a ORDER BY grp" + got = _run(g, query, engine) assert got == ("ok", [(("grp", "x"), ("s", ("num", 0.0)), ("a", None)), (("grp", "y"), ("s", ("num", 0.0)), ("a", None))]), got + # The substituted 0 is a LITERAL, not a kernel answer, so it carries whatever dtype the + # literal was built with -- a bare `pl.lit(0)` is Int32, a width pandas/cuDF never produce. + assert _dtype_kind(g.gfql(query, engine=engine)._nodes, "s") == "int64" @pytest.mark.parametrize("engine", ALL_ENGINES) @@ -282,9 +291,14 @@ def test_all_null_column_sums_to_zero_and_averages_to_null(engine): on dtype (pandas) or raise for both str and null dtypes (polars).""" _require_engine(engine) g = _graph() - got = _run(g, "MATCH (n) RETURN n.grp AS grp, sum(n.allnull_col) AS s, avg(n.allnull_col) AS a ORDER BY grp", engine) + query = ("MATCH (n) RETURN n.grp AS grp, sum(n.allnull_col) AS s, avg(n.allnull_col) AS a " + "ORDER BY grp") + got = _run(g, query, engine) assert got == ("ok", [(("grp", "x"), ("s", ("num", 0.0)), ("a", None)), (("grp", "y"), ("s", ("num", 0.0)), ("a", None))]), got + # Same substituted-literal dtype question as the polars-native Null column above, reached by + # the OTHER branch: this column arrives typed (an all-None pandas object column lands String). + assert _dtype_kind(g.gfql(query, engine=engine)._nodes, "s") == "int64" @pytest.mark.parametrize("engine", ALL_ENGINES) @@ -526,6 +540,367 @@ def test_empty_group_aggregation_sets_cover_both_distinct_spellings(): ) +# -------------------------------------------------------------------------------------- +# The BOOLEAN extension: sum/avg over BOOLEAN, pinned as VALUES **and** RETURN TYPES +# -------------------------------------------------------------------------------------- + +#: The contract (agg_types.py): sum -> INTEGER, avg -> FLOAT, min/max -> BOOLEAN, count -> INTEGER. +_BOOL_RESULT_DTYPES = {"s": "int64", "a": "float64", "mn": "bool", "mx": "bool", "c": "int64"} + +_BOOL_AGGS = ("sum(n.flag) AS s, avg(n.flag) AS a, min(n.flag) AS mn, " + "max(n.flag) AS mx, count(n.flag) AS c") + +#: The four rows the return-type gap and the min/max-as-AND/OR misreading both surface on. +#: `all_null` is the discriminating one: a logical fold answers min->true / max->false there +#: (the conventional AND/OR empty identities), and every engine answers NULL. +_BOOL_ROWS = { + "mixed": ([True, False, True], {"s": 2, "a": 2.0 / 3.0, "mn": False, "mx": True, "c": 3}), + "with_null": ([True, None, False], {"s": 1, "a": 0.5, "mn": False, "mx": True, "c": 2}), + "all_null": ([None, None], {"s": 0, "a": None, "mn": None, "mx": None, "c": 0}), + "all_true": ([True, True], {"s": 2, "a": 1.0, "mn": True, "mx": True, "c": 2}), + "all_false": ([False, False], {"s": 0, "a": 0.0, "mn": False, "mx": False, "c": 2}), +} + + +def _bool_graph(values): + """Nullable ``boolean``, not numpy ``bool``: the contract's null rows are unrepresentable in a + numpy bool column, and the nullable dtype is what survives the trip to polars and cuDF.""" + nodes = pd.DataFrame({"id": list(range(len(values))), "grp": ["x"] * len(values), + "flag": pd.array(values, dtype="boolean")}) + edges = pd.DataFrame({"src": [0], "dst": [0]}) + return graphistry.nodes(nodes, "id").edges(edges, "src", "dst") + + +def _dtype_kind(df, col): + """Engine-neutral dtype label: ``int64`` / ``float64`` / ``bool``, else the raw spelling. + + Collapses ONLY the nullability spelling -- pandas ``Int64``/``boolean``, polars + ``Int64``/``Boolean``, cuDF ``int64``/``bool`` all name the same contract type, and which of + them a column lands on is the separate nullable-merge axis (#1796 BU1). WIDTH and SIGNEDNESS + are deliberately NOT collapsed, so polars' ``UInt32`` reports as ``UInt32`` and fails. + """ + raw = str(df.schema[col]) if "polars" in type(df).__module__ else str(df[col].dtype) + lowered = raw.lower() + if lowered == "int64": + return "int64" + if lowered == "float64": + return "float64" + if lowered in {"bool", "boolean"}: + return "bool" + return raw + + +def _scalar(df, col): + """The single aggregate row's value, normalized to a python scalar with NULL as ``None``. + + Per-value, NOT ``df.where(df.notna(), None)``: py3.13 pandas renders a missing value as ``nan`` + where 3.12 gave ``None``, and this lane is ABOUT null behaviour, so the null test is explicit. + """ + if "polars" in type(df).__module__: + value = df.to_dicts()[0][col] + else: + value = df[col].iloc[0] + if value is pd.NA: + value = None + if hasattr(value, "item") and not isinstance(value, bool): + value = value.item() + if isinstance(value, float) and value != value: + value = None + return value + + +def _rows(df): + """Records with NULL as ``None`` on every engine, normalized PER VALUE. + + Not ``df.where(df.notna(), None)``: that reshapes the frame and, on py3.13 pandas, renders a + missing value as ``nan`` rather than ``None`` -- the exact distinction this lane tests. + """ + if "polars" in type(df).__module__: + records = df.to_dicts() + else: + if hasattr(df, "to_pandas"): # cudf + df = df.to_pandas() + records = df.to_dict("records") + out = [] + for record in records: + row = {} + for key, value in record.items(): + if value is pd.NA: + value = None + if hasattr(value, "item") and not isinstance(value, bool): + value = value.item() + row[key] = None if isinstance(value, float) and value != value else value + out.append(row) + return out + + +def _assert_bool_contract(df, expected): + """VALUES and DTYPES together: values alone already agreed across engines before this lane.""" + for col, want in expected.items(): + got = _scalar(df, col) + if want is None: + assert got is None, f"{col}: expected NULL, got {got!r}" + elif isinstance(want, bool): + # `is` on the identity, not `==`: True == 1 would let an integer min/max pass. + assert isinstance(got, bool) and got == want, f"{col}: expected {want!r}, got {got!r}" + elif isinstance(want, float): + assert abs(got - want) < 1e-9, f"{col}: expected {want!r}, got {got!r}" + else: + assert got == want and not isinstance(got, bool), f"{col}: expected {want!r}, got {got!r}" + assert _dtype_kind(df, col) == _BOOL_RESULT_DTYPES[col], ( + f"{col}: dtype {_dtype_kind(df, col)!r} != contract {_BOOL_RESULT_DTYPES[col]!r}") + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.parametrize("row", sorted(_BOOL_ROWS)) +@pytest.mark.parametrize("grouped", [True, False]) +def test_boolean_aggregate_values_and_return_types(engine, row, grouped, request): + """sum -> INTEGER(int64), avg -> FLOAT(float64), min/max -> BOOLEAN, count -> INTEGER(int64), + identically on every engine. Polars used to answer sum/count with ``UInt32``: the same value + behind a different return type, which is exactly the cross-engine divergence class the + aggregate type contract exists to close.""" + _require_engine(engine) + values, expected = _BOOL_ROWS[row] + query = (f"MATCH (n) RETURN n.grp AS grp, {_BOOL_AGGS} ORDER BY grp" if grouped + else f"MATCH (n) RETURN {_BOOL_AGGS}") + out = _bool_graph(values).gfql(query, engine=engine)._nodes + assert len(out) == 1, out + _assert_bool_contract(out, expected) + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_boolean_min_max_are_an_ordering_not_a_logical_fold(engine): + """``min == AND`` / ``max == OR`` is a DERIVATION from ``false < true``, not a definition, and + it predicts the wrong answer on the empty fold: AND over zero elements is conventionally + ``true`` and OR over zero elements ``false``, while every engine answers NULL. Pinned as the + ordering instead -- the same one ``ORDER BY`` gives booleans.""" + _require_engine(engine) + out = _bool_graph([None, None]).gfql( + f"MATCH (n) RETURN {_BOOL_AGGS}", engine=engine)._nodes + assert _scalar(out, "mn") is None, "min over no non-null values must be NULL, not the AND identity true" + assert _scalar(out, "mx") is None, "max over no non-null values must be NULL, not the OR identity false" + ordered = _bool_graph([True, False, True]).gfql( + f"MATCH (n) RETURN {_BOOL_AGGS}", engine=engine)._nodes + assert _scalar(ordered, "mn") is False and _scalar(ordered, "mx") is True + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_boolean_sum_over_zero_rows_is_cypher_zero_not_sql_null(engine): + """Cypher's ``sum()`` returns **0** over zero rows where SQL's returns NULL, and its ``avg()`` + returns null; both engines already matched Cypher, so this is conformance rather than a + compromise. The 0-row shape reaches it by a DIFFERENT route than the all-null column above -- + the ungrouped-aggregate identity row, not the aggregate kernel.""" + _require_engine(engine) + out = _bool_graph([True, False, True]).gfql( + f"MATCH (n) WHERE n.id > 9999 RETURN {_BOOL_AGGS}", engine=engine)._nodes + assert len(out) == 1, out + assert _scalar(out, "s") == 0 and _scalar(out, "c") == 0 + assert _dtype_kind(out, "s") == "int64" and _dtype_kind(out, "c") == "int64" + for col in ("a", "mn", "mx"): + assert _scalar(out, col) is None, col + # The identity row carries no type evidence for the NULL columns (`avg`/`min`/`max` land on + # pandas `object` / polars `Null`), so their dtypes are NOT asserted here. That gap is not + # boolean-specific -- `avg` over an empty INTEGER column loses its dtype the same way -- and + # is registered on the per-engine semantics matrix rather than pinned to today's behaviour. + + +@pytest.mark.parametrize("engine", ["polars", "cudf", "polars-gpu"]) +@pytest.mark.parametrize("row", sorted(_BOOL_ROWS)) +def test_boolean_aggregate_return_types_match_the_pandas_oracle(engine, row): + """The differential form of the lane above: every engine's boolean aggregate must land on the + SAME dtype kind as pandas, so a future engine-local dtype drift fails here even if someone + edits the contract table.""" + _require_engine(engine) + values, _ = _BOOL_ROWS[row] + query = f"MATCH (n) RETURN n.grp AS grp, {_BOOL_AGGS} ORDER BY grp" + oracle = _bool_graph(values).gfql(query, engine="pandas")._nodes + got = _bool_graph(values).gfql(query, engine=engine)._nodes + for col in _BOOL_RESULT_DTYPES: + assert _dtype_kind(got, col) == _dtype_kind(oracle, col), col + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_count_returns_integer_on_every_input_type(engine): + """``count()`` is INTEGER in Cypher for ANY input. Polars answered it ``UInt32`` for every + dtype while pandas/cuDF answered ``int64``, so aligning it is wider than the boolean rule -- + but it is the same divergence and the same direction.""" + _require_engine(engine) + g = _graph() + out = g.gfql( + "MATCH (n) RETURN n.grp AS grp, count(n.int_col) AS ci, count(n.str_col) AS cs, " + "count(n.bool_col) AS cb, count(DISTINCT n.str_col) AS cd, count(*) AS ca ORDER BY grp", + engine=engine)._nodes + for col in ("ci", "cs", "cb", "cd", "ca"): + assert _dtype_kind(out, col) == "int64", f"{col}: {_dtype_kind(out, col)}" + + +def _bool_fast_path_graph(): + """Fast-path shape with TWO cities, the second of which has only null flags -- the group whose + ``sum`` cuDF answers NULL and Cypher answers 0. A single-city graph never produces that group, + so it cannot exercise the repair.""" + nodes = pd.DataFrame({ + "id": [0, 1, 2, 3, 10, 11], + "node_type": ["Person"] * 4 + ["City"] * 2, + "age": [20, 30, 40, 50, None, None], + "flag": pd.array([True, False, None, None, None, None], dtype="boolean"), + "allnull": [None] * 6, + "city": [None] * 4 + ["LA", "NYC"], + }) + edges = pd.DataFrame({"s": [0, 1, 2, 3], "d": [11, 11, 10, 10], "rel": ["LIVES_IN"] * 4}) + return graphistry.nodes(nodes, "id").edges(edges, "s", "d") + + +_BOOL_FAST_PATH_HEAD = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "RETURN c.city AS city, ") + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_fast_path_boolean_aggregate_follows_the_same_contract(engine): + """The OLAP fast path reimplements the aggregates on both engine branches, so without its own + conformance the SAME boolean query would answer with a different return type depending on + whether it happened to match the fast-path shape. The LA group has no non-null flag: its + ``sum`` is Cypher's 0, which cuDF's kernel answers NULL.""" + _require_engine(engine) + g = _bool_fast_path_graph() + entered, _ = _run_watching_fast_path(g, _fast_path_query("sum", "p.flag"), engine) + assert entered, "fast path not engaged -- this lane would not be exercising its aggregates" + out = g.gfql( + _BOOL_FAST_PATH_HEAD + "sum(p.flag) AS s, avg(p.flag) AS a, count(p.flag) AS c " + "ORDER BY city", engine=engine)._nodes + rows = _rows(out) + assert rows[0]["city"] == "LA" and rows[1]["city"] == "NYC" + assert rows[0]["s"] == 0 and rows[0]["a"] is None and rows[0]["c"] == 0 + assert rows[1]["s"] == 1 and abs(rows[1]["a"] - 0.5) < 1e-9 and rows[1]["c"] == 2 + assert _dtype_kind(out, "s") == "int64" and _dtype_kind(out, "a") == "float64" + assert _dtype_kind(out, "c") == "int64" + + +@pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) +def test_fast_path_eager_polars_twin_conforms_when_the_fused_lane_declines(engine): + """The fast path has THREE polars aggregate formulations -- a fused lazy lane, a + ``value_counts`` plan for a low-cardinality pure ``count(*)``, and the eager twin the other two + decline to. All three must land on the same return types, or which one a query happens to + route to becomes observable. An all-null aggregate input declines the fused lane, so this + query reaches the eager twin with the other aggregates still on it.""" + _require_engine(engine) + out = _bool_fast_path_graph().gfql( + _BOOL_FAST_PATH_HEAD + "sum(p.allnull) AS z, count(*) AS n, count(p.age) AS ca, " + "sum(p.flag) AS s ORDER BY city", engine=engine)._nodes + for col in ("z", "n", "ca", "s"): + assert _dtype_kind(out, col) == "int64", f"{col}: {_dtype_kind(out, col)}" + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_fast_path_pure_count_star_returns_integer(engine): + """A single-key pure ``count(*)`` over statically-bounded-low inputs takes a ``value_counts`` + formulation that skips the aggregate expressions entirely, so it needs its own conformance -- + it answered ``UInt32`` while the ``group_by`` formulation beside it answered ``int64``, which + made the two lanes value-identical but NOT type-identical.""" + _require_engine(engine) + out = _bool_fast_path_graph().gfql( + _BOOL_FAST_PATH_HEAD + "count(*) AS n ORDER BY city", engine=engine)._nodes + assert _dtype_kind(out, "n") == "int64", _dtype_kind(out, "n") + assert [row["n"] for row in _rows(out)] == [2, 2] + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_fast_path_count_star_beside_another_aggregate_returns_integer(engine): + """The ``value_counts`` plan above serves a PURE ``count(*)`` only, so a ``count(*)`` sharing + its RETURN with another aggregate reaches the fused lane's own ``pl.len()`` instead -- a + fourth count formulation, and one the pure-count test cannot reach.""" + _require_engine(engine) + out = _bool_fast_path_graph().gfql( + _BOOL_FAST_PATH_HEAD + "count(*) AS n, sum(p.age) AS s, count(p.flag) AS cf " + "ORDER BY city", engine=engine)._nodes + for col in ("n", "cf"): + assert _dtype_kind(out, col) == "int64", f"{col}: {_dtype_kind(out, col)}" + assert [row["n"] for row in _rows(out)] == [2, 2] + + +def test_boolean_result_contract_is_the_one_in_agg_types(): + """The table above is a RESTATEMENT of the shipped contract, so a change to one that is not a + change to the other is a drift this catches rather than a silent disagreement.""" + from graphistry.compute.gfql.agg_types import agg_result_is_integer + for func, alias in [("sum", "s"), ("count", "c")]: + assert agg_result_is_integer(func, True), func + assert _BOOL_RESULT_DTYPES[alias] == "int64" + for func in ("avg", "mean", "min", "max"): + assert not agg_result_is_integer(func, True), func + assert agg_result_is_integer("count_distinct", False) + # sum is INTEGER only BECAUSE the input is boolean -- a numeric sum keeps its own width + assert not agg_result_is_integer("sum", False) + + +def test_polars_agg_result_cast_fires_only_where_polars_misses_the_contract(): + """The cast is scoped, not blanket: widening a FLOAT sum or a DURATION sum to Int64 would be a + silent wrong answer, so the helper must decline everything except the two INTEGER cells.""" + from graphistry.compute.gfql.agg_types import polars_agg_result_cast as cast_to + assert cast_to("sum", pl.Boolean) == pl.Int64 + assert cast_to("count", pl.Boolean) == pl.Int64 + assert cast_to("count", pl.String) == pl.Int64 # count is INTEGER over ANY input + assert cast_to("count", None) == pl.Int64 # count(*) has no input column + assert cast_to("count_distinct", pl.Float64) == pl.Int64 + assert cast_to("sum", pl.Int64) is None # polars already sums ints to Int64 + assert cast_to("sum", pl.Int8) is None + assert cast_to("sum", pl.Float64) is None # FLOAT sum must stay FLOAT + assert cast_to("sum", pl.Duration) is None # DURATION sum must stay DURATION + assert cast_to("sum", None) is None + assert cast_to("avg", pl.Boolean) is None + assert cast_to("mean", pl.Boolean) is None + assert cast_to("min", pl.Boolean) is None + assert cast_to("max", pl.Boolean) is None + assert cast_to("collect", pl.Boolean) is None + + +def test_polars_boolean_sum_null_fill_is_scoped_to_exact_boundary(): + """Only Boolean ``sum`` repairs a null aggregate result to Cypher's zero.""" + from graphistry.compute.gfql.agg_types import polars_conform_agg_dtype as conform + + source = pl.DataFrame({"one": [1]}) + + boolean_sum = source.select(conform(pl.lit(None), "sum", pl.Boolean, "out")) + assert boolean_sum.schema["out"] == pl.Int64 + assert boolean_sum["out"][0] == 0 + + boolean_count = source.select(conform(pl.lit(None), "count", pl.Boolean, "out")) + assert boolean_count.schema["out"] == pl.Int64 + assert boolean_count["out"][0] is None + + integer_sum = source.select(conform(pl.lit(None), "sum", pl.Int64, "out")) + assert integer_sum.schema["out"] == pl.Null + assert integer_sum["out"][0] is None + + +def test_polars_all_null_literal_is_a_typed_integer_zero(): + """A bare ``pl.lit(0)`` is ``Int32`` -- a width neither pandas nor cuDF ever produces, so the + all-null substitution would reintroduce the very dtype divergence the cast above removes.""" + from graphistry.compute.gfql.agg_types import polars_all_null_agg_literal + frame = pl.DataFrame({"x": [1]}).select(polars_all_null_agg_literal("sum", "s"), + polars_all_null_agg_literal("avg", "a")) + assert frame.schema["s"] == pl.Int64, frame.schema + assert frame["s"][0] == 0 + assert frame["a"][0] is None + + +def test_pandas_agg_kernel_null_fill_repairs_only_sum(): + """Cypher's ``sum()`` never answers null, so a null kernel answer is a bug to repair; ``avg`` + and ``min``/``max`` DO answer null and must not be filled, or an all-null group would silently + report 0 for an average.""" + from graphistry.compute.gfql.agg_types import pandas_agg_kernel_null_fill as fill + assert fill("sum", pd.Series([1, 2], dtype="Int64")) == 0 + assert fill("sum", pd.Series([True], dtype="boolean")) == 0 + assert fill("sum", pd.Series([1.0])) == 0 + assert fill("avg", pd.Series([1, 2])) is None + assert fill("mean", pd.Series([1, 2])) is None + assert fill("min", pd.Series([True])) is None + assert fill("max", pd.Series([True])) is None + assert fill("count", pd.Series([1])) is None + assert fill("collect", pd.Series([1])) is None + # object columns keep the existing bool-retype route rather than gaining a second one + assert fill("sum", pd.Series([True, None], dtype="object")) is None + + def test_numeric_only_aggregation_set_covers_both_spellings(): """``avg`` is the cypher name and ``mean`` GFQL's internal one (GFQL_GROUPBY_AGG_METHODS maps avg -> mean); a set holding only one of them would leave the other unguarded.""" diff --git a/graphistry/tests/compute/gfql/test_alias_scoping_semantics.py b/graphistry/tests/compute/gfql/test_alias_scoping_semantics.py index 25975f4212..0497def84e 100644 --- a/graphistry/tests/compute/gfql/test_alias_scoping_semantics.py +++ b/graphistry/tests/compute/gfql/test_alias_scoping_semantics.py @@ -21,11 +21,14 @@ Discriminating controls from the probe are pinned alongside each fix so a future regression cannot pass by repairing only one side. """ +from typing import Literal + import pandas as pd import pytest import graphistry -from graphistry.compute.exceptions import ErrorCode, GFQLValidationError +from graphistry.compute.exceptions import ErrorCode, GFQLTypeError, GFQLValidationError +from graphistry.compute.gfql.cypher.api import compile_cypher pl = pytest.importorskip("polars") @@ -88,13 +91,16 @@ def _graph(nodes: pd.DataFrame, edges: pd.DataFrame, engine: str): return graphistry.nodes(nodes, "id").edges(edges, "s", "d") +def _null(v): + return None if isinstance(v, float) and v != v else v + + def _rows(g, query: str, engine: str): frame = g.gfql(query, engine=engine)._nodes if frame is None: return [] - if isinstance(frame, pl.DataFrame): - return frame.to_dicts() - return frame.to_dict("records") + records = frame.to_dicts() if isinstance(frame, pl.DataFrame) else frame.to_dict("records") + return [{k: _null(v) for k, v in r.items()} for r in records] def _run(nodes, edges, query: str, engine: str): @@ -188,10 +194,15 @@ def test_with_non_rebind_shapes_are_unaffected(query: str, expected, engine: str @pytest.mark.parametrize("engine", ENGINES) def test_terminal_return_rename_onto_live_alias_still_works(engine: str) -> None: """CONTROL: a terminal `RETURN a AS b` only names an output column -- no later clause - resolves against it -- so it must keep working on both engines.""" + resolves against it -- so it must keep working on both engines. + + The KNOWS bag is a->b, b->c, a->c, c->d, so the a-side is [a, b, a, c] and Alice + appears twice. This used to expect the 3-name node set; the sibling property spelling + (`WITH a.name AS b RETURN b`, in test_with_non_rebind_shapes_are_unaffected) already + expected the 4-row bag, and whole-entity projection now agrees with it.""" rows = _run(PEOPLE_NODES, PEOPLE_EDGES, "MATCH (a:Person)-[r:KNOWS]->(b:Person) RETURN a AS b", engine) - assert sorted(r["b.name"] for r in rows) == ["Alice", "Bob", "Carol"] + assert sorted(r["b.name"] for r in rows) == ["Alice", "Alice", "Bob", "Carol"] @pytest.mark.parametrize("engine", ENGINES) @@ -318,46 +329,43 @@ def test_whole_entity_projection_of_shadowing_alias_omits_the_column(engine: str assert sorted(r["kind.name"] for r in rows) == ["One", "Three", "Two"] -# ------------------------------------------------------------------ residuals +# ---------------------------------------- defect-4 remaining shapes (fixed) @pytest.mark.parametrize("engine", ENGINES) -def test_residual_single_alias_and_cartesian_alias_marker_still_leaks(engine: str) -> None: # noqa: ARG001 - """RESIDUAL (#1911 defect 4, not fixed this cycle): the single-alias - `rows(table='nodes', source=...)` and the cartesian binding paths read properties off - the chain output frame, where the alias marker has ALREADY overwritten the user - column -- the pre-marker values are gone by then, so the fix needs the marker itself - to move to a safe internal name (which chain.py's own labeling machinery reads back). - Pinned CONSISTENT across engines so the divergence cannot reappear silently.""" - for query in ["MATCH (kind:P) RETURN kind.kind AS k", - "MATCH (kind:P), (b:P) RETURN kind.kind AS k, b.id AS bi", - # a bare relationship-property projection also skips the connected- - # bindings property attach that the fix hooks. - "MATCH (a:P)-[w:K]->(b:P) RETURN w.w AS k"]: - rows = _run(SHADOW_NODES, SHADOW_EDGES, query, "pandas") - assert {r["k"] for r in rows} == {True}, f"{query} -> {rows}" - - -def test_residual_multihop_relationship_alias_marker_divergence() -> None: - """RESIDUAL (#1911 defect 4): a relationship alias in a MULTI-hop pattern still takes - a different path than the single-hop one fixed above -- pandas reads the marker, - polars reads the value (and also over-multiplies the rows, a separate pre-existing - polars defect independent of the alias name). Pinned so the state is explicit.""" +def test_single_alias_and_cartesian_shadowed_alias_reads_the_user_value(engine: str) -> None: + """#1911 defect 4 (was RESIDUAL): the single-alias ``rows(table=..., source=...)`` + route and the cartesian binding path read properties off the chain output frame, + where the alias marker had overwritten the user column. The rows route now restores + the user values from the base frame (dotted self-column, marker kept boolean); the + cartesian paths unshadow like the connected one. Anti-vacuity: 3 node rows / 9 + cartesian rows / 2 relationship rows, distinct values -- an all-``True`` marker + leak or an empty frame cannot pass.""" + rows = _run(SHADOW_NODES, SHADOW_EDGES, "MATCH (kind:P) RETURN kind.kind AS k", engine) + assert sorted(r["k"] for r in rows) == ["K1", "K2", "K3"] + rows = _run(SHADOW_NODES, SHADOW_EDGES, + "MATCH (kind:P), (b:P) RETURN kind.kind AS k, b.id AS bi", engine) + assert len(rows) == 9 and sorted({r["k"] for r in rows}) == ["K1", "K2", "K3"] + # bare relationship-property projection (rows(table='edges', source=alias) route) + rows = _run(SHADOW_NODES, SHADOW_EDGES, "MATCH (a:P)-[w:K]->(b:P) RETURN w.w AS k", engine) + assert sorted(r["k"] for r in rows) == [7, 8] + + +def test_multihop_relationship_shadowed_alias_pandas_fixed_polars_multiplicity_residual() -> None: + """#1911 defect 4: pandas now reads the user value in the MULTI-hop shape too + (was the marker ``True``). polars still over-multiplies the rows -- a separate + pre-existing multiplicity defect independent of the alias name -- pinned as-is.""" query = "MATCH (a:P)-[w:K]->(b:P)-[:K]->(c:P) RETURN w.w AS x" - assert [r["x"] for r in _run(SHADOW_NODES, SHADOW_EDGES, query, "pandas")] == [True] + assert [r["x"] for r in _run(SHADOW_NODES, SHADOW_EDGES, query, "pandas")] == [7] assert sorted(r["x"] for r in _run(SHADOW_NODES, SHADOW_EDGES, query, "polars")) == [7, 8] -def test_residual_empty_result_drops_a_shadowed_alias_column_on_pandas() -> None: - """RESIDUAL (#1911 defect 4): when an alias is named after ANY existing node column - and the match is EMPTY, pandas' chain output loses that column outright, so the - downstream `rows` op fails its schema check; polars returns the correct empty result. - Reproduces with a non-colliding property too (`kind.name`), so it is a chain - empty-frame column-preservation bug rather than a property-resolution one.""" - from graphistry.compute.exceptions import GFQLSchemaError - +def test_empty_result_keeps_a_shadowed_alias_column_on_both_engines() -> None: + """#1911 defect 4 (was RESIDUAL): an EMPTY match on an alias named after an existing + node column used to lose that column on pandas (chain empty-frame column drop), so the + downstream ``rows`` op raised a schema error; the marker-aware coalesce now keeps the + column even at zero rows and both engines return the correct empty result.""" query = "MATCH (kind:P) WHERE kind.name = 'ZZ' RETURN kind.name AS n" - with pytest.raises(GFQLSchemaError): - _run(SHADOW_NODES, SHADOW_EDGES, query, "pandas") + assert _run(SHADOW_NODES, SHADOW_EDGES, query, "pandas") == [] assert _run(SHADOW_NODES, SHADOW_EDGES, query, "polars") == [] @@ -451,21 +459,36 @@ def test_with_rebind_edge_alias_onto_edge_alias_declines(engine: str) -> None: assert exc_info.value.context["value"] == "r AS q" +def test_node_onto_edge_entity_rebind_declines_at_compile_time() -> None: + query = "MATCH (a:Person)-[r:KNOWS]->(b:Person) WITH a AS r RETURN r.type AS t" + with pytest.raises(GFQLValidationError) as exc_info: + compile_cypher(query) + assert exc_info.value.code == ErrorCode.E108 + assert "rebind an entity alias" in str(exc_info.value) + assert exc_info.value.context["value"] == "a AS r" + + +def test_edge_onto_node_entity_rebind_declines_at_compile_time() -> None: + query = "MATCH (a:Person)-[r:KNOWS]->(b:Person) WITH r AS b RETURN b.w AS t" + with pytest.raises(GFQLValidationError) as exc_info: + compile_cypher(query) + assert exc_info.value.code == ErrorCode.E108 + assert "rebind an entity alias" in str(exc_info.value) + assert exc_info.value.context["value"] == "r AS b" + + @pytest.mark.parametrize("engine", ENGINES) -@pytest.mark.parametrize("query", [ - # node alias onto an edge-alias name and the reverse: outside the guard's - # same-kind scope, but they must stay ERRORS (never a silent split-read). - "MATCH (a:Person)-[r:KNOWS]->(b:Person) WITH a AS r RETURN r.type AS t", - "MATCH (a:Person)-[r:KNOWS]->(b:Person) WITH r AS b RETURN b.w AS t", - # a property read off a scalar rebind is a type error, not the shadowed entity - "MATCH (a:Person)-[r:KNOWS]->(b:Person) WITH a.name AS b RETURN b.name AS t", -], ids=["node_onto_edge", "edge_onto_node", "scalar_then_property"]) -def test_cross_kind_and_scalar_rebinds_stay_errors(query: str, engine: str) -> None: - with pytest.raises(Exception) as exc_info: - _run(PEOPLE_NODES, PEOPLE_EDGES, query, engine) - assert type(exc_info.value).__name__ in ( - "GFQLTypeError", "GFQLValidationError", "NotImplementedError" - ) +def test_scalar_rebind_stays_an_error(engine: Literal["pandas", "polars"]) -> None: + query = "MATCH (a:Person)-[r:KNOWS]->(b:Person) WITH a.name AS b RETURN b.name AS t" + if engine == "pandas": + with pytest.raises(GFQLTypeError) as exc_info: + _run(PEOPLE_NODES, PEOPLE_EDGES, query, engine) + assert exc_info.value.code == ErrorCode.E303 + assert exc_info.value.context["field"] == "function" + assert exc_info.value.context["value"] == "select" + else: + with pytest.raises(NotImplementedError): + _run(PEOPLE_NODES, PEOPLE_EDGES, query, engine) @pytest.mark.parametrize("engine", ENGINES) @@ -489,16 +512,172 @@ def test_unwind_alias_collision_still_declines_before_the_rebind_guard(engine: s assert "UNWIND alias collides" in str(exc_info.value) -def test_edge_alias_named_like_source_column_is_a_schema_error_residual() -> None: - """RESIDUAL: an edge alias named after the edge SOURCE column has its marker - destroy that column -- surfaced as a typed GFQLSchemaError (column-not-found), - not a silent answer. Pinned so a change here is deliberate; a typed decline - naming the alias collision (like the node-ID one) would be the upgrade.""" +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("alias", ["s", "d"]) +def test_edge_alias_named_like_an_endpoint_binding_is_a_typed_decline(alias: str, engine: str) -> None: + """#1911 defect-4 sibling (was RESIDUAL): an edge alias named after the edge + SOURCE/DESTINATION binding column has its marker destroy the endpoints. It used to + surface as an incidental GFQLSchemaError on pandas; the marker-aware coalesce would + have turned it into a silent EMPTY result, so it is now the same typed decline as + the node-ID collision, on both engines.""" + with pytest.raises(GFQLValidationError) as exc_info: + _run(PEOPLE_NODES, PEOPLE_EDGES, + f"MATCH (a:Person)-[{alias}:KNOWS]->(b:Person) RETURN {alias}.type AS t", engine) + assert exc_info.value.code == ErrorCode.E108 + assert "edge endpoint binding column" in str(exc_info.value) + + +# -------------------- #1911 defect-4 round-2: single-alias rows-route restore + +# NULL cell on purpose: the restore must carry the NULL through (the NULL-vs-membership +# class), never drop the row or backfill the marker. +SELF_NAMED_NODES = pd.DataFrame({ + "id": ["a1", "a2", "b1", "b2"], + "name": ["Sa", "Sb", None, "Tb"], + "label__P": [True, True, True, True], +}) +SELF_NAMED_EDGES = pd.DataFrame({ + "s": ["a1", "a2"], "d": ["b1", "b2"], "type": ["K", "K"], "w": [7, 8], +}) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_rows_route_alias_named_as_its_property_reads_user_values(engine: str) -> None: + """#1911 defect-4 (rows route): ``MATCH (name:P) RETURN name.name`` answered the + alias marker ``[True] x 4`` on both engines (cuDF crashed with a raw mixed-types + TypeError). Anti-vacuity: 4 rows with 3 distinct values plus a preserved NULL.""" + rows = _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, "MATCH (name:P) RETURN name.name", engine) + assert [r["name.name"] for r in rows] == ["Sa", "Sb", None, "Tb"] + + +@pytest.mark.parametrize("engine", ENGINES) +def test_rows_route_where_on_self_named_alias_filters_and_projects_user_values(engine: str) -> None: + """WHERE already compared user values (1 row matched) while RETURN projected the + marker ``True`` -- both sides must read the same user column.""" + rows = _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, + "MATCH (name:P) WHERE name.name = 'Sa' RETURN name.name AS n", engine) + assert rows == [{"n": "Sa"}] + + +@pytest.mark.parametrize("engine", ENGINES) +def test_rows_route_edge_alias_named_as_its_property_reads_user_values(engine: str) -> None: + """Edge twin of the rows route (``rows(table='edges', source=alias)``): the marker + also overwrote the same-named edge payload column.""" + rows = _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, + "MATCH (a:P)-[w:K]->(b:P) RETURN w.w AS x", engine) + assert sorted(r["x"] for r in rows) == [7, 8] + + +def test_rows_route_edge_alias_colliding_with_its_own_type_filter() -> None: + """``MATCH (a)-[type:K]->(b) RETURN type.type``: the alias shadows the very column + its ``:K`` filter reads. pandas/cuDF now answer the user values; polars' chain + machinery re-applies the type filter against the stamped marker and raises a typed + GFQLSchemaError -- an honest decline, pinned so it cannot rot into a silent wrong + answer (residual polish for #1911).""" from graphistry.compute.exceptions import GFQLSchemaError + query = "MATCH (a:P)-[type:K]->(b:P) RETURN type.type AS t" + assert _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, query, "pandas") == [ + {"t": "K"}, {"t": "K"}] with pytest.raises(GFQLSchemaError): - _run(PEOPLE_NODES, PEOPLE_EDGES, - "MATCH (a:Person)-[s:KNOWS]->(b:Person) RETURN s.type AS t", "pandas") + _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, query, "polars") + + +@pytest.mark.parametrize("engine", ENGINES) +def test_rows_route_self_named_plus_other_property(engine: str) -> None: + """The restore must not turn OTHER property reads of the same alias into NA: the + row table is not a bindings table just because the shadowed value was re-keyed + (an early restore design leaked exactly that, NA-ing ``name.id``).""" + rows = _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, + "MATCH (name:P) RETURN name.name AS n, name.id AS i", engine) + assert [(r["n"], r["i"]) for r in rows] == [ + ("Sa", "a1"), ("Sb", "a2"), (None, "b1"), ("Tb", "b2")] + + +def test_connected_join_carried_relationship_projection_unaffected() -> None: + """CONTROL (regression found in-flight): a comma-pattern whose base graph is an + intermediate dispatch frame must not have the restore misread the arm's marker as + user data — ``r.weight`` stays 7, never NA.""" + nodes = pd.DataFrame({"id": ["a1", "b1"], "label__A": [True, False], "label__B": [False, True]}) + edges = pd.DataFrame({"s": ["a1", "b1"], "d": ["b1", "a1"], "type": ["R", "S"], "w": [7, 9]}) + rows = _run(nodes, edges, + "MATCH (a:A {id: 'a1'})-[r:R]->(b:B), (b)-[:S]->(a) RETURN r.w AS w", "pandas") + assert rows == [{"w": 7}] + + +@pytest.mark.parametrize("engine", ENGINES) +def test_mixed_whole_entity_and_self_named_property_projection(engine: str) -> None: + """``RETURN name, name.name AS n``: the whole-entity flatten keeps omitting the + shadowed column (pinned above) while the explicit property column must read the + restored user values, NULL included.""" + rows = _run(SELF_NAMED_NODES, SELF_NAMED_EDGES, + "MATCH (name:P) RETURN name, name.name AS n", engine) + assert [r["n"] for r in rows] == ["Sa", "Sb", None, "Tb"] + assert [r["name.id"] for r in rows] == ["a1", "a2", "b1", "b2"] + + +def test_restore_alias_shadowed_user_column_branches() -> None: + """Helper-level pins for the rows-route restore: no-op without a base, a shadowed + column, or when the base column is itself a boolean marker (intermediate dispatch + graph); index-keyed restore; key-merge fallback when the index cannot re-key.""" + from types import SimpleNamespace + + from graphistry.compute.gfql.identifiers import shadow_restore_column + from graphistry.compute.gfql.row.frame_ops import _restore_alias_shadowed_user_column + + restore_col = shadow_restore_column("kind") + + def ctx_for(base_graph): + return SimpleNamespace(_gfql_rows_base_graph=base_graph, _g=None) + + marked = pd.DataFrame({"id": ["a", "b"], "kind": [True, True]}) + # no base graph / alias shadows nothing: unchanged + assert _restore_alias_shadowed_user_column(ctx_for(None), marked, "nodes", "kind") is marked + base_no_col = SimpleNamespace(_nodes=pd.DataFrame({"id": ["a", "b"]}), _edges=None, _node="id", _edge=None) + assert _restore_alias_shadowed_user_column(ctx_for(base_no_col), marked, "nodes", "kind") is marked + # base column is itself a boolean marker (an intermediate dispatch graph): unchanged + base_marker = SimpleNamespace(_nodes=pd.DataFrame({"id": ["a", "b"], "kind": [True, False]}), _edges=None, _node="id", _edge=None) + assert _restore_alias_shadowed_user_column(ctx_for(base_marker), marked, "nodes", "kind") is marked + # index-keyed restore adds the internal restore column and keeps the marker boolean + base = SimpleNamespace(_nodes=pd.DataFrame({"id": ["a", "b"], "kind": ["K1", "K2"]}), _edges=None, _node="id", _edge=None) + out = _restore_alias_shadowed_user_column(ctx_for(base), marked, "nodes", "kind") + assert list(out[restore_col]) == ["K1", "K2"] and list(out["kind"]) == [True, True] + # base index cannot re-key (duplicate labels): fall back to the id-key merge + dup_index_nodes = pd.DataFrame({"id": ["a", "b"], "kind": ["K1", "K2"]}, index=[0, 0]) + base_dup = SimpleNamespace(_nodes=dup_index_nodes, _edges=None, _node="id", _edge=None) + out = _restore_alias_shadowed_user_column(ctx_for(base_dup), marked, "nodes", "kind") + assert list(out[restore_col]) == ["K1", "K2"] + # neither index nor key can re-key: unchanged (marker stays, as before) + base_no_key = SimpleNamespace(_nodes=dup_index_nodes, _edges=None, _node=None, _edge=None) + assert _restore_alias_shadowed_user_column(ctx_for(base_no_key), marked, "nodes", "kind") is marked + # row-table labels absent from a unique base index: guarded .loc declines to the key merge + shifted = pd.DataFrame({"id": ["a", "b"], "kind": [True, True]}, index=[10, 11]) + out = _restore_alias_shadowed_user_column(ctx_for(base), shifted, "nodes", "kind") + assert list(out[restore_col]) == ["K1", "K2"] + # polars: id-keyed join replaces the marker column in place; no key -> unchanged + marked_pl = pl.DataFrame({"id": ["a", "b"], "kind": [True, True]}) + base_pl = SimpleNamespace(_nodes=pl.DataFrame({"id": ["a", "b"], "kind": ["K1", "K2"]}), _edges=None, _node="id", _edge=None) + out_pl = _restore_alias_shadowed_user_column(ctx_for(base_pl), marked_pl, "nodes", "kind") + assert out_pl["kind"].to_list() == ["K1", "K2"] + base_pl_no_key = SimpleNamespace(_nodes=base_pl._nodes, _edges=None, _node=None, _edge=None) + assert _restore_alias_shadowed_user_column(ctx_for(base_pl_no_key), marked_pl, "nodes", "kind") is marked_pl + # polars marker-carrying base: unchanged + base_pl_marker = SimpleNamespace(_nodes=pl.DataFrame({"id": ["a", "b"], "kind": [True, False]}), _edges=None, _node="id", _edge=None) + assert _restore_alias_shadowed_user_column(ctx_for(base_pl_marker), marked_pl, "nodes", "kind") is marked_pl + + +def test_cudf_rows_route_self_named_alias_parity() -> None: + """cuDF (dataframe ops only): the node shape crashed with a raw mixed-types + TypeError from the marker/user-column coalesce; both shapes now match pandas.""" + cudf = pytest.importorskip("cudf") + + g = graphistry.nodes(cudf.from_pandas(SELF_NAMED_NODES), "id").edges( + cudf.from_pandas(SELF_NAMED_EDGES), "s", "d" + ) + out = g.gfql("MATCH (name:P) RETURN name.name", engine="cudf")._nodes.to_pandas() + assert list(out["name.name"].fillna("")) == ["Sa", "Sb", "", "Tb"] + out = g.gfql("MATCH (a:P)-[type:K]->(b:P) RETURN type.type AS t", engine="cudf")._nodes + assert sorted(out.to_pandas()["t"]) == ["K", "K"] def test_cudf_unshadow_and_rebind_guard_parity() -> None: diff --git a/graphistry/tests/compute/gfql/test_count_and_param_semantics.py b/graphistry/tests/compute/gfql/test_count_and_param_semantics.py index 854bb44faf..ae42f90e9b 100644 --- a/graphistry/tests/compute/gfql/test_count_and_param_semantics.py +++ b/graphistry/tests/compute/gfql/test_count_and_param_semantics.py @@ -317,12 +317,10 @@ def test_undirected_unbounded_declines_typed_on_polars() -> None: # residuals -- pinned so they stop being invisible # --------------------------------------------------------------------------- -@pytest.mark.xfail( - strict=True, - reason="#1905: a node label absent from the graph schema hard-errors " - "[column-not-found]; openCypher treats a missing label as matching nothing " - "(0 rows). Needs an owner decision: strict-schema vs missing-as-absent.", -) +# WAS a strict xfail for #1905 ("a node label absent from the graph schema hard-errors +# [column-not-found]; openCypher treats a missing label as matching nothing"). The owner +# decision it waited on landed in #1916: strictness levels, defaulting to warn, under which +# an absent name resolves to null. Un-xfailed; the oracles below are UNCHANGED. @pytest.mark.parametrize("query", [ "MATCH (n:Nope) RETURN n.id AS id", "MATCH (n) WHERE n:Nope RETURN n.id AS id", @@ -331,13 +329,7 @@ def test_nonexistent_node_label_should_be_zero_rows(query: str) -> None: assert _rows(_param_graph("pandas").gfql(query)) == [] -@pytest.mark.xfail( - strict=True, - reason="#1905: a nonexistent label on an OPTIONAL arm hard-errors [column-not-found]; " - "openCypher null-extends the mandatory rows instead. Same owner decision. " - "The raised error's available-columns list also leaks the binder's alias " - "marker column.", -) +# WAS a strict xfail for #1905 (same owner decision as above); resolved by #1916. def test_nonexistent_optional_arm_label_should_null_extend() -> None: rows = _rows(_param_graph("pandas").gfql( "MATCH (a)-->(b) OPTIONAL MATCH (a)-->(c:Nope) RETURN a.id AS a, c.id AS c" diff --git a/graphistry/tests/compute/gfql/test_duration_month_division_1937.py b/graphistry/tests/compute/gfql/test_duration_month_division_1937.py new file mode 100644 index 0000000000..49cef54ea4 --- /dev/null +++ b/graphistry/tests/compute/gfql/test_duration_month_division_1937.py @@ -0,0 +1,158 @@ +"""End-to-end pins for scaling a duration whose month group has to be SPLIT (#1937). + +Scaling used to decline whenever the result was fractional in month-space, so +``duration('P1M') / 2`` raised instead of answering while ``duration('P2M') / 2`` +answered ``P1M``. openCypher resolves the split at the average month of 30.436875 days +(365.2425 / 12, the same constant this codebase already used for ``duration('P0.5M')``), +cascading months -> days -> seconds and truncating toward zero at each step. The +truncation is pinned alongside because the seconds group used to ROUND, which is a +visible 1ns disagreement (``PT2S / 3`` was ...667S where openCypher says ...666S). + +Every expected value below is an ORACLE taken from Neo4j 5.26.26 via ``cypher-shell``, +not from this implementation. The two halves are pinned together on purpose: the split +cases are the new behaviour, and the exact cases are the fence that keeps the average +month out of results that are exact in month-space. +""" +from __future__ import annotations + +import pandas as pd +import pytest + +import graphistry +from graphistry.compute.exceptions import GFQLTypeError +from graphistry.Plottable import Plottable + +pl = pytest.importorskip("polars") + +ENGINES = ["pandas", "polars"] + + +def _one_row_graph() -> Plottable: + nodes = pd.DataFrame({"id": ["a"]}) + edges = pd.DataFrame({"src": ["a"], "dst": ["a"]}) + return graphistry.nodes(nodes, "id").edges(edges, "src", "dst") + + +def _value(query: str, engine: str) -> object: + column = _one_row_graph().gfql(f"RETURN {query} AS x", engine=engine)._nodes["x"] + return (column.to_list() if hasattr(column, "to_list") else column.tolist())[0] + + +# Half a month is 30.436875 / 2 == 15.2184375 days == 15 days + 18873 seconds. +SPLIT_MONTH = [ + ("div_month_by_two", "duration('P1M') / 2", "P15DT5H14M33S"), + ("mul_month_by_half", "duration('P1M') * 0.5", "P15DT5H14M33S"), + ("half_times_month_commutes", "0.5 * duration('P1M')", "P15DT5H14M33S"), + ("div_month_by_three", "duration('P1M') / 3", "P10DT3H29M42S"), + # The day remainder carries on into a fractional SECOND, not just whole seconds. + ("div_month_by_four", "duration('P1M') / 4", "P7DT14H37M16.5S"), + # Whole months survive the split; only the leftover fraction becomes days. + ("div_three_months_by_two", "duration('P3M') / 2", "P1M15DT5H14M33S"), + ("div_year_and_month_by_two", "duration('P1Y1M') / 2", "P6M15DT5H14M33S"), + ("mul_month_by_one_and_a_half", "duration('P1M') * 1.5", "P1M15DT5H14M33S"), + # A day group already present is scaled first and the month spill lands on top. + ("div_month_and_days_by_two", "duration('P1M2D') / 2", "P16DT5H14M33S"), + ("div_month_by_six", "duration('P1M') / 6", "P5DT1H44M51S"), + # The month divides evenly, so only the DAY group splits and no month is spilled. + ("div_two_months_and_a_day_by_two", "duration('P2M1D') / 2", "P1MT12H"), + ("div_map_form_by_three", "duration({days: 14, minutes: 12, seconds: 70, nanoseconds: 1}) / 3", "P4DT16H4M23.333333333S"), +] + +# Every cascade step truncates toward zero; it never rounds to nearest. +TRUNCATION = [ + ("two_seconds_by_three", "duration('PT2S') / 3", "PT0.666666666S"), + ("eight_seconds_by_nine", "duration('PT8S') / 9", "PT0.888888888S"), + ("one_second_by_seven", "duration('PT1S') / 7", "PT0.142857142S"), + ("six_seconds_by_seven", "duration('PT6S') / 7", "PT0.857142857S"), + ("one_second_by_three", "duration('PT1S') / 3", "PT0.333333333S"), + ("negative_two_seconds_by_three", "duration('PT2S') / -3", "PT-0.666666666S"), + ("negative_one_second_by_three", "duration('PT1S') / -3", "PT-0.333333333S"), + # Observable at 1ns even where the exact answer is a round 38.4 seconds: the double + # residue lands just under 384000000ns and truncation keeps it there. + ("month_by_two_and_a_half", "duration('P1M') / 2.5", "P12DT4H11M38.399999999S"), +] + +# Truncation toward zero, so a negative scale mirrors its positive twin exactly. +SPLIT_MONTH_SIGNS = [ + ("negative_divisor", "duration('P1M') / -2", "P-15DT-5H-14M-33S"), + ("negative_duration", "duration('-P1M') / 2", "P-15DT-5H-14M-33S"), + ("negative_both", "duration('-P1M') / -2", "P15DT5H14M33S"), + ("negative_fractional_factor", "duration('P1M') * -0.5", "P-15DT-5H-14M-33S"), + ("negative_fractional_second", "duration('P1M') / -4", "P-7DT-14H-37M-16.5S"), + ("negative_whole_month_remainder", "duration('-P3M') / 2", "P-1M-15DT-5H-14M-33S"), + ("negative_divisor_whole_month_remainder", "duration('P3M') / -2", "P-1M-15DT-5H-14M-33S"), + ("negative_duration_with_days", "duration('-P1M2D') / 2", "P-16DT-5H-14M-33S"), + ("negative_divisor_with_days", "duration('P1M2D') / -2", "P-16DT-5H-14M-33S"), +] + +# The fence: exact in month-space stays in month-space, no average month anywhere. +EXACT_MONTH = [ + ("div_two_months_by_two", "duration('P2M') / 2", "P1M"), + ("div_month_by_one", "duration('P1M') / 1", "P1M"), + ("div_year_by_two", "duration('P1Y') / 2", "P6M"), + ("mul_month_by_two", "duration('P1M') * 2", "P2M"), + ("mul_month_by_two_point_zero", "duration('P1M') * 2.0", "P2M"), + ("div_month_by_one_negative", "duration('P1M') / -1", "P-1M"), + ("div_twelve_months_by_two", "duration('P12M') / 2", "P6M"), + ("div_seven_months_by_seven", "duration('P7M') / 7", "P1M"), + ("mul_month_by_zero", "duration('P1M') * 0", "PT0S"), + ("mul_month_by_minus_one", "duration('P1M') * -1", "P-1M"), + # ... and the non-month groups keep the fixed ratios they always had. + ("div_days_by_two", "duration('P3D') / 2", "P1DT12H"), + ("div_hour_by_two", "duration('PT1H') / 2", "PT30M"), + ("date_plus_month_clamps", "date('2026-01-31') + duration('P1M')", "2026-02-28"), +] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize( + "name,query,expected", SPLIT_MONTH, ids=[case[0] for case in SPLIT_MONTH] +) +def test_splitting_a_month_resolves_at_the_average_month(engine, name, query, expected): + assert _value(query, engine) == expected + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize( + "name,query,expected", SPLIT_MONTH_SIGNS, ids=[case[0] for case in SPLIT_MONTH_SIGNS] +) +def test_splitting_a_month_truncates_toward_zero(engine, name, query, expected): + assert _value(query, engine) == expected + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize( + "name,query,expected", TRUNCATION, ids=[case[0] for case in TRUNCATION] +) +def test_every_cascade_step_truncates_rather_than_rounds(engine, name, query, expected): + assert _value(query, engine) == expected + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize( + "name,query,expected", EXACT_MONTH, ids=[case[0] for case in EXACT_MONTH] +) +def test_a_month_that_divides_evenly_stays_exact(engine, name, query, expected): + assert _value(query, engine) == expected + + +@pytest.mark.parametrize("engine", ENGINES) +def test_split_month_agrees_with_the_fractional_month_literal(engine): + """The constructor already split fractional months at 30.436875 days; scaling must + land on the same value rather than introduce a second average month.""" + assert _value("duration('P1M') / 2", engine) == _value("duration('P0.5M')", engine) + assert _value("duration('P1M') * 1.5", engine) == _value("duration('P1.5M')", engine) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_splitting_a_month_does_not_round_trip(engine): + """ACCEPTED, not a defect: the average month is one-way. Halving then doubling a + month lands on days+time and must NOT be 'restored' to P1M.""" + assert _value("(duration('P1M') / 2) * 2", engine) == "P30DT10H29M6S" + + +def test_dividing_a_duration_by_zero_still_declines(): + """Splitting a month resolves; dividing by zero is still not a duration. Pinned on + pandas alone because the polars row path declines the unfolded node its own way.""" + with pytest.raises(GFQLTypeError): + _value("duration('P1M') / 0", "pandas") diff --git a/graphistry/tests/compute/gfql/test_endpoint_closure_matrix.py b/graphistry/tests/compute/gfql/test_endpoint_closure_matrix.py index c8cb29b6c5..ecc278f488 100644 --- a/graphistry/tests/compute/gfql/test_endpoint_closure_matrix.py +++ b/graphistry/tests/compute/gfql/test_endpoint_closure_matrix.py @@ -28,6 +28,7 @@ import graphistry from graphistry.compute.ast import n, e_forward, e_undirected +from graphistry.compute.exceptions import GFQLValidationError from .polars_test_utils import ( edge_pair_set, gpu_environment_reason, node_id_set, to_pandas_any, @@ -1025,24 +1026,18 @@ def test_undirected_zero_hop_seed_under_an_output_window_labels_before_it_strips assert node_id_set(out) == set() -# --- AXIS: a NULL endpoint id, and the NULL node row that backs it ----------------------------- -# -# Round 6. Membership is the gate's whole implementation, and the engines disagree about NULL: -# pandas/cuDF ``isin`` answers True for NULL-in-{..., NULL}; polars ``is_in`` answers NULL, and -# ``filter`` drops a NULL predicate. So the gate can silently over-filter on polars alone. -# -# NULLEP nodes: ids 0, 1, 2, and a fourth row whose id is NULL. -# NULLEP edges: (0,1), (1,2), (NULL,2). -# EVERY endpoint -- NULL included -- has a node row, so the graph is CLOSED end to end and the -# gate must remove nothing. Hand-walked undirected walk from seed 0 with hops=3: -# hop 1: 0 --(0,1)--> 1 reaches 1 -# hop 2: 1 --(1,2)--> 2 reaches 2 -# hop 3: 2 --(NULL,2)--> NULL reaches NULL -# so all three edges are traversed and the answer is the whole graph. - NULL_ENDPOINT_NODES = pd.DataFrame({"id": [0.0, 1.0, 2.0, None]}) -NULL_ENDPOINT_EDGES = pd.DataFrame({"s": [0.0, 1.0, None], "d": [1.0, 2.0, 2.0]}) -_NULL_ENDPOINT_CLOSED = {(0.0, 1.0), (1.0, 2.0), ("NULL", 2.0)} +NULL_ENDPOINT_EDGES = pd.DataFrame({"s": [0.0, 1.0, None, 2.0], "d": [1.0, 2.0, 2.0, None]}) +_NULL_ENDPOINT_MATCHED = {(0.0, 1.0), (1.0, 2.0)} +_NULL_ENDPOINT_REACHED = {0.0, 1.0, 2.0} + +NULL_FREE_TWIN_NODES = pd.DataFrame({"id": [0.0, 1.0, 2.0, 3.0]}) +NULL_FREE_TWIN_EDGES = pd.DataFrame({"s": [0.0, 1.0, 3.0, 2.0], "d": [1.0, 2.0, 2.0, 3.0]}) +_NULL_FREE_TWIN_MATCHED = {(0.0, 1.0), (1.0, 2.0), (3.0, 2.0), (2.0, 3.0)} + +NULL_STR_NODES = pd.DataFrame({"id": ["a", "b", "c", None]}) +NULL_STR_EDGES = pd.DataFrame({"s": ["a", "b", None, "c"], "d": ["b", "c", "c", None]}) +_NULL_STR_MATCHED = {("a", "b"), ("b", "c")} def _pairs_with_nulls_named(g): @@ -1052,135 +1047,203 @@ def _pairs_with_nulls_named(g): return set() def _v(x): - return "NULL" if pd.isna(x) else float(x) + return "NULL" if pd.isna(x) else x return {(_v(a), _v(b)) for a, b in zip(df[g._source].tolist(), df[g._destination].tolist())} +def _ids_with_nulls_named(g): + """``node_id_set`` with NULL spelled ``"NULL"`` -- NaN != NaN makes a raw set unusable.""" + df = to_pandas_any(g._nodes) + if df is None or len(df) == 0: + return set() + return {"NULL" if pd.isna(x) else x for x in df[g._node].tolist()} + + +def _scalar(g, col): + return to_pandas_any(g._nodes)[col].tolist() + + @pytest.mark.parametrize("engine", ALL_ENGINES) -def test_a_null_endpoint_backed_by_a_null_node_row_survives_the_gate(engine): - """The bound-table side. A NULL endpoint id resolves to the NULL node row, so the closed - graph comes back whole. REGRESSION PIN: red on the polars arm before the round-6 fix to - ``_keep_edges_with_both_endpoints_resolvable`` (it answered 2 edges where pandas, cuDF and - the merge-base 526976e91 all answer 3).""" +def test_a_null_edge_endpoint_matches_nothing_on_a_direct_hop(engine): _require_engine(engine) out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).hop( - nodes=_seed(engine, [0.0]), hops=3, direction="undirected", engine=engine) - assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_CLOSED + nodes=_seed(engine, [0.0]), hops=4, direction="undirected", engine=engine) + assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_MATCHED + assert _ids_with_nulls_named(out) == _NULL_ENDPOINT_REACHED + + +_SYNTH_NULL_ORACLE = [ + ("float_undirected", NULL_ENDPOINT_EDGES, 0.0, "undirected", _NULL_ENDPOINT_MATCHED), + ("float_forward", NULL_ENDPOINT_EDGES, 0.0, "forward", {(0.0, 1.0), (1.0, 2.0)}), + ("float_reverse_from_2", NULL_ENDPOINT_EDGES, 2.0, "reverse", _NULL_ENDPOINT_MATCHED), + ("float_forward_from_2", NULL_ENDPOINT_EDGES, 2.0, "forward", set()), + ("str_undirected", NULL_STR_EDGES, "a", "undirected", _NULL_STR_MATCHED), + ("str_reverse_from_c", NULL_STR_EDGES, "c", "reverse", _NULL_STR_MATCHED), +] @pytest.mark.parametrize("engine", ALL_ENGINES) -def test_a_null_endpoint_survives_the_vacuously_closed_synthesized_table(engine): - """The synthesized-table side of the same fixture: with no node table bound the id universe - is built FROM the endpoints, so it holds the NULL too and the gate must still remove nothing. - Same hand-walked answer as the bound case. - - CONTROL, not a pin (round 7 re-derivation): round 6 justified this cell against the - ``node_table_bound = True`` mutation, but its own null-aware fix made that mutation - equivalent on the polars hop -- forcing the gate on now leaves the whole gfql suite - byte-identical (93 failures either way). Kept as the vacuous-closure oracle it is.""" +@pytest.mark.parametrize("label,edges,seed,direction,want", _SYNTH_NULL_ORACLE) +def test_a_null_edge_endpoint_matches_nothing_on_the_synthesized_table( + engine, label, edges, seed, direction, want +): _require_engine(engine) - out = _edges_only(engine, NULL_ENDPOINT_EDGES).hop( - nodes=_seed(engine, [0.0]), hops=3, direction="undirected", engine=engine) - assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_CLOSED + out = _edges_only(engine, edges).hop( + nodes=_seed(engine, [seed]), hops=4, direction=direction, engine=engine) + assert _pairs_with_nulls_named(out) == want, label + assert "NULL" not in _ids_with_nulls_named(out), label -def test_the_synthesized_table_does_not_gate_a_null_endpoint_on_the_polars_chain(): - """The polars single-hop chain fast path reaches the same question through a semi-JOIN, and - a polars join never matches NULL to NULL -- so ``node_table_bound`` there is load-bearing, - not the no-op the round-5 audit called it: forcing the gate on drops (NULL,2). +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.parametrize("direction,want", [ + ("forward", set()), + ("reverse", {(1.0, 2.0)}), +]) +def test_each_null_endpoint_side_is_dropped_on_its_own(engine, direction, want): + _require_engine(engine) + out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).hop( + nodes=_seed(engine, [2.0]), hops=1, direction=direction, engine=engine) + assert _pairs_with_nulls_named(out) == want - Hand-walked ``(n)-[e]->(n)`` over NULLEP: an unconstrained forward pattern selects every - edge, so all three come back. (pandas and cuDF answer {(0,1),(1,2)} on this SURFACE -- a - pre-existing chain divergence, identical at the merge-base 526976e91 and unrelated to the - #1888 gate, so it is reported rather than pinned here.)""" - out = _edges_only("polars", NULL_ENDPOINT_EDGES).gfql([n(), e_forward(), n()], engine="polars") - assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_CLOSED +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_a_null_seed_id_reaches_nothing(engine): + _require_engine(engine) + out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).hop( + nodes=_seed(engine, [None]), hops=2, direction="undirected", engine=engine) + assert _pairs_with_nulls_named(out) == set() + assert _ids_with_nulls_named(out) == set() -# --- AXIS: the rest of the NULL surface, past the one site round 6 fixed ---------------------- -# -# Round 7. Membership is null-blind on polars in MORE than the hop gate: the hop's node-output -# epilogue and the chain's endpoint gates are semi-JOINs, and a polars join never matches NULL -# to NULL either. Each cell below is the SAME NULLEP fixture, hand-walked, on a surface the -# contract at the top of this file names. Strict xfails carry the measured wrong answer. -_NULL_NODE_ROW_POLARS_XFAIL = pytest.mark.xfail(strict=True, raises=AssertionError, reason=( - "polars hop keeps the (NULL,2) edge but its node-output semi-join " - "(all_nodes.join(needed, how='semi')) never matches NULL to NULL, so the kept edge's NULL " - "endpoint has NO node row -- the output is not endpoint-closed. Measured identical at the " - "merge base 526976e91, so pre-existing, not this PR; #1888's fix reached the edge arm only.")) +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_the_chain_answers_the_same_null_endpoint_question_as_hop(engine): + _require_engine(engine) + out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).gfql( + [n(), e_undirected(), n()], engine=engine) + assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_MATCHED + assert _ids_with_nulls_named(out) == _NULL_ENDPOINT_REACHED -_CHAIN_NULL_XFAIL = pytest.mark.xfail(strict=True, raises=AssertionError, reason=( - "the chain surface answers the NULL-endpoint question differently from hop(): on the SAME " - "bound closed graph hop() keeps all three edges and an undirected chain returns two. pandas " - "and cuDF do this at the merge base too; the polars arm XPASSes at 526976e91 (it answered 3 " - "before #1888 attached a null-blind semi-join gate to the chain fast path).")) -_CYPHER_COUNT_POLARS_XFAIL = pytest.mark.xfail(strict=True, raises=AssertionError, reason=( - "polars counts 4 where pandas/cuDF count 6: the two orientations of the NULL-endpoint edge " - "are lost in the chain's null-blind endpoint semi-joins. Pre-existing at 526976e91.")) +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_naming_the_ops_does_not_change_the_null_endpoint_answer(engine): + _require_engine(engine) + g = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES) + unnamed = g.gfql([n(), e_undirected(), n()], engine=engine) + named = g.gfql([n(name="a"), e_undirected(name="x"), n(name="b")], engine=engine) + assert _pairs_with_nulls_named(named) == _pairs_with_nulls_named(unnamed) + assert _pairs_with_nulls_named(named) == _NULL_ENDPOINT_MATCHED + + +_CHAIN_NULL_ORACLE = [ + ("bound_float", NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES, _NULL_ENDPOINT_MATCHED), + ("synth_float", None, NULL_ENDPOINT_EDGES, _NULL_ENDPOINT_MATCHED), + ("bound_str", NULL_STR_NODES, NULL_STR_EDGES, _NULL_STR_MATCHED), + ("synth_str", None, NULL_STR_EDGES, _NULL_STR_MATCHED), +] -_SYNTH_CHAIN_NULL_XFAIL = pytest.mark.xfail(strict=True, raises=AssertionError, reason=( - "pandas/cuDF gate a NULL endpoint out of a SYNTHESIZED (vacuously closed) node table, " - "answering 2 where polars answers the contract's 3. Pre-existing at 526976e91; round 6 " - "reported this in a docstring, this cell pins it.")) +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.parametrize("edge_op", [e_forward, e_undirected]) +@pytest.mark.parametrize("label,nodes,edges,want", _CHAIN_NULL_ORACLE) +def test_the_chain_gates_a_null_endpoint_bound_or_synthesized( + engine, edge_op, label, nodes, edges, want +): + _require_engine(engine) + g = (_edges_only(engine, edges) if nodes is None else _bind(engine, nodes, edges)) + out = g.gfql([n(), edge_op(), n()], engine=engine) + assert _pairs_with_nulls_named(out) == want, label + assert "NULL" not in _ids_with_nulls_named(out), label -def _engines(**per_engine_mark): - return [pytest.param(e, marks=per_engine_mark[e]) if e in per_engine_mark else e - for e in ALL_ENGINES] +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.parametrize("query,want", [ + ("MATCH (a)-[x]-(b) RETURN count(*) AS c", 4), + ("MATCH (a)-[x]->(b) RETURN count(*) AS c", 2), +]) +def test_cypher_count_counts_only_matchable_edges(engine, query, want): + _require_engine(engine) + out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).gfql(query, engine=engine) + assert _scalar(out, "c") == [want] + + +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.xfail( + strict=False, + raises=(AssertionError, GFQLValidationError), + reason="#1995 follow-up: NULL-id source-row validity is outside endpoint resolution", +) +def test_current_node_only_scan_preserves_a_null_id_source_row(engine): + _require_engine(engine) + out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).gfql( + "MATCH (a) RETURN count(*) AS c", engine=engine) + assert _scalar(out, "c") == [4] -def _ids_with_nulls_named(g): - """``node_id_set`` with NULL spelled ``"NULL"`` -- NaN != NaN makes a raw set unusable.""" - df = to_pandas_any(g._nodes) - if df is None or len(df) == 0: - return set() - return {"NULL" if pd.isna(x) else float(x) for x in df[g._node].tolist()} +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.parametrize("surface", ["hop", "chain"]) +def test_no_output_frame_references_a_node_it_does_not_carry(engine, surface): + _require_engine(engine) + g = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES) + out = (g.hop(nodes=_seed(engine, [0.0]), hops=4, direction="undirected", engine=engine) + if surface == "hop" else g.gfql([n(), e_undirected(), n()], engine=engine)) + ids = _ids_with_nulls_named(out) + endpoints = {i for pair in _pairs_with_nulls_named(out) for i in pair} + assert endpoints <= ids, f"edge endpoints with no node row: {endpoints - ids}" + assert "NULL" not in endpoints -@pytest.mark.parametrize("engine", _engines(polars=_NULL_NODE_ROW_POLARS_XFAIL)) -def test_the_kept_null_endpoint_edge_also_gets_its_node_row(engine): - """Round 6 pinned that the (NULL,2) edge survives; nothing pinned that its NULL endpoint - still has a node row. Same hand-walked undirected walk from seed 0 (0->1->2->NULL): the - whole closed graph comes back, so the node set is every id in the bound table.""" + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_the_null_free_twin_matches_every_edge(engine): _require_engine(engine) - out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).hop( - nodes=_seed(engine, [0.0]), hops=3, direction="undirected", engine=engine) - assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_CLOSED - assert _ids_with_nulls_named(out) == {0.0, 1.0, 2.0, "NULL"} + g = _bind(engine, NULL_FREE_TWIN_NODES, NULL_FREE_TWIN_EDGES) + out = g.hop(nodes=_seed(engine, [0.0]), hops=4, direction="undirected", engine=engine) + assert _pairs_with_nulls_named(out) == _NULL_FREE_TWIN_MATCHED + assert _ids_with_nulls_named(out) == {0.0, 1.0, 2.0, 3.0} + assert _pairs_with_nulls_named( + g.gfql([n(), e_undirected(), n()], engine=engine)) == _NULL_FREE_TWIN_MATCHED + assert _scalar(g.gfql("MATCH (a)-[x]-(b) RETURN count(*) AS c", engine=engine), "c") == [8] + assert _scalar(g.gfql("MATCH (a)-[x]->(b) RETURN count(*) AS c", engine=engine), "c") == [4] -@_CHAIN_NULL_XFAIL @pytest.mark.parametrize("engine", ALL_ENGINES) -def test_the_chain_answers_the_same_null_endpoint_question_as_hop(engine): - """One rule on every surface: NULLEP is closed end to end (the NULL endpoint has its own - node row), so an unconstrained undirected chain selects every edge -- the same three the - direct hop returns.""" +def test_the_null_endpoint_contract_holds_on_string_ids(engine): _require_engine(engine) - out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).gfql( - [n(), e_undirected(), n()], engine=engine) - assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_CLOSED + g = _bind(engine, NULL_STR_NODES, NULL_STR_EDGES) + out = g.hop(nodes=_seed(engine, ["a"]), hops=4, direction="undirected", engine=engine) + assert _pairs_with_nulls_named(out) == _NULL_STR_MATCHED + assert _ids_with_nulls_named(out) == {"a", "b", "c"} + assert _pairs_with_nulls_named( + g.gfql([n(), e_forward(), n()], engine=engine)) == _NULL_STR_MATCHED + assert _scalar(g.gfql("MATCH (a)-[x]-(b) RETURN count(*) AS c", engine=engine), "c") == [4] + + +_INDEXED_NULL_ORACLE = [ + (0.0, "forward", {(0.0, 1.0)}), + (2.0, "forward", set()), + (0.0, "reverse", set()), + (2.0, "reverse", {(1.0, 2.0)}), + (0.0, "undirected", {(0.0, 1.0)}), + (2.0, "undirected", {(1.0, 2.0)}), +] -@pytest.mark.parametrize("engine", _engines(polars=_CYPHER_COUNT_POLARS_XFAIL)) -def test_cypher_undirected_count_counts_the_null_endpoint_edge(engine): - """Cypher count(*) is one of the surfaces the matrix header names. An undirected pattern - matches each of the three closed edges in both orientations, so the hand count is 3*2 = 6. - (Control: the same query over the NULL-free 3-edge graph answers 6 on all three engines.)""" +@pytest.mark.parametrize("engine", ALL_ENGINES) +@pytest.mark.parametrize("seed,direction,want", _INDEXED_NULL_ORACLE) +def test_the_index_backed_route_answers_the_null_contract_too(engine, seed, direction, want): _require_engine(engine) - out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).gfql( - "MATCH (a)-[x]-(b) RETURN count(*) AS c", engine=engine) - assert to_pandas_any(out._nodes)["c"].tolist() == [6] + from graphistry.compute.gfql.index import gfql_index_edges + g = gfql_index_edges(_bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES)) + out = g.hop(nodes=_seed(engine, [seed]), hops=1, direction=direction, engine=engine) + assert _pairs_with_nulls_named(out) == want -@_SYNTH_CHAIN_NULL_XFAIL -@pytest.mark.parametrize("engine", ["pandas", "cudf"]) -def test_the_synthesized_chain_is_not_gated_for_a_null_endpoint(engine): - """The pandas/cuDF counterpart of the polars cell above -- same query, same hand-walked - answer. With no node table bound the id universe is built from the endpoints, so it holds - the NULL: vacuously closed, and an unconstrained forward pattern selects all three edges.""" + +@pytest.mark.parametrize("engine", ALL_ENGINES) +def test_get_degrees_counts_raw_edge_rows_not_matchable_edges(engine): _require_engine(engine) - out = _edges_only(engine, NULL_ENDPOINT_EDGES).gfql([n(), e_forward(), n()], engine=engine) - assert _pairs_with_nulls_named(out) == _NULL_ENDPOINT_CLOSED + out = _bind(engine, NULL_ENDPOINT_NODES, NULL_ENDPOINT_EDGES).get_degrees() + rows = to_pandas_any(out._nodes) + got = {("NULL" if pd.isna(r["id"]) else r["id"]): + (int(r["degree_in"]), int(r["degree_out"])) for _, r in rows.iterrows()} + assert got == {0.0: (0, 1), 1.0: (1, 1), 2.0: (2, 1), "NULL": (0, 0)} diff --git a/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py b/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py index 883429d615..6a8dc672ce 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py @@ -180,15 +180,15 @@ def test_polars_cartesian_binding_rows_raw_meaningful_cols(): def test_polars_cartesian_alias_name_collides_with_property(): - """A node property named the same as a MATCH alias is shadowed by the leaked - named-op flag (``alias.alias = True``) on BOTH engines — polars mirrors the - pandas quirk exactly rather than surfacing the real property value.""" + """A node property named the same as a MATCH alias used to be shadowed by the leaked + named-op flag (``alias.alias = True``) on BOTH engines; the cartesian builders now + unshadow it (#1911 defect-4), so the real property values surface identically.""" nodes = pd.DataFrame({"id": [0, 1, 2], "kind": ["a", "b", "a"], "a": [10, 20, 30], "b": [1, 2, 3]}) g = graphistry.nodes(nodes, "id").edges(pd.DataFrame({"s": [0], "d": [1]}), "s", "d") q = "MATCH (a {kind: 'a'}), (b {kind: 'b'}) RETURN a.id AS ai, a.a AS aa, b.id AS bi, b.b AS bb" rpd = g.gfql(q, engine="pandas")._nodes.reset_index(drop=True) rpl = g.gfql(q, engine="polars")._nodes.to_pandas().reset_index(drop=True) - assert list(rpd["aa"]) == [True, True] and list(rpd["bb"]) == [True, True] # flag, not 10/30 + assert sorted(rpd["aa"]) == [10, 30] and list(rpd["bb"]) == [2, 2] # user values, not the flag pd.testing.assert_frame_equal( rpd.sort_values(["ai", "bi"]).reset_index(drop=True), rpl[rpd.columns.tolist()].sort_values(["ai", "bi"]).reset_index(drop=True), diff --git a/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py b/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py index fb2274094d..b54ccce403 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py @@ -275,8 +275,8 @@ def test_conformance_cypher_expressions(label, query): "pandas astype(float) RAISES on non-numeric strings; strict=False nulls would fabricate data"), ("tointeger_string", "MATCH (n) RETURN n.id AS id, toInteger(n.name) AS i", "raises", "pandas astype(float) RAISES on non-numeric strings; strict=False nulls would fabricate data"), - ("size_numeric", "MATCH (n) RETURN n.id AS id, size(n.num) AS sz", "ok", - "pandas size(non-string/non-list) = ROW-COUNT quirk we refuse to replicate"), + ("size_numeric", "MATCH (n) RETURN n.id AS id, size(n.num) AS sz", "raises", + "size(non-string/non-list) has no defined answer: polars declines, pandas/cuDF raise"), ("substring_negative_start", "MATCH (n) RETURN n.id AS id, substring(n.name, -2) AS sub", "ok", "negative start diverges: pandas Python-slice vs polars offset/length — silent wrong slice"), ("tostring_float", "MATCH (n) RETURN n.id AS id, toString(n.f) AS s", "ok", diff --git a/graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py b/graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py index 6bdf7b2483..a9e9d9ea08 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py @@ -109,6 +109,11 @@ def _assert_parity(g, query): # NaN from a FUNCTION / division result (AST inference missed these; output-dtype # guard catches them — polars NaN-as-largest would otherwise leak) "RETURN abs(0.0 / 0.0) > 1 AS a, coalesce(0.0 / 0.0, 0.0) > 1 AS b", + # literal temporal comparison now constant-folds to the CIP2016-06-14 instant + # semantics on BOTH engines (was a polars NIE decline); parity + oracle-checked + "RETURN time({hour: 10, timezone: '+01:00'}) > time({hour: 9, timezone: '+00:00'}) AS x", + "RETURN date({year: 1984, month: 10, day: 12}) < date({year: 1985, month: 5, day: 6}) AS x", + "RETURN datetime('2020-01-02T05:00:00+05:00') = datetime('2020-01-02T00:00:00Z') AS x", "MATCH (n) RETURN n.val > 50 AS big, n.kind", "MATCH (n) RETURN n.val >= 50 AND n.val <= 80 AS mid", # Kleene 3-valued booleans over bare null literals — must not crash on Null dtype (polars @@ -180,10 +185,6 @@ def test_cypher_conformance_corpus(query): # a value/null), so the lowering must decline rather than crash "MATCH (n) RETURN n.val > 'a' AS x", "MATCH (n) WHERE n.val < 'z' RETURN n.id", - # ISO temporal comparison: cypher time()/date()/datetime() lower to ISO strings; - # polars would compare them lexicographically (wrong across timezones) -> NIE - "RETURN time({hour: 10, timezone: '+01:00'}) > time({hour: 9, timezone: '+00:00'}) AS x", - "RETURN date({year: 1984, month: 10, day: 12}) < date({year: 1985, month: 5, day: 6}) AS x", # temporal arithmetic: duration(...) lowers to an ISO string literal, so # a.time + duration(...) must NOT silently become string concatenation "MATCH (n) RETURN n.val + duration({minutes: 6}) AS t", diff --git a/graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py b/graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py index 1c5e94af3c..b7ddf74293 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py @@ -79,6 +79,7 @@ def _assert_parity(query, *, order_sensitive=True): # Row ops lowered to NATIVE polars (no pandas): select/with_/return_ projection (property/ # arith/comparison/boolean/literal), order_by, group_by (count/sum/avg/min/max), unwind. NATIVE_LOWERED = [ + "MATCH (n)-[e]->(m) RETURN n, m", "MATCH (n) RETURN n.val", "MATCH (n) RETURN n.val AS v, n.kind", "MATCH (n) RETURN n.val, n.name", @@ -125,7 +126,6 @@ def _assert_parity(query, *, order_sensitive=True): # lowered via rows(binding_ops) are native. DEFERRED = [ "MATCH (n)-[e]->(m) WHERE n.val < m.val RETURN n, m", # cross-entity WHERE - "MATCH (n)-[e]->(m) RETURN n, m", # whole-row multi-entity render # whole-entity collect: agg arg is the __node_entity__(n) whole-entity token (not the bare # identity sentinel), whose native list-of-entities representation isn't ported yet -> NIE. "MATCH (n) RETURN collect(n) AS xs", diff --git a/graphistry/tests/compute/gfql/test_gfql_unified_routing_contracts.py b/graphistry/tests/compute/gfql/test_gfql_unified_routing_contracts.py index 9ffd4c845e..77d862e1aa 100644 --- a/graphistry/tests/compute/gfql/test_gfql_unified_routing_contracts.py +++ b/graphistry/tests/compute/gfql/test_gfql_unified_routing_contracts.py @@ -2,12 +2,15 @@ Each test here replaces a comment the reviewer asked to be turned into a test: - * fast paths always run on the CPU execution target, whatever engine was requested (#1824) + * a fast path runs on the GPU execution target exactly when ``polars-gpu`` was requested, + and on CPU for every other engine (#1824) * a NotImplementedError from a fast path on the CPU target is a real error, not a decline * a policied AUTO query is routed to pandas by ``resolve_engine`` -- NOT by a frame-shape check, because a frame-shape check lets MIXED frames slip past a denying policy * the AUTO polars-native decline serves via pandas with COERCED frames """ +import importlib.util + import pandas as pd import pytest @@ -15,12 +18,16 @@ from graphistry.Engine import Engine, EngineAbstract from graphistry.compute import gfql_unified from graphistry.compute.gfql_unified import ( - _fast_path_execution_target_ignoring_requested_engine, + _fast_path_execution_target, _policied_auto_serves_via_pandas_until_the_polars_route_emits_hooks as _policied_auto_to_pandas, ) pl = pytest.importorskip("polars") +#: Without the RAPIDS stack a ``polars-gpu`` query still consults the fast paths (they run +#: before chain dispatch), then reports the missing install once it reaches the generic route. +HAS_CUDF_POLARS = importlib.util.find_spec("cudf_polars") is not None + NODES = pd.DataFrame({"id": [0, 1, 2], "v": [10, 20, 30]}) EDGES = pd.DataFrame({"s": [0, 1], "d": [1, 2]}) @@ -31,38 +38,112 @@ def _polars_graph(): .edges(pl.from_pandas(EDGES), "s", "d")) +#: A connected comma-pattern that routes through ``_apply_connected_match_join``'s two-star +#: arms and on into the FUSED lazy lane, whose single collect carries the engine label. +Q_TWO_STAR = ( + "MATCH (p {node_type:'Person'})-[{rel:'HAS_INTEREST'}]->(i {node_type:'Interest'}), " + "(p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "WHERE toLower(i.interest) = 'fine dining' AND p.age >= 20 AND p.age <= 40 " + "RETURN c.city AS city, count(p) AS n ORDER BY n DESC, city ASC") + + +def _two_star_graph(): + nodes = pl.DataFrame({ + "node_id": [1, 2, 3, 4, 5, 6, 7], + "node_type": ["Person", "Person", "Person", "Interest", "Interest", "City", "City"], + "age": [25, 30, 55, None, None, None, None], + "interest": [None, None, None, "Fine Dining", "tennis", None, None], + "city": [None, None, None, None, None, "London", "Paris"], + }) + edges = pl.DataFrame({ + "src": [1, 2, 3, 1, 2, 3], + "dst": [4, 4, 5, 6, 6, 7], + "rel": ["HAS_INTEREST"] * 3 + ["LIVES_IN"] * 3, + }) + return graphistry.nodes(nodes, "node_id").edges(edges, "src", "dst") + + +def _run_tolerating_absent_rapids(graph, query, engine): + try: + return graph.gfql(query, engine=engine) + except ImportError: + if engine == "polars-gpu" and not HAS_CUDF_POLARS: + return None + raise + + # --- fast-path execution target ----------------------------------------------------------- -@pytest.mark.parametrize("engine", [ - "pandas", "polars", "polars-gpu", "cudf", "auto", - Engine.POLARS_GPU, EngineAbstract.AUTO, -]) -def test_fast_paths_target_cpu_whatever_engine_was_requested(engine): - """The whole point of the name: the REQUESTED engine does not move the target. Flipping - this to GPU without making each arm GPU-or-decline is the #1824 regression.""" +Q_GROUPED = "MATCH (a)-[]->(b) RETURN b.v AS v, count(*) AS c" + +#: The two ``_apply_connected_match_join`` fast-path arms, in consultation order. +TWO_STAR_ARMS = [ + "_connected_join_two_star_fast_grouped_count", + "_connected_join_two_star_fast_rows", +] + + +def _isolate_two_star_arm(monkeypatch, arm): + """Make ``arm`` the arm under test: an earlier arm serves Q_TWO_STAR and would + short-circuit it, so decline the earlier arms.""" + for earlier in TWO_STAR_ARMS[:TWO_STAR_ARMS.index(arm)]: + monkeypatch.setattr(gfql_unified, earlier, lambda *a, **k: None) + + +@pytest.mark.parametrize("engine", ["pandas", "polars", "cudf", "auto", EngineAbstract.AUTO]) +def test_fast_path_target_is_cpu_for_every_engine_that_is_not_polars_gpu(engine): from graphistry.compute.gfql.lazy import ExecutionTarget - assert _fast_path_execution_target_ignoring_requested_engine(engine) is ExecutionTarget.CPU + assert _fast_path_execution_target(engine) is ExecutionTarget.CPU -def test_fast_path_body_actually_runs_under_the_cpu_target(monkeypatch): - """Not just the constant -- the fast-path call is really wrapped in that target_mode.""" - from graphistry.compute.gfql.lazy import active_target, ExecutionTarget +@pytest.mark.parametrize("engine", ["polars-gpu", Engine.POLARS_GPU]) +def test_fast_path_target_is_gpu_for_an_explicitly_requested_polars_gpu(engine): + """An explicit ``polars-gpu`` must reach the GPU collect target; serving it on CPU and + labelling the result GPU is the whole defect.""" + from graphistry.compute.gfql.lazy import ExecutionTarget + assert _fast_path_execution_target(engine) is ExecutionTarget.GPU - seen = [] +def _record_target(seen): def _record(*args, **kwargs): + from graphistry.compute.gfql.lazy import active_target seen.append(active_target()) - return None # decline, so the chain route answers + return None # decline, so the generic route answers + return _record + + +@pytest.mark.parametrize("engine,expected", [("polars", "CPU"), ("auto", "CPU"), ("polars-gpu", "GPU")]) +def test_grouped_aggregate_fast_path_body_runs_under_the_requested_engines_target( + monkeypatch, engine, expected): + """Not just the constant -- the fast-path call is really wrapped in that target_mode.""" + from graphistry.compute.gfql.lazy import ExecutionTarget + seen: list = [] monkeypatch.setattr( - gfql_unified, "_execute_single_hop_grouped_aggregate_fast_path", _record) - _polars_graph().gfql( - "MATCH (a)-[]->(b) RETURN b.v AS v, count(*) AS c", engine="polars") + gfql_unified, "_execute_single_hop_grouped_aggregate_fast_path", _record_target(seen)) + _run_tolerating_absent_rapids(_polars_graph(), Q_GROUPED, engine) assert seen, "the grouped-aggregate fast path was never consulted" - assert all(t == ExecutionTarget.CPU for t in seen), seen + assert all(t is getattr(ExecutionTarget, expected) for t in seen), seen -def test_cpu_fast_path_not_implemented_error_is_not_swallowed(monkeypatch): +@pytest.mark.parametrize("arm", TWO_STAR_ARMS) +@pytest.mark.parametrize("engine,expected", [("polars", "CPU"), ("auto", "CPU"), ("polars-gpu", "GPU")]) +def test_connected_join_two_star_fast_path_runs_under_the_requested_engines_target( + monkeypatch, arm, engine, expected): + """The connected-join arms are a SEPARATE call site from the same_path ``_try_fast`` arms; + wrapping only one of the two leaves half the OLAP surface collecting on the wrong target.""" + from graphistry.compute.gfql.lazy import ExecutionTarget + + seen: list = [] + _isolate_two_star_arm(monkeypatch, arm) + monkeypatch.setattr(gfql_unified, arm, _record_target(seen)) + _run_tolerating_absent_rapids(_two_star_graph(), Q_TWO_STAR, engine) + assert seen, f"{arm} was never consulted" + assert all(t is getattr(ExecutionTarget, expected) for t in seen), seen + + +@pytest.mark.parametrize("engine", ["polars", "auto"]) +def test_cpu_fast_path_not_implemented_error_is_not_swallowed(monkeypatch, engine): """On the CPU target an NIE is a real bug and must surface. Only the GPU target may treat it as 'plan not executable here, fall back'.""" def _boom(*args, **kwargs): @@ -71,8 +152,101 @@ def _boom(*args, **kwargs): monkeypatch.setattr( gfql_unified, "_execute_single_hop_grouped_aggregate_fast_path", _boom) with pytest.raises(NotImplementedError, match="fast path exploded"): - _polars_graph().gfql( - "MATCH (a)-[]->(b) RETURN b.v AS v, count(*) AS c", engine="polars") + _polars_graph().gfql(Q_GROUPED, engine=engine) + + +@pytest.mark.parametrize("arm", TWO_STAR_ARMS) +@pytest.mark.parametrize("engine", ["polars", "auto"]) +def test_cpu_connected_join_two_star_not_implemented_error_is_not_swallowed( + monkeypatch, arm, engine): + def _boom(*args, **kwargs): + raise NotImplementedError("two star exploded") + + _isolate_two_star_arm(monkeypatch, arm) + monkeypatch.setattr(gfql_unified, arm, _boom) + with pytest.raises(NotImplementedError, match="two star exploded"): + _two_star_graph().gfql(Q_TWO_STAR, engine=engine) + + +def _assert_declines_rather_than_raising(graph, query, marker, expected): + """A non-GPU-executable fast path is a DECLINE: the generic route -- itself GPU-or-raise -- + answers with the same values. Without RAPIDS installed the generic route reports the missing + install instead, which still proves the fast path's NIE did not escape as the answer.""" + try: + out = graph.gfql(query, engine="polars-gpu") + except NotImplementedError as ex: + if marker in str(ex): + pytest.fail(f"fast-path decline escaped to the caller as a raise: {ex}") + raise + except ImportError: + assert not HAS_CUDF_POLARS + return + assert out._nodes.to_dicts() == expected + + +def test_gpu_fast_path_not_implemented_error_declines_to_the_generic_route(monkeypatch): + marker = "plan is not GPU-executable" + + def _boom(*args, **kwargs): + raise NotImplementedError(marker) + + expected = _polars_graph().gfql(Q_GROUPED, engine="polars")._nodes.to_dicts() + monkeypatch.setattr( + gfql_unified, "_execute_single_hop_grouped_aggregate_fast_path", _boom) + _assert_declines_rather_than_raising(_polars_graph(), Q_GROUPED, marker, expected) + + +@pytest.mark.parametrize("arm", TWO_STAR_ARMS) +def test_gpu_connected_join_two_star_not_implemented_error_declines_to_the_generic_route( + monkeypatch, arm): + marker = "plan is not GPU-executable" + + def _boom(*args, **kwargs): + raise NotImplementedError(marker) + + expected = _two_star_graph().gfql(Q_TWO_STAR, engine="polars")._nodes.to_dicts() + _isolate_two_star_arm(monkeypatch, arm) + monkeypatch.setattr(gfql_unified, arm, _boom) + _assert_declines_rather_than_raising(_two_star_graph(), Q_TWO_STAR, marker, expected) + + +# --- the fused OLAP lane on a real GPU ------------------------------------------------------ + +def _fused_lane_collect_engines(query, engine): + """Engine object of every collect issued from inside the fused two-star lane.""" + import traceback + seen = [] + original = pl.LazyFrame.collect + + def _spy(self, *args, **kwargs): + if any(f.name == "_connected_join_two_star_fused_polars" + for f in traceback.extract_stack()): + seen.append(type(kwargs.get("engine")).__name__) + return original(self, *args, **kwargs) + + pl.LazyFrame.collect = _spy + try: + out = _two_star_graph().gfql(query, engine=engine) + finally: + pl.LazyFrame.collect = original + return seen, out._nodes.to_dicts() + + +@pytest.mark.skipif(not HAS_CUDF_POLARS, reason="needs the RAPIDS cudf_polars stack") +def test_fused_two_star_lane_collects_on_the_gpu_engine_when_polars_gpu_requested(): + """The fused lane's collect carries a GPUEngine, and its values equal the CPU lane's.""" + gpu_engines, gpu_rows = _fused_lane_collect_engines(Q_TWO_STAR, "polars-gpu") + cpu_engines, cpu_rows = _fused_lane_collect_engines(Q_TWO_STAR, "polars") + assert "GPUEngine" in gpu_engines, gpu_engines + assert "GPUEngine" not in cpu_engines, cpu_engines + assert gpu_rows == cpu_rows + + +@pytest.mark.parametrize("engine", ["polars", "auto"]) +def test_fused_two_star_lane_never_reaches_the_gpu_engine_off_polars_gpu(engine): + engines, _ = _fused_lane_collect_engines(Q_TWO_STAR, engine) + assert engines, "the fused two-star lane never collected" + assert "GPUEngine" not in engines, engines # --- the policied-AUTO -> pandas predicate -------------------------------------------------- diff --git a/graphistry/tests/compute/gfql/test_hop_boundary_matrix.py b/graphistry/tests/compute/gfql/test_hop_boundary_matrix.py index d61b912609..bae540745c 100644 --- a/graphistry/tests/compute/gfql/test_hop_boundary_matrix.py +++ b/graphistry/tests/compute/gfql/test_hop_boundary_matrix.py @@ -26,6 +26,11 @@ star 0-1, 0-2, 0-3 acyclic, hub 0 twocomp 0-1, 2-3 two components isolated 1-2 (node 0 has no edges) isolated node 0 + tailcycle 0-1-2-0, 0->2, 2->3->4 cycle plus a tail ending below max + +C. MIN-HOP PRUNE RETENTION (#1944): min_hops/max_hops are INCLUSIVE bounds and + the prune removes only branches that never REACH min_hops, so a branch that + reaches min_hops and terminates below max_hops is retained whole. """ import pandas as pd import pytest @@ -52,6 +57,10 @@ "star": ([0, 1, 2, 3], [(0, 1), (0, 2), (0, 3)]), "twocomp": ([0, 1, 2, 3], [(0, 1), (2, 3)]), "isolated": ([0, 1, 2], [(1, 2)]), + # #1944: a 3-cycle 0-1-2-0 with a 2-long tail 2->3->4 hanging off node 2, + # plus the chord 0->2. From seed 2 the tail terminates at hop 2 -- BELOW + # max_hops -- which is exactly the branch shape the min-hop prune dropped. + "tailcycle": ([0, 1, 2, 3, 4], [(0, 1), (1, 2), (2, 3), (3, 4), (0, 2), (2, 0)]), } @@ -433,3 +442,86 @@ def test_undirected_tfp_equals_saturated_bounded_filtered( return_as_wave_front=True, hops=9, engine=engine, **filt) assert node_ids(g.hop(to_fixed_point=True, **kw)) == \ node_ids(g.hop(to_fixed_point=False, **kw)) == expected + + +# ======================================================================== C +# MIN-HOP PRUNE RETENTION (#1944). +# +# Contract: min_hops/max_hops are INCLUSIVE traversal bounds, and the prune +# only removes "dead-end branches that do not reach min_hops". So a branch that +# reaches min_hops and then STOPS -- below max_hops -- qualifies and must be +# retained whole. +# +# Hand derivation on ``tailcycle`` (0->1, 1->2, 2->3, 3->4, 0->2, 2->0) seeded +# at {2}, forward, wavefront. First-traversal hop of each edge from seed 2: +# +# hop 1: 2->3, 2->0 +# hop 2: 3->4, 0->1, 0->2 +# hop 3: 1->2 (2->3 / 2->0 are revisits, keep hop 1) +# +# min_hops=2, max_hops=3: every walk of length >= 2 qualifies. The tail walk +# 2->3->4 ENDS at hop 2, inside [2, 3], so 3 and 4 and both tail edges are +# retained; the cycle walks contribute the rest. -> all 5 nodes, all 6 edges. +# Pre-#1944 the backward walk seeded targets only from the TOP level, so the +# tail's terminating edge 3->4 was never a target when level 2 was processed: +# pandas/cuDF dropped (2,3) and (3,4) while leaking node 4 (an incoherent +# frame), and the polars chain mirror dropped node 4 as well. +# +# min_hops=3, max_hops=3 (anti-vacuity control, correct before AND after): +# only 1->2 is traversed at hop 3, so the goal set is {2}; level 2 must feed +# it (0->2 survives, 3->4 and 0->1 do not), level 1 must feed {0} (2->0 +# survives, 2->3 does not). The tail is a genuine sub-min dead end and stays +# pruned. -> nodes {0, 1, 2}, edges {(0,1), (0,2), (1,2), (2,0)} minus the +# never-retained ones, i.e. exactly (0,1), (1,2), (2,0) via the retained tree. +# +# reverse, min_hops=2 (control): the tail hangs the wrong way, so it never +# enters the answer in either direction of the fix. + +def edge_pairs(g): + edf = g._edges + edf = edf.to_pandas() if hasattr(edf, "to_pandas") else edf + return sorted(map(tuple, edf[["s", "d"]].to_numpy().tolist())) + + +HOP_ENGINES = ["pandas", pytest.param("cudf", marks=cudf_only)] + +MIN_HOP_RETENTION_ORACLE = [ + # min_hops, expected nodes, expected edges + (2, [0, 1, 2, 3, 4], [(0, 1), (0, 2), (1, 2), (2, 0), (2, 3), (3, 4)]), + (3, [0, 1, 2], [(0, 1), (1, 2), (2, 0)]), +] + + +@pytest.mark.parametrize("engine", HOP_ENGINES) +@pytest.mark.parametrize("min_hops,exp_nodes,exp_edges", MIN_HOP_RETENTION_ORACLE, + ids=["min2-branch-ends-below-max", "min3-submin-tail-pruned"]) +def test_min_hops_prune_retains_qualifying_short_branch_hop( + engine, min_hops, exp_nodes, exp_edges): + g = _graph("tailcycle", engine) + r = g.hop(nodes=_frame(engine, pd.DataFrame({"id": [2]})), + min_hops=min_hops, max_hops=3, direction="forward", + return_as_wave_front=True, engine=engine) + assert node_ids(r) == exp_nodes + assert edge_pairs(r) == exp_edges + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("min_hops,exp_nodes,exp_edges", MIN_HOP_RETENTION_ORACLE, + ids=["min2-branch-ends-below-max", "min3-submin-tail-pruned"]) +def test_min_hops_prune_retains_qualifying_short_branch_chain( + engine, min_hops, exp_nodes, exp_edges): + from graphistry.compute.ast import n, e_forward + g = _graph("tailcycle", engine) + r = g.chain([n({"id": 2}), e_forward(min_hops=min_hops, max_hops=3)], engine=engine) + assert node_ids(r) == exp_nodes + assert edge_pairs(r) == exp_edges + + +@pytest.mark.parametrize("engine", ENGINES) +def test_min_hops_prune_reverse_unchanged_control(engine): + # The tail points away from the reverse walk, so this cell must NOT move. + from graphistry.compute.ast import n, e_reverse + g = _graph("tailcycle", engine) + r = g.chain([n({"id": 2}), e_reverse(min_hops=2, max_hops=3)], engine=engine) + assert node_ids(r) == [0, 1, 2] + assert edge_pairs(r) == [(0, 1), (0, 2), (1, 2), (2, 0)] diff --git a/graphistry/tests/compute/gfql/test_hop_kernel_contracts.py b/graphistry/tests/compute/gfql/test_hop_kernel_contracts.py index 5f5502ac27..aab7950c58 100644 --- a/graphistry/tests/compute/gfql/test_hop_kernel_contracts.py +++ b/graphistry/tests/compute/gfql/test_hop_kernel_contracts.py @@ -231,13 +231,10 @@ def test_duplicate_node_rows_are_deduped_when_an_endpoint_is_backfilled(engine): f"duplicate node rows survived the backfill: {nodes_pdf['id'].tolist()}") -@pytest.mark.xfail(strict=True, reason=( - "PRE-EXISTING cross-engine divergence (reproduces at 86013f4, before the #1895 " - "remediation): the pandas hop de-dups its output node table by id, the polars hop " - "does not, so duplicate node rows survive on polars. Not introduced here; pinned " - "executable so the fix flips an xfail.")) @pytest.mark.parametrize("engine", ["polars"]) def test_duplicate_node_rows_are_deduped_on_polars_too(engine): + """The polars node output is a semi-join, which emits every matching input row; + without the pandas-matching epilogue a duplicated input id survives as two rows.""" out = _dup_node_graph(engine).hop(engine=engine) nodes_pdf = to_pandas_any(out._nodes) assert nodes_pdf["id"].tolist() == sorted(set(nodes_pdf["id"].tolist())) diff --git a/graphistry/tests/compute/gfql/test_hop_semantics_pins.py b/graphistry/tests/compute/gfql/test_hop_semantics_pins.py index 5cf2b24f3e..411b3f78eb 100644 --- a/graphistry/tests/compute/gfql/test_hop_semantics_pins.py +++ b/graphistry/tests/compute/gfql/test_hop_semantics_pins.py @@ -29,6 +29,7 @@ import graphistry from graphistry.compute.ast import n, e_forward, e_reverse, e_undirected +from graphistry.tests.compute.gfql.polars_test_utils import typed_frame_sig from graphistry.compute.exceptions import GFQLValidationError from graphistry.compute.predicates.is_in import IsIn @@ -273,14 +274,11 @@ def test_cypher_whole_entity_return_pandas_answers(): @polars_only -def test_cypher_whole_entity_return_polars_current_nie(): - # AUDIT NOTE (F-04): this graph is all-int64/str, yet the decline message - # blames "float/temporal/nested/label/multi-entity columns" -- the gate - # fires on data its message does not describe. When the gate is fixed or - # narrowed, flip this pin to a row-level parity assertion vs pandas. - g = _graph("polars") - with pytest.raises(NotImplementedError, match="cypher result projection"): - g.gfql("MATCH (a)-[e]->(b) RETURN a, b", engine="polars") +def test_cypher_whole_entity_return_polars_parity_with_pandas(): + query = "MATCH (a)-[e]->(b) RETURN a, b" + got = _pd(_graph("polars").gfql(query, engine="polars")._nodes) + want = _pd(_graph("pandas").gfql(query, engine="pandas")._nodes) + assert typed_frame_sig(got) == typed_frame_sig(want) # ================================================================ T-07 greens diff --git a/graphistry/tests/compute/gfql/test_optional_match_with_pipeline_boundaries.py b/graphistry/tests/compute/gfql/test_optional_match_with_pipeline_boundaries.py index 22aa585810..aea6e4c80e 100644 --- a/graphistry/tests/compute/gfql/test_optional_match_with_pipeline_boundaries.py +++ b/graphistry/tests/compute/gfql/test_optional_match_with_pipeline_boundaries.py @@ -493,14 +493,6 @@ def test_expression_over_a_carried_alias_declines_as_irreproducible(engine): @pytest.mark.parametrize("engine", ["pandas", "cudf"]) -@pytest.mark.xfail( - strict=True, - reason="KNOWN WRONG (pandas and cuDF agree it is wrong, and disagree on " - "how): the null-extended row for an unmatched carried row leaves " - "the whole-entity carried alias NULL instead of the node it is " - "still bound to. cuDF additionally renders the NULL boolean columns " - "as False. Predates #1897 -- byte-identical at base 21167e08.", -) def test_whole_entity_carried_alias_keeps_its_values_on_the_null_extended_row(engine): """`WITH a AS p, a.id AS pid` binds p for every carried row. OPTIONAL MATCH cannot unbind it, so a2's null-extended row must still carry a2's own node diff --git a/graphistry/tests/compute/gfql/test_path_trail_semantics.py b/graphistry/tests/compute/gfql/test_path_trail_semantics.py index 6bf74b4b87..a02e4e5183 100644 --- a/graphistry/tests/compute/gfql/test_path_trail_semantics.py +++ b/graphistry/tests/compute/gfql/test_path_trail_semantics.py @@ -337,10 +337,9 @@ def test_grouped_agg_lane_consistent(engine): @pytest.mark.parametrize("engine", ENGINES) -@pytest.mark.xfail(strict=True, reason="#1903 addendum A-2 residual: the seeded typed-hop " - "lane (fast path AND its fallback) projects the destination NODE SET -- " - "parallel edges from a unique seed collapse ([1,2] vs bag [1,1,2])") -def test_seeded_parallel_edge_multiplicity_residual(engine): +def test_seeded_parallel_edge_multiplicity(engine): + """The seeded typed-hop lane keeps trail multiplicity: two parallel 0->1 edges + are two rows, so the destination bag is [1, 1, 2] and not the node set [1, 2].""" nodes = pd.DataFrame({"id": [0, 1, 2], "kind": ["a", "b", "b"]}) edges = pd.DataFrame({"s": [0, 0, 0], "d": [1, 1, 2], "type": ["KNOWS"] * 3}) if engine == "polars": diff --git a/graphistry/tests/compute/gfql/test_polars_lane_completeness.py b/graphistry/tests/compute/gfql/test_polars_lane_completeness.py index e613bbbd9f..8d6b02cc19 100644 --- a/graphistry/tests/compute/gfql/test_polars_lane_completeness.py +++ b/graphistry/tests/compute/gfql/test_polars_lane_completeness.py @@ -64,6 +64,11 @@ "test_const_fold_engine_parity.py, which IS in the lane -- and is what caught the " "engine-blind key in the first place" ), + "graphistry/tests/compute/test_remote_engine_contract.py": ( + "remote preflight contract only: 'polars' and 'polars-gpu' are plain engine strings " + "that must be rejected before upload or POST; the module imports no polars runtime " + "and builds no polars frame, so every test runs in the ordinary core lanes" + ), "graphistry/tests/compute/gfql/index/test_index_gpu_edge_match.py": ( "cudf/GPU-gated (module-level importorskip('cudf') + skipif no GPU), not polars-gated; " "belongs to the separate GPU-lane gap, and the polars CPU lane could not run it" diff --git a/graphistry/tests/compute/gfql/test_reentry_carry_seed_restriction.py b/graphistry/tests/compute/gfql/test_reentry_carry_seed_restriction.py new file mode 100644 index 0000000000..b7a87c3c54 --- /dev/null +++ b/graphistry/tests/compute/gfql/test_reentry_carry_seed_restriction.py @@ -0,0 +1,215 @@ +"""#1712 residual: WITH->MATCH re-entry seeds must restrict EVERY execution route. + +The bare-carry single-pattern shape was fixed earlier; these pin the routes that still +re-matched the trailing MATCH from the WHOLE graph, silently widening the carried set: + +1. projection carry (``WITH p, p.x AS t``) + grouped aggregate: the single-hop + grouped-aggregate fast path derived its seed from filter_dicts alone, leaking the + un-carried rows as an extra NULL-keyed group (2 extra persons here); +2. comma-pattern trailing MATCH: the connected match-join re-ran each arm globally + (bare carry AND projection carry both leaked); +3. two-hop trailing MATCH + count: the two-hop count fast path, same seed-blindness. + +Hand-computed oracle throughout: persons 0,1,2; only person 0 has the Books interest, +so every carried re-entry answers 1 (the unrestricted answer is 3 — or a spurious +second group of 2 — so a vacuous pass is impossible). + +polars runs the shapes it supports natively and typed-declines projection carry +(scalar WITH columns into trailing MATCH); those declines are pinned as declines. +""" +from typing import Any, List + +import pandas as pd +import pytest + +import graphistry +from graphistry.compute.exceptions import GFQLValidationError + +NODES = pd.DataFrame({ + "node_id": [0, 1, 2, 10, 20, 30, 40], + "node_type": ["Person", "Person", "Person", "City", "Interest", "Pet", "Country"], + # nickname is NULL for the carried person 0: the NULL must survive the carry + # (the NULL-vs-membership class), never a sentinel like '0000-00-00'. + "nickname": [None, "Bee", "Cee", None, None, None, None], + "interest": [None, None, None, None, "Books", None, None], +}) +EDGES = pd.DataFrame({ + "src": [0, 1, 2, 0, 0, 1, 10], + "dst": [10, 10, 10, 20, 30, 30, 40], + "rel": ["LIVES_IN", "LIVES_IN", "LIVES_IN", "HAS_INTEREST", "OWNS", "OWNS", "IN_COUNTRY"], +}) + +CARRY_PREFIX = ( + "MATCH (p {node_type:'Person'})-[{rel:'HAS_INTEREST'}]->(i {node_type:'Interest'})\n" + "WHERE i.interest='Books'\n" +) + +GROUPED_COUNT_PROJ = CARRY_PREFIX + ( + "WITH p, p.node_type AS t\n" + "MATCH (p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'})\n" + "RETURN t, count(p) AS numPersons" +) +GROUPED_COUNT_NULL_CARRY = CARRY_PREFIX + ( + "WITH p, p.nickname AS t\n" + "MATCH (p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'})\n" + "RETURN t, count(p) AS numPersons" +) +ROWS_PROJ = CARRY_PREFIX + ( + "WITH p, p.node_type AS t\n" + "MATCH (p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'})\n" + "RETURN t, p.node_id AS pid" +) +CONNECTED_JOIN_BARE = CARRY_PREFIX + ( + "WITH p\n" + "MATCH (p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'}), (p)-[{rel:'OWNS'}]->(d {node_type:'Pet'})\n" + "RETURN p.node_id AS pid, count(d) AS pets" +) +CONNECTED_JOIN_PROJ = CARRY_PREFIX + ( + "WITH p, p.node_type AS t\n" + "MATCH (p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'}), (p)-[{rel:'OWNS'}]->(d {node_type:'Pet'})\n" + "RETURN t, count(d) AS pets" +) +TWO_HOP_COUNT = CARRY_PREFIX + ( + "WITH p\n" + "MATCH (p)-[{rel:'LIVES_IN'}]->(c)-[{rel:'IN_COUNTRY'}]->(x)\n" + "RETURN count(*) AS n" +) + +POLARS_SCALAR_CARRY_DECLINE = "carries scalar WITH columns into the trailing MATCH" + + +def _graph(engine: str) -> Any: + if engine == "polars": + pl = pytest.importorskip("polars") + return graphistry.nodes(pl.from_pandas(NODES), "node_id").edges( + pl.from_pandas(EDGES), "src", "dst" + ) + if engine == "cudf": + cudf = pytest.importorskip("cudf") + return graphistry.nodes(cudf.from_pandas(NODES), "node_id").edges( + cudf.from_pandas(EDGES), "src", "dst" + ) + return graphistry.nodes(NODES, "node_id").edges(EDGES, "src", "dst") + + +def _rows(engine: str, query: str) -> List[dict]: + frame = _graph(engine).gfql(query, engine=engine)._nodes + if hasattr(frame, "collect"): + frame = frame.collect() + if hasattr(frame, "to_pandas"): + frame = frame.to_pandas() + return frame.to_dict("records") + + +# ------------------------------ route 1: single-hop grouped-aggregate fast path + +@pytest.mark.parametrize("engine", ["pandas", "cudf"]) +def test_projection_carry_grouped_count_restricts_to_carried_rows(engine: str) -> None: + """Was ``[{'t': 'Person', 'numPersons': 1}, {'t': NaN, 'numPersons': 2}]`` — the + fast path re-matched all 3 persons and parked the 2 un-carried ones in a NULL + group. Exactly one group may remain.""" + assert _rows(engine, GROUPED_COUNT_PROJ) == [{"t": "Person", "numPersons": 1}] + + +def test_projection_carry_grouped_count_polars_declines_typed() -> None: + pytest.importorskip("polars") + with pytest.raises(NotImplementedError) as exc_info: + _rows("polars", GROUPED_COUNT_PROJ) + assert POLARS_SCALAR_CARRY_DECLINE in str(exc_info.value) + + +@pytest.mark.parametrize("engine", ["pandas", "cudf"]) +def test_projection_carry_null_scalar_survives_the_carry(engine: str) -> None: + """NULL cell in the carried column: person 0's nickname is NULL, and the carried + group key must stay NULL (this rendered as the sentinel string '0000-00-00' + via a vacuously-true all-null temporal-constructor probe).""" + rows = _rows(engine, GROUPED_COUNT_NULL_CARRY) + assert len(rows) == 1 and rows[0]["numPersons"] == 1 + assert pd.isna(rows[0]["t"]) + + +@pytest.mark.parametrize("engine", ["pandas", "cudf"]) +def test_projection_carry_row_form_control(engine: str) -> None: + """CONTROL (discriminator): the non-aggregate row form of the same query was + already restricted — the leak was fast-path-specific.""" + assert _rows(engine, ROWS_PROJ) == [{"t": "Person", "pid": 0}] + + +# ------------------------------------- route 2: connected comma-pattern join + +@pytest.mark.parametrize("engine", ["pandas", "polars", "cudf"]) +def test_bare_carry_connected_join_restricts_to_carried_rows(engine: str) -> None: + """Was ``[{'pid': 0, ...}, {'pid': 1, ...}]`` on all three engines: person 1 owns a + pet and lives in the city but was never carried.""" + assert _rows(engine, CONNECTED_JOIN_BARE) == [{"pid": 0, "pets": 1}] + + +@pytest.mark.parametrize("engine", ["pandas", "cudf"]) +def test_projection_carry_connected_join_restricts_to_carried_rows(engine: str) -> None: + assert _rows(engine, CONNECTED_JOIN_PROJ) == [{"t": "Person", "pets": 1}] + + +def test_projection_carry_connected_join_polars_declines_typed() -> None: + pytest.importorskip("polars") + with pytest.raises(NotImplementedError) as exc_info: + _rows("polars", CONNECTED_JOIN_PROJ) + assert POLARS_SCALAR_CARRY_DECLINE in str(exc_info.value) + + +# --------------------------------------------- route 3: two-hop count fast path + +@pytest.mark.parametrize("engine", ["pandas", "polars", "cudf"]) +def test_two_hop_count_after_carry_restricts_to_carried_rows(engine: str) -> None: + """Was ``n=3`` (all persons re-matched). Only carried person 0's path counts.""" + assert _rows(engine, TWO_HOP_COUNT) == [{"n": 1}] + + +# ----------------------------------------------------- helper-level unit pins + +def test_restrict_helper_filters_by_bare_alias_column() -> None: + from graphistry.compute.gfql.cypher.reentry.execution import ( + restrict_connected_join_rows_to_reentry_seed, + ) + + joined = pd.DataFrame({"p": [0, 1, 2], "d": [30, 30, 31], "x": [None, "v", None]}) + seeds = pd.DataFrame({"node_id": [0, 2]}) + out = restrict_connected_join_rows_to_reentry_seed( + joined, start_nodes=seeds, reentry_alias="p", node_col="node_id" + ) + # NULL cells in non-key columns ride along untouched + assert list(out["p"]) == [0, 2] and pd.isna(out["x"]).tolist() == [True, True] + + +def test_restrict_helper_declines_without_alias_or_seed_columns() -> None: + from graphistry.compute.gfql.cypher.reentry.execution import ( + restrict_connected_join_rows_to_reentry_seed, + ) + + joined = pd.DataFrame({"q": [0, 1]}) + seeds = pd.DataFrame({"node_id": [0]}) + with pytest.raises(GFQLValidationError): + restrict_connected_join_rows_to_reentry_seed( + joined, start_nodes=seeds, reentry_alias=None, node_col="node_id" + ) + with pytest.raises(GFQLValidationError): + restrict_connected_join_rows_to_reentry_seed( + joined, start_nodes=seeds, reentry_alias="p", node_col="node_id" + ) + with pytest.raises(GFQLValidationError): + restrict_connected_join_rows_to_reentry_seed( + joined, start_nodes=pd.DataFrame({"other": [0]}), reentry_alias="q", node_col="node_id" + ) + + +def test_restrict_helper_polars_frames() -> None: + pl = pytest.importorskip("polars") + from graphistry.compute.gfql.cypher.reentry.execution import ( + restrict_connected_join_rows_to_reentry_seed, + ) + + joined = pl.DataFrame({"p": [0, 1, 2]}) + seeds = pl.DataFrame({"node_id": [2]}) + out = restrict_connected_join_rows_to_reentry_seed( + joined, start_nodes=seeds, reentry_alias="p", node_col="node_id" + ) + assert out["p"].to_list() == [2] diff --git a/graphistry/tests/compute/gfql/test_rollout.py b/graphistry/tests/compute/gfql/test_rollout.py index d671261c08..6039902948 100644 --- a/graphistry/tests/compute/gfql/test_rollout.py +++ b/graphistry/tests/compute/gfql/test_rollout.py @@ -95,3 +95,55 @@ def test_reexports_from_compute_gfql(monkeypatch: pytest.MonkeyPatch) -> None: assert pkg_default() is False monkeypatch.setenv(STRICT_SCHEMA_ENV, "true") assert pkg_default() is True + + +def test_strict_schema_env_does_not_change_absent_label_behavior(monkeypatch) -> None: + """The documented env var is inert: all three states give the identical answer.""" + import pandas as pd + import graphistry + from graphistry.compute.exceptions import GFQLSchemaError + + g = (graphistry + .nodes(pd.DataFrame({"id": [0, 1]}), "id") + .edges(pd.DataFrame({"s": [0], "d": [1]}), "s", "d")) + + codes = [] + for value in (None, "0", "1"): + if value is None: + monkeypatch.delenv("GRAPHISTRY_GFQL_STRICT_SCHEMA", raising=False) + else: + monkeypatch.setenv("GRAPHISTRY_GFQL_STRICT_SCHEMA", value) + try: + g.gfql("MATCH (n:Nope) RETURN n.id", engine="pandas") + codes.append("answered") + except GFQLSchemaError as err: + codes.append(err.code) + + assert len(set(codes)) == 1, f"env var changed behavior: {codes}" + assert codes[0] == "answered" # the resolved strictness level decides this, not the env var + + +def test_strict_schema_env_does_not_change_absent_label_behavior_under_strict(monkeypatch) -> None: + """Still inert at the level that does raise, so the env var is not a back door.""" + import pandas as pd + import graphistry + from graphistry.compute.exceptions import GFQLSchemaError + + g = (graphistry + .nodes(pd.DataFrame({"id": [0, 1]}), "id") + .edges(pd.DataFrame({"s": [0], "d": [1]}), "s", "d")) + + codes = [] + for value in (None, "0", "1"): + if value is None: + monkeypatch.delenv("GRAPHISTRY_GFQL_STRICT_SCHEMA", raising=False) + else: + monkeypatch.setenv("GRAPHISTRY_GFQL_STRICT_SCHEMA", value) + try: + g.gfql("MATCH (n:Nope) RETURN n.id", engine="pandas", strict="strict") + codes.append("answered") + except GFQLSchemaError as err: + codes.append(err.code) + + assert len(set(codes)) == 1, f"env var changed behavior: {codes}" + assert codes[0] == "column-not-found" diff --git a/graphistry/tests/compute/gfql/test_row_multiplicity_semantics.py b/graphistry/tests/compute/gfql/test_row_multiplicity_semantics.py index 27c01b3ac5..481a3620c8 100644 --- a/graphistry/tests/compute/gfql/test_row_multiplicity_semantics.py +++ b/graphistry/tests/compute/gfql/test_row_multiplicity_semantics.py @@ -122,13 +122,10 @@ def test_pair_projection_control_unchanged(engine): @pytest.mark.parametrize("engine", ENGINES) -@pytest.mark.xfail(strict=True, reason="#1899 residual: whole-row endpoint projection still " - "collapses multiplicity (b bound twice to node 3 must yield two rows)") -def test_whole_row_endpoint_projection_multiplicity_residual(engine): - """Residual pin: `RETURN b` (whole entity) over the same match should be a - 4-row bag (node 3 twice). Flip when the whole-row lane joins binding rows.""" +def test_whole_row_endpoint_projection_multiplicity(engine): df = _run("MATCH (a)-->(b) RETURN b", engine) assert len(df) == 4 + assert _bag(df, "b.id") == [2, 3, 3, 4] # =========================================================================== @@ -352,10 +349,9 @@ def test_polars_nonintegral_float_endpoint_declines_typed(): @pytest.mark.parametrize("engine", ENGINES) -@pytest.mark.xfail(strict=True, reason="#1899 residual: leading OPTIONAL MATCH single-endpoint " - "projection still collapses bag multiplicity (guarded off binding rows to " - "protect null extension); expected [1, 1, 2, 3]") -def test_leading_optional_match_multiplicity_residual(engine): +def test_leading_optional_match_keeps_multiplicity(engine): + """A leading OPTIONAL MATCH binds nothing before it, so nothing can go + unmatched and it is a plain MATCH for row purposes: same [1, 1, 2, 3] bag.""" assert _bag(_run("OPTIONAL MATCH (a)-->(b) RETURN a.id AS x", engine), "x") == [1, 1, 2, 3] @@ -368,9 +364,9 @@ def test_optional_match_no_match_null_extension_preserved(engine): @pytest.mark.parametrize("engine", ENGINES) -@pytest.mark.xfail(strict=True, reason="#1899 residual: the seeded typed-hop fast path dedupes " - "parallel edges; Ann has two edges to Bob so b.id must be [2, 2]") -def test_seeded_parallel_edge_multiplicity_residual(engine): +def test_seeded_parallel_edge_multiplicity(engine): + """A selective seed does not change bag semantics: Ann has two edges to Bob, + so the seeded hop is [2, 2], the same bag the unseeded control below keeps.""" edges = pd.DataFrame({"s": [1, 1, 2, 3], "d": [2, 2, 3, 4]}) assert _bag(_run("MATCH (a {name: 'Ann'})-->(b) RETURN b.id AS x", engine, edges=edges), "x") == [2, 2] diff --git a/graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py b/graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py index abccc0491f..73d063430a 100644 --- a/graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py +++ b/graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py @@ -13,14 +13,11 @@ including full-path side-channels (policy hooks, same-path WHERE, OPTIONAL null rows, WITH..MATCH carried seeds, list-`labels` columns, null ids). """ -from typing import Dict, Tuple - import numpy as np import pandas as pd import pytest import graphistry -from graphistry.Plottable import Plottable from graphistry.compute.ast import n, e_forward, e_reverse import graphistry.compute.chain as chain_mod import graphistry.compute.gfql_unified as gfql_unified @@ -266,6 +263,48 @@ def spy(*a, **k): g.gfql(f"MATCH (m:Message {{id: {seed}}})-[:HAS_CREATOR]->(p:Person) RETURN p", engine="pandas") assert hits["n"] >= 1 + def test_fast_path_engages_on_the_bag_lowering_of_a_property_return(self, monkeypatch): + """A property RETURN lowers to the multiplicity-preserving ``rows(binding_ops=...)`` + form. The fast path must still engage there: declining would be value-correct but + would drop this shape (LDBC IS5) back onto the general lane.""" + g, P = _graph() + seed = P + 42 + hits = {"n": 0} + real = gfql_unified._execute_seeded_typed_hop_fast_path + + def spy(*a, **k): + r = real(*a, **k) + if r is not None: + hits["n"] += 1 + return r + + monkeypatch.setattr(gfql_unified, "_execute_seeded_typed_hop_fast_path", spy) + g.gfql( + f"MATCH (m:Message {{id: {seed}}})-[:HAS_CREATOR]->(p:Person) RETURN p.age AS age", + engine="pandas", + ) + assert hits["n"] >= 1 + + @pytest.mark.parametrize("engine", ["pandas", "polars", "cudf"]) + def test_parallel_edges_keep_their_row_through_the_fast_path(self, engine): + """Two parallel seed->dest edges are two openCypher rows. The reduction's + destination-node dedup must not eat one, on either the property lowering or + the whole-entity one.""" + pytest.importorskip(engine) if engine != "pandas" else None + nodes = pd.DataFrame({"id": [0, 1], "type": ["Message", "Person"]}) + edges = pd.DataFrame({"src": [0, 0], "dst": [1, 1], "type": ["HAS_CREATOR"] * 2}) + if engine == "polars": + import polars as pl + g = graphistry.nodes(pl.from_pandas(nodes), "id").edges(pl.from_pandas(edges), "src", "dst") + elif engine == "cudf": + import cudf + g = graphistry.nodes(cudf.from_pandas(nodes), "id").edges(cudf.from_pandas(edges), "src", "dst") + else: + g = graphistry.nodes(nodes, "id").edges(edges, "src", "dst") + q = "MATCH (m {id: 0})-[:HAS_CREATOR]->(p) RETURN p.id AS pid" + got = _canon_nodes(g.gfql(q, engine=engine)) + assert got["pid"].tolist() == [1, 1] + @pytest.mark.parametrize("cy_tmpl,reason", [ ("MATCH (m:Message {{id: {s}}})-[:HAS_CREATOR]->(p:Person) RETURN m, p", "multi-alias"), ("MATCH (m:Message {{id: {s}}})-[:HAS_CREATOR]->(p:Person) RETURN m.id, p.age", "cross-alias field projection"), @@ -745,32 +784,11 @@ def _pl_graph(self): # 3. This repo's own POLARS full path already returns int64/bool for this exact # query — only the pandas merge upcasts. So the upcast is engine-local, which is # what an artifact looks like and what a semantic does not. - # => int64/bool is the CONFORMANT result. Where a fast path serves, it keeps the - # conformant dtype rather than casting back to reproduce the defect (casting back was - # also measured to be per-column and unprincipled: a blanket cast took the suite from - # 20 to 45 failures). - # NOT YET ALIGNED (deliberately out of scope, tracked as follow-up): the pandas full - # path itself, and the Cypher-layer projection pinned by - # `test_pandas_int_bool_dtype_parity` below, still emit the artifact. - _FULL_PATH_PANDAS_UPCASTS: Dict[str, Tuple[str, str]] = { - "a": ("int64", "float64"), "f": ("bool", "object")} - - def _assert_values_equal_conformant_dtypes(self, fast: Plottable, full: Plottable) -> None: - """Values identical; where the pandas full path upcast, the served path keeps the - Cypher-conformant dtype and we assert BOTH sides explicitly.""" - f, u = _canon_nodes(fast), _canon_nodes(full) - assert list(f.columns) == list(u.columns) - for col in f.columns: - rule = self._FULL_PATH_PANDAS_UPCASTS.get(col) - if rule is not None and str(u[col].dtype) == rule[1]: - conformant, artifact = rule - assert str(f[col].dtype) == conformant, ( - f"{col}: served path must keep the conformant dtype " - f"{conformant}, got {f[col].dtype}") - pd.testing.assert_series_equal( - f[col].astype(artifact), u[col], check_names=False) - else: - pd.testing.assert_series_equal(f[col], u[col], check_names=False) + # => int64/bool is the CONFORMANT result. NOT YET ALIGNED anywhere on the Cypher + # surface (deliberately out of scope, tracked as follow-up): the pandas full path, + # the Cypher-layer projection pinned by `test_pandas_int_bool_dtype_parity` below, + # and (since the seeded property RETURN lowers to binding rows) the shapes the + # Cypher fast path declines all emit the artifact, so every pandas lane agrees. def _fast_and_full(self, g, engine, q, expect_engage=True): hits = {"n": 0} @@ -802,16 +820,17 @@ def test_polars_int_bool_dtype_parity(self): pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full)) def test_pandas_datetime_property_declines(self): - """M2/dtype pin: the CYPHER seeded projection still DECLINES a datetime property. - DELIBERATE CHANGE: the assertion used to be a plain frame_equal against the full - path, which incidentally locked the pandas merge upcast (`a` as float64). That was - never the guarantee this test exists for — the guarantee is "the Cypher fast path - declines, and the answer is still right". Since the native chain fast path now - serves the declined shape, `a` comes back int64, which is the conformant type - (see _FULL_PATH_PANDAS_UPCASTS above). Values are still asserted identical.""" + """M2/dtype pin: the CYPHER seeded projection still DECLINES a datetime property, + and the answer is still right. The declined shape no longer reaches the NATIVE + chain fast path either — a seeded property RETURN now lowers to the + multiplicity-preserving binding-rows form, whose kernel the native seeded reduction + does not serve — so both sides land on the same lane and carry the same (pandas + pivot-upcast) dtypes. That artifact is unchanged and still tracked; what moved is + which shapes see it, and the declined shape now agrees with the ENGAGED one, which + casts to the artifact deliberately (test_pandas_int_bool_dtype_parity).""" q = self.Q.replace("p.flag AS f", "p.ts AS t") fast, full = self._fast_and_full(self._typed_graph(), "pandas", q, expect_engage=False) - self._assert_values_equal_conformant_dtypes(fast, full) + pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full)) @pytest.mark.parametrize("engine", ["pandas", "polars"]) def test_edges_empty_frame_not_none(self, engine): @@ -826,17 +845,14 @@ def test_edges_empty_frame_not_none(self, engine): def test_engine_mismatch_declines(self): """M2 pin: a requested-vs-actual engine mismatch DECLINES the Cypher seeded - projection. DELIBERATE CHANGE, same reason as test_pandas_datetime_property_ - declines: the first arm's frame_equal incidentally locked the pandas merge upcast. - The declined shape is now served by the native chain fast path, so `a`/`f` come - back int64/bool — the conformant types. Note arm 2 needs no change at all: the - POLARS full path already returns int64/bool, which is the third piece of evidence - that the pandas float64 is engine-local artifact rather than Cypher semantics.""" + projection, and the answer is still right. Same lane note as + test_pandas_datetime_property_declines: the declined shape no longer reaches the + native chain fast path, so both sides land on the binding-rows lane.""" pytest.importorskip("polars") # polars frames + engine='pandas': full converts to pandas; fast must decline fast, full = self._fast_and_full(self._pl_graph(), "pandas", self.Q, expect_engage=False) assert type(fast._nodes).__module__.startswith("pandas") - self._assert_values_equal_conformant_dtypes(fast, full) + pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full)) # pandas frames + engine='polars' (reentry direction): also declines fast2, full2 = self._fast_and_full(self._typed_graph(), "polars", self.Q, expect_engage=False) assert type(fast2._nodes).__module__ == type(full2._nodes).__module__ diff --git a/graphistry/tests/compute/gfql/test_size_nonlist_decline_1985.py b/graphistry/tests/compute/gfql/test_size_nonlist_decline_1985.py new file mode 100644 index 0000000000..6d7c07c8eb --- /dev/null +++ b/graphistry/tests/compute/gfql/test_size_nonlist_decline_1985.py @@ -0,0 +1,267 @@ +"""``size()`` and its list-walking siblings must DECLINE a non-sequence column, never answer. + +openCypher defines ``size()`` over lists and strings. Applied to a numeric/bool/temporal +column it is a type error. The pandas/cuDF row pipeline used to fall through to +``len()``, so the answer was the TABLE ROW COUNT: it changed when unrelated +rows were added and was never a property of the data. ``WHERE size(n.age) = 3`` then kept +every row of a 3-row table and no row of a 4-row table. + +Three call sites shared that swallow, and each gets both sides here: + +========================== ========================== ============================== +surface declines (no defined size) still serves (has a size) +========================== ========================== ============================== +``size(x)`` int / float / bool column string col, list col, literals +``any/all/none/single`` int column list column +``[x IN xs | ...]`` int column list column +``WHERE size(x) = k`` int column list column +========================== ========================== ============================== + +The decline fires on EVIDENCE, not on dtype alone: a column with no non-null cell (an empty +zero-row intermediate, an all-null column) proves nothing about its element type — an empty +``collect()`` is still a list — so those answer null (or no rows) instead of being refused. + +Every serving expectation is hand-computed from the fixtures below; ``size()`` over a +string column is CHARACTER length (openCypher), which all three engines already served and +which this decline must not touch. No engine is used as another engine's oracle. + +The polars engine reaches these shapes through its own native lowering, which already +declines a non-sequence operand with a typed ``NotImplementedError``; it is pinned here so +the cross-engine story stays "value or typed decline, never the row count". +""" +from __future__ import annotations + +import os +from typing import Any, List, Tuple, Union + +import pandas as pd +import pytest + +import graphistry +from graphistry.Plottable import Plottable +from graphistry.compute.exceptions import GFQLValidationError + +try: + import polars as pl + HAS_POLARS = True +except ImportError: + HAS_POLARS = False + +polars_only = pytest.mark.skipif(not HAS_POLARS, reason="polars not installed") +cudf_only = pytest.mark.skipif( + "TEST_CUDF" not in os.environ, reason="cuDF lane: set TEST_CUDF=1" +) + +ENGINES = [ + "pandas", + pytest.param("polars", marks=polars_only), + pytest.param("cudf", marks=cudf_only), +] + +#: Text every pandas/cuDF decline from the three fixed call sites must carry. +NAMED_LIMIT = "requires list/string input" + +EDGES = pd.DataFrame({"s": [0], "d": [0]}) + + +def _graph(engine: str, nodes: pd.DataFrame) -> Plottable: + if engine == "polars": + return graphistry.nodes(pl.from_pandas(nodes), "id").edges( + pl.from_pandas(EDGES), "s", "d") + if engine == "cudf": + cudf = pytest.importorskip("cudf") + return graphistry.nodes(cudf.from_pandas(nodes), "id").edges( + cudf.from_pandas(EDGES), "s", "d") + return graphistry.nodes(nodes, "id").edges(EDGES, "s", "d") + + +def _ints(n_rows: int) -> pd.DataFrame: + return pd.DataFrame({"id": list(range(n_rows)), "age": [10 + i for i in range(n_rows)]}) + + +STRINGS = pd.DataFrame({"id": [0, 1, 2], "name": ["a", "bb", "ccc"]}) +LISTS = pd.DataFrame({"id": [0, 1, 2], "tags": [["a"], ["a", "b"], []]}) +FLOATS = pd.DataFrame({"id": [0, 1, 2], "v": [1.5, 2.5, 3.5]}) +BOOLS = pd.DataFrame({"id": [0, 1, 2], "v": [True, False, True]}) + + +def _cell(v: Any) -> Any: + return None if isinstance(v, float) and v != v else v + + +def _run(g: Plottable, query: str, column: str, engine: str) -> Tuple[str, Union[str, List[Any]]]: + """``("decline", message)`` or ``("values", [cells])`` — the only two acceptable outcomes.""" + try: + out = g.gfql(query, engine=engine) + except NotImplementedError as exc: + return "decline", str(exc) + except GFQLValidationError as exc: + return "decline", str(exc) + nodes = out._nodes + if hasattr(nodes, "to_pandas"): + nodes = nodes.to_pandas() + return "values", [_cell(v) for v in list(nodes[column])] + + +def _assert_declines(outcome: Tuple[str, Any], engine: str, label: str) -> None: + kind, payload = outcome + assert kind == "decline", f"{engine}: {label} answered {payload!r} instead of declining" + if engine != "polars": + assert NAMED_LIMIT in payload, f"{engine}: {label} declined without naming the limit: {payload}" + + +class TestSizeOnNonSequenceColumnDeclines: + + @pytest.mark.parametrize("engine", ENGINES) + @pytest.mark.parametrize("n_rows", [3, 7]) + def test_size_of_int_column_declines_and_never_returns_the_row_count( + self, engine: str, n_rows: int + ) -> None: + g = _graph(engine, _ints(n_rows)) + outcome = _run(g, "MATCH (n) RETURN size(n.age) AS z", "z", engine) + _assert_declines(outcome, engine, f"size(int) over {n_rows} rows") + + @pytest.mark.parametrize("engine", ENGINES) + @pytest.mark.parametrize("nodes,label", [(FLOATS, "float"), (BOOLS, "bool")]) + def test_size_of_float_or_bool_column_declines( + self, engine: str, nodes: pd.DataFrame, label: str + ) -> None: + g = _graph(engine, nodes) + _assert_declines(_run(g, "MATCH (n) RETURN size(n.v) AS z", "z", engine), engine, f"size({label})") + + @pytest.mark.parametrize("engine", ENGINES) + def test_where_size_of_int_column_declines_instead_of_filtering_on_table_height( + self, engine: str + ) -> None: + """The damaging form: the swallowed answer made this keep ALL rows of a 3-row table + and NO row of a 4-row table, for the same data and the same predicate.""" + for n_rows in (3, 4): + g = _graph(engine, _ints(n_rows)) + outcome = _run(g, "MATCH (n) WHERE size(n.age) = 3 RETURN n.id AS id", "id", engine) + _assert_declines(outcome, engine, f"WHERE size(int)=3 over {n_rows} rows") + + +class TestSizeKeepsServingWhatHasASize: + + @pytest.mark.parametrize("engine", ENGINES) + def test_size_of_string_column_is_character_length(self, engine: str) -> None: + g = _graph(engine, STRINGS) + assert _run(g, "MATCH (n) RETURN size(n.name) AS z", "z", engine) == ("values", [1, 2, 3]) + + @pytest.mark.parametrize("engine", ENGINES) + def test_size_of_list_column_is_element_count(self, engine: str) -> None: + g = _graph(engine, LISTS) + assert _run(g, "MATCH (n) RETURN size(n.tags) AS z", "z", engine) == ("values", [1, 2, 0]) + + @pytest.mark.parametrize("engine", ENGINES) + def test_size_of_all_null_column_is_null_not_a_decline(self, engine: str) -> None: + nodes = pd.DataFrame({"id": [0, 1, 2], "nl": [None, None, None]}) + g = _graph(engine, nodes) + assert _run(g, "MATCH (n) RETURN size(n.nl) AS z", "z", engine) == ("values", [None, None, None]) + + @pytest.mark.parametrize("engine", ENGINES) + @pytest.mark.parametrize("literal,expected", [("[1,2,3]", 3), ("'abc'", 3)]) + def test_size_of_a_literal_still_counts_the_literal( + self, engine: str, literal: str, expected: int + ) -> None: + g = _graph(engine, STRINGS) + assert _run(g, f"MATCH (n) RETURN size({literal}) AS z", "z", engine) == ( + "values", [expected] * 3) + + +class TestUnknownElementTypeAnswersRatherThanDeclining: + """A column with no non-null cell is UNKNOWN, not proven non-sequence — an empty + ``collect()`` is still a list, so these must answer rather than be refused on dtype.""" + + @pytest.mark.parametrize("engine", ENGINES) + def test_size_of_all_null_float_column_is_null_not_the_row_count(self, engine: str) -> None: + nodes = pd.DataFrame({"id": [0, 1, 2], "v": [float("nan")] * 3}) + g = _graph(engine, nodes) + outcome = _run(g, "MATCH (n) RETURN size(n.v) AS z", "z", engine) + if engine == "polars": + _assert_declines(outcome, engine, "size(all-null float)") + else: + assert outcome == ("values", [None, None, None]) + + @pytest.mark.parametrize("engine", ENGINES) + def test_size_of_int_column_over_a_zero_row_table_serves_no_rows(self, engine: str) -> None: + nodes = pd.DataFrame({"id": pd.Series([], dtype="int64"), + "age": pd.Series([], dtype="int64")}) + edges = pd.DataFrame({"s": pd.Series([], dtype="int64"), + "d": pd.Series([], dtype="int64")}) + if engine == "polars": + g = graphistry.nodes(pl.from_pandas(nodes), "id").edges(pl.from_pandas(edges), "s", "d") + elif engine == "cudf": + cudf = pytest.importorskip("cudf") + g = graphistry.nodes(cudf.from_pandas(nodes), "id").edges( + cudf.from_pandas(edges), "s", "d") + else: + g = graphistry.nodes(nodes, "id").edges(edges, "s", "d") + outcome = _run(g, "MATCH (n) RETURN size(n.age) AS z", "z", engine) + if engine == "polars": + _assert_declines(outcome, engine, "size(int) over 0 rows") + else: + assert outcome == ("values", []) + + @pytest.mark.parametrize("engine", ["pandas", pytest.param("cudf", marks=cudf_only)]) + def test_size_of_comprehension_over_an_empty_collect_is_zero(self, engine: str) -> None: + """An OPTIONAL MATCH that binds nothing collects to ``[]``, whose size is 0.""" + nodes = pd.DataFrame({"id": ["n1"]}) + edges = pd.DataFrame({"s": [], "d": [], "type": []}) + if engine == "cudf": + cudf = pytest.importorskip("cudf") + g = graphistry.nodes(cudf.from_pandas(nodes), "id").edges( + cudf.from_pandas(edges), "s", "d") + else: + g = graphistry.nodes(nodes, "id").edges(edges, "s", "d") + query = ("MATCH (n) OPTIONAL MATCH (n)-[r]->(m) " + "RETURN size([x IN collect(r) WHERE x <> null]) AS cn") + assert _run(g, query, "cn", engine) == ("values", [0]) + + +class TestQuantifiersOverNonSequenceColumnDecline: + + @pytest.mark.parametrize("engine", ENGINES) + @pytest.mark.parametrize("fn", ["any", "all", "none", "single"]) + def test_quantifier_over_int_column_declines_instead_of_answering_from_zero_elements( + self, engine: str, fn: str + ) -> None: + """The swallow made the element count 0, so any/single said False and all/none said + True — four confident answers about a column that has no elements at all.""" + g = _graph(engine, _ints(3)) + outcome = _run(g, f"MATCH (n) RETURN {fn}(x IN n.age WHERE x > 0) AS z", "z", engine) + _assert_declines(outcome, engine, f"{fn}(int)") + if engine != "polars": + assert f"{fn}()" in outcome[1], f"{engine}: decline must name {fn}(): {outcome[1]}" + + @pytest.mark.parametrize("engine", ENGINES) + def test_quantifier_over_list_column_still_serves(self, engine: str) -> None: + g = _graph(engine, LISTS) + outcome = _run(g, "MATCH (n) RETURN any(x IN n.tags WHERE x = 'a') AS z", "z", engine) + if engine == "polars": + _assert_declines(outcome, engine, "any(list)") + else: + assert outcome == ("values", [True, True, False]) + + +class TestListComprehensionOverNonSequenceColumnDeclines: + + @pytest.mark.parametrize("engine", ENGINES) + def test_list_comprehension_over_int_column_declines_instead_of_yielding_empty( + self, engine: str + ) -> None: + g = _graph(engine, _ints(3)) + outcome = _run(g, "MATCH (n) RETURN [x IN n.age | x] AS z", "z", engine) + _assert_declines(outcome, engine, "comprehension(int)") + if engine != "polars": + assert "list comprehension" in outcome[1], \ + f"{engine}: decline must name the comprehension: {outcome[1]}" + + @pytest.mark.parametrize("engine", ENGINES) + def test_list_comprehension_over_list_column_still_serves(self, engine: str) -> None: + g = _graph(engine, LISTS) + outcome = _run(g, "MATCH (n) RETURN [x IN n.tags | x] AS z", "z", engine) + if engine == "polars": + _assert_declines(outcome, engine, "comprehension(list)") + else: + assert outcome == ("values", [["a"], ["a", "b"], []]) diff --git a/graphistry/tests/compute/gfql/test_strictness_levels.py b/graphistry/tests/compute/gfql/test_strictness_levels.py new file mode 100644 index 0000000000..8ffd85cbe6 --- /dev/null +++ b/graphistry/tests/compute/gfql/test_strictness_levels.py @@ -0,0 +1,610 @@ +"""Strictness levels for absent labels/properties (#1916). + +Pins the three levels, the bool mapping, the validator/executor agreement matrix, +schema-declared typo-vs-narrow-instance disambiguation, and the remote wire field. +""" + +from typing import Any, Dict, List, Optional +from unittest import mock +import typing +import warnings + +import pandas as pd +import pytest + +import graphistry +from graphistry.Plottable import Plottable +from graphistry.compute.chain_remote import chain_remote_generic +from graphistry.compute.exceptions import GFQLSchemaError, GFQLValidationError +from graphistry.compute.gfql.strictness import ( + DEFAULT_STRICT_LEVEL, + UNSCOPED_STRICT_LEVEL, + StrictLevel, + absent_column_matches, + normalize_strict_level, + resolve_strict_level, + schema_declared_names, + strict_level_to_bool, +) +from graphistry.compute.gfql_validate import gfql_validate +from graphistry.tests.compute.gfql.polars_test_utils import engine_skip_reason +from graphistry.schema import EdgeType, GraphSchema, NodeType + + +ABSENT_LABEL = "MATCH (n:Nope) RETURN n.id AS id" +ABSENT_PROP_RETURN = "MATCH (n) RETURN n.nope_col AS c" +ABSENT_PROP_WHERE = "MATCH (n) WHERE n.nope_col = 1 RETURN n.id AS id" +ABSENT_PROP_PATTERN = "MATCH (n {nope_col: 1}) RETURN n.id AS id" +ABSENT_EDGE_LABEL = "MATCH (n)-[e:NOPE]->(m) RETURN n.id AS id" + +FOUR_SHAPES = [ABSENT_LABEL, ABSENT_PROP_RETURN, ABSENT_PROP_WHERE, ABSENT_PROP_PATTERN] + +_StrictTestLevel = typing.Union[StrictLevel, bool] + + +ENGINES = ("pandas", "polars", "cudf") + + +def _graph(engine: str = "pandas") -> Plottable: + nodes = pd.DataFrame({"id": [1, 2, 3], "t": ["a", "b", "a"]}) + edges = pd.DataFrame({"s": [1, 2], "d": [2, 3]}) + if engine == "polars": + pl = pytest.importorskip("polars") + nodes, edges = pl.from_pandas(nodes), pl.from_pandas(edges) + elif engine == "cudf": + cudf = pytest.importorskip("cudf") + nodes, edges = cudf.from_pandas(nodes), cudf.from_pandas(edges) + return graphistry.edges(edges, "s", "d").nodes(nodes, "id") + + +def _polars_gpu_graph() -> Plottable: + graph = _graph("polars") + reason = engine_skip_reason( + "polars-gpu", + lambda: graph.gfql( + "MATCH (n) RETURN n.id AS id", engine="polars-gpu", strict="quiet" + ), + ) + if reason is not None: + pytest.skip(reason) + return graph + + +def _norm(value: Any) -> Any: # hygiene-ok: explicit-any -- row cells are heterogeneous + # py3.13 pandas renders a null cell as float nan where 3.12 gave None + if isinstance(value, float) and value != value: + return None + return None if value is pd.NA else value + + +def _rows(g: Plottable) -> List[Dict[str, Any]]: # hygiene-ok: explicit-any -- row cells are heterogeneous + if g._nodes is None: + return [] + frame = g._nodes + if hasattr(frame, "to_pandas") and not isinstance(frame, pd.DataFrame): + frame = frame.to_pandas() + return [{k: _norm(v) for k, v in row.items()} for row in frame.to_dict("records")] + + +def _gfql_warnings(g: Plottable, query: str, **kwargs: Any) -> List[str]: # hygiene-ok: explicit-any -- passthrough kwargs + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + g.gfql(query, **kwargs) + return [str(w.message) for w in caught if issubclass(w.category, UserWarning) and "GFQL" in str(w.message)] + + +# --------------------------------------------------------------------------- +# level resolution + bool mapping +# --------------------------------------------------------------------------- + +def test_default_level_is_warn() -> None: + assert DEFAULT_STRICT_LEVEL == "warn" + assert resolve_strict_level(_graph()) == "warn" + + +def test_bool_true_maps_to_strict_and_false_to_quiet() -> None: + assert normalize_strict_level(True) == "strict" + assert normalize_strict_level(False) == "quiet" + assert normalize_strict_level(None) is None + assert strict_level_to_bool("strict") is True + assert strict_level_to_bool("warn") is False + assert strict_level_to_bool("quiet") is False + + +def test_unknown_level_rejected() -> None: + with pytest.raises(ValueError): + normalize_strict_level("loose") + + +def test_precedence_explicit_beats_schema() -> None: + g = _graph().bind(schema=GraphSchema(node_types=[NodeType("P", properties={"id": int})], strict=True)) + assert resolve_strict_level(g) == "strict" + assert resolve_strict_level(g, strict="quiet") == "quiet" + + +def test_precedence_schema_metadata_tier() -> None: + from graphistry.compute.gfql.ir.compilation import GraphSchemaCatalog + + catalog = GraphSchemaCatalog.from_schema_parts( + node_columns=("id",), edge_columns=("s", "d"), metadata={"strict": "quiet"} + ) + g = _graph().bind(schema=catalog) + assert resolve_strict_level(g) == "quiet" + + +def test_unscoped_runtime_stays_strict() -> None: + # a direct filter_nodes_by_dict is not a GFQL call and keeps raising + assert UNSCOPED_STRICT_LEVEL == "strict" + with pytest.raises(GFQLSchemaError): + _graph().filter_nodes_by_dict({"nope_col": 1}) + + +# --------------------------------------------------------------------------- +# execution semantics: absent name resolves to null (openCypher) +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_absent_label_is_zero_rows(engine: str, level: str) -> None: + assert _rows(_graph(engine).gfql(ABSENT_LABEL, strict=level)) == [] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_absent_edge_label_is_zero_rows(engine: str, level: str) -> None: + assert _rows(_graph(engine).gfql(ABSENT_EDGE_LABEL, strict=level)) == [] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_absent_property_in_where_is_zero_rows(engine: str, level: str) -> None: + assert _rows(_graph(engine).gfql(ABSENT_PROP_WHERE, strict=level)) == [] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_absent_property_in_pattern_is_zero_rows(engine: str, level: str) -> None: + assert _rows(_graph(engine).gfql(ABSENT_PROP_PATTERN, strict=level)) == [] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_absent_property_in_return_is_null_column(engine: str, level: str) -> None: + assert _rows(_graph(engine).gfql(ABSENT_PROP_RETURN, strict=level)) == [{"c": None}] * 3 + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_polars_gpu_absent_label_is_zero_rows(level: StrictLevel) -> None: + graph = _polars_gpu_graph() + assert _rows( + graph.gfql(ABSENT_LABEL, engine="polars-gpu", strict=level) + ) == [] + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_polars_gpu_absent_return_property_is_null(level: StrictLevel) -> None: + graph = _polars_gpu_graph() + assert _rows( + graph.gfql(ABSENT_PROP_RETURN, engine="polars-gpu", strict=level) + ) == [{"c": None}] * 3 + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +@pytest.mark.parametrize("query", [ABSENT_PROP_WHERE, ABSENT_PROP_PATTERN]) +def test_polars_gpu_absent_property_predicates_are_zero_rows( + level: StrictLevel, query: str +) -> None: + graph = _polars_gpu_graph() + assert _rows(graph.gfql(query, engine="polars-gpu", strict=level)) == [] + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_polars_gpu_strict_absent_names_raise(query: str) -> None: + with pytest.raises(GFQLSchemaError): + _polars_gpu_graph().gfql( + query, engine="polars-gpu", strict="strict" + ) + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_polars_gpu_warns_once_for_absent_names(query: str) -> None: + messages = _gfql_warnings( + _polars_gpu_graph(), query, engine="polars-gpu", strict="warn" + ) + assert len(messages) == 1 + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_polars_gpu_quiet_emits_no_warning(query: str) -> None: + messages = _gfql_warnings( + _polars_gpu_graph(), query, engine="polars-gpu", strict="quiet" + ) + assert messages == [] + + +@pytest.mark.parametrize("engine", ENGINES) +def test_absent_property_is_null_so_is_null_matches_every_row(engine: str) -> None: + # 3VL: every comparison against null is null, but IS NULL on an absent property is TRUE + rows = _rows(_graph(engine).gfql("MATCH (n) WHERE n.nope_col IS NULL RETURN n.id AS id", strict="quiet")) + assert [r["id"] for r in rows] == [1, 2, 3] + + +@pytest.mark.parametrize("engine", ENGINES) +def test_absent_property_is_not_null_matches_no_row(engine: str) -> None: + rows = _rows(_graph(engine).gfql("MATCH (n) WHERE n.nope_col IS NOT NULL RETURN n.id AS id", strict="quiet")) + assert rows == [] + + +ABSENT_PROP_OR_EXPR = "MATCH (n) WHERE n.t = 'a' OR n.nope_col = 1 RETURN n.id AS id" +ABSENT_PROP_NOT_EXPR = "MATCH (n) WHERE NOT n.nope_col = 1 RETURN n.id AS id" + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_absent_property_in_a_row_expression_is_null_not_false(level: str) -> None: + # `null OR true` is true, so the t='a' disjunct still carries its two rows + assert len(_rows(_graph().gfql(ABSENT_PROP_OR_EXPR, strict=level))) == 2 + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_negating_an_absent_property_still_matches_nothing(level: str) -> None: + # `NOT null` is null, not true + assert _rows(_graph().gfql(ABSENT_PROP_NOT_EXPR, strict=level)) == [] + + +@pytest.mark.parametrize("query", [ABSENT_PROP_OR_EXPR, ABSENT_PROP_NOT_EXPR]) +def test_row_expression_absent_property_raises_under_strict(query: str) -> None: + # master served these leniently while its own validator rejected them + with pytest.raises(GFQLSchemaError): + _graph().gfql(query, strict="strict") + + +@pytest.mark.parametrize("query", [ABSENT_PROP_OR_EXPR, ABSENT_PROP_NOT_EXPR]) +@pytest.mark.parametrize("level", ["strict", "warn", "quiet"]) +def test_row_expression_validator_and_executor_agree(query: str, level: str) -> None: + g = _graph() + assert _validator_verdict(g, query, level) == _executor_verdict(g, query, level) + + +def test_present_column_absent_value_is_unchanged() -> None: + # scope discipline: only ABSENT names change; a present column keeps its semantics + assert _rows(_graph().gfql("MATCH (n) WHERE n.t = 'zzz' RETURN n.id AS id", strict="warn")) == [] + assert len(_rows(_graph().gfql("MATCH (n) WHERE n.t = 'a' RETURN n.id AS id", strict="warn"))) == 2 + + +def test_type_mismatch_still_raises_at_every_level() -> None: + # E302 is not an absent name; leniency must not swallow it + for level in ("strict", "warn", "quiet"): + with pytest.raises(GFQLSchemaError): + _graph().gfql("MATCH (n {id: 'not-a-number'}) RETURN n.id AS id", strict=level) + + +# --------------------------------------------------------------------------- +# strict is behavior-preserving; warn warns once; quiet is silent +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("query", [ABSENT_LABEL, ABSENT_PROP_WHERE, ABSENT_PROP_PATTERN, ABSENT_EDGE_LABEL]) +@pytest.mark.parametrize("level", ["strict", True]) +def test_strict_still_raises_the_same_error(query: str, level: Any) -> None: # hygiene-ok: explicit-any -- level is bool | str by design + with pytest.raises(GFQLSchemaError) as exc: + _graph().gfql(query, strict=level) + assert exc.value.code == "column-not-found" + + +def test_strict_raises_on_absent_return_property() -> None: + # the validator already rejected this under strict; the executor now agrees + with pytest.raises(GFQLSchemaError): + _graph().gfql(ABSENT_PROP_RETURN, strict="strict") + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_warn_emits_exactly_one_warning(query: str) -> None: + assert len(_gfql_warnings(_graph(), query, strict="warn")) == 1 + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_quiet_emits_no_warning(query: str) -> None: + assert _gfql_warnings(_graph(), query, strict="quiet") == [] + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_bool_false_is_quiet_not_warn(query: str) -> None: + assert _gfql_warnings(_graph(), query, strict=False) == [] + + +def test_warn_once_per_distinct_name_not_per_row() -> None: + nodes = pd.DataFrame({"id": list(range(50)), "t": ["a"] * 50}) + edges = pd.DataFrame({"s": [0], "d": [1]}) + g = graphistry.edges(edges, "s", "d").nodes(nodes, "id") + assert len(_gfql_warnings(g, "MATCH (n) RETURN n.nope_col AS c", strict="warn")) == 1 + + +def test_warn_once_per_name_two_distinct_names_warn_twice() -> None: + msgs = _gfql_warnings( + _graph(), "MATCH (n) RETURN n.nope_a AS a, n.nope_b AS b", strict="warn" + ) + assert len(msgs) == 2 + + +def test_warning_names_the_absent_name() -> None: + (msg,) = _gfql_warnings(_graph(), ABSENT_PROP_WHERE, strict="warn") + assert "nope_col" in msg + + +def test_absent_label_warning_says_label() -> None: + (msg,) = _gfql_warnings(_graph(), ABSENT_LABEL, strict="warn") + assert "label" in msg and "Nope" in msg + + +# --------------------------------------------------------------------------- +# validator and executor agree at every level (#1889 pattern) +# --------------------------------------------------------------------------- + +def _validator_verdict(g: Plottable, query: str, level: Any) -> str: # hygiene-ok: explicit-any -- level is bool | str by design + try: + gfql_validate(g, query, strict=level) + return "ok" + except GFQLValidationError: + return "raise" + + +def _executor_verdict(g: Plottable, query: str, level: Any) -> str: # hygiene-ok: explicit-any -- level is bool | str by design + try: + g.gfql(query, strict=level) + return "ok" + except GFQLValidationError: + return "raise" + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +@pytest.mark.parametrize("level", ["strict", "warn", "quiet", True, False, None]) +def test_validator_and_executor_agree(query: str, level: Any) -> None: # hygiene-ok: explicit-any -- level is bool | str by design + g = _graph() + assert _validator_verdict(g, query, level) == _executor_verdict(g, query, level) + + +@pytest.mark.parametrize("query", FOUR_SHAPES) +def test_agreement_matrix_values(query: str) -> None: + g = _graph() + assert _validator_verdict(g, query, "strict") == "raise" + assert _validator_verdict(g, query, "warn") == "ok" + assert _validator_verdict(g, query, "quiet") == "ok" + + +@pytest.mark.parametrize("level", ["warn", "quiet", False, None]) +def test_edge_label_agrees_wherever_the_validator_can_judge(level: Any) -> None: # hygiene-ok: explicit-any -- level is bool | str by design + g = _graph() + assert _validator_verdict(g, ABSENT_EDGE_LABEL, level) == _executor_verdict(g, ABSENT_EDGE_LABEL, level) + + +@pytest.mark.parametrize("level", ["strict", True]) +def test_absent_relationship_type_without_a_carrier_agrees_under_strict(level: _StrictTestLevel) -> None: + g = _graph() + assert _validator_verdict(g, ABSENT_EDGE_LABEL, level) == "raise" + assert _executor_verdict(g, ABSENT_EDGE_LABEL, level) == "raise" + + +@pytest.mark.parametrize("level", ["strict", True]) +def test_generic_relationship_type_carrier_remains_unjudgeable_without_a_scan(level: _StrictTestLevel) -> None: + g = _graph() + assert isinstance(g._edges, pd.DataFrame) + g = g.edges(g._edges.assign(type=["KNOWS", "KNOWS"]), "s", "d") + + assert _validator_verdict(g, ABSENT_EDGE_LABEL, level) == "ok" + assert _executor_verdict(g, ABSENT_EDGE_LABEL, level) == "ok" + + +def test_a_declared_relationship_type_makes_the_validator_judge_it() -> None: + with pytest.raises(GFQLValidationError): + _schema_graph("strict").gfql_validate("MATCH (n)-[e:NOPE]->(m) RETURN n.id", strict="strict") + + +def test_validate_true_on_gfql_uses_the_resolved_level() -> None: + # gfql(validate=True) used to hardcode strict=True regardless of the caller's choice + assert _rows(_graph().gfql(ABSENT_PROP_WHERE, validate=True, strict="quiet")) == [] + with pytest.raises(GFQLValidationError): + _graph().gfql(ABSENT_PROP_WHERE, validate=True, strict="strict") + + +# --------------------------------------------------------------------------- +# schema-declared typo vs narrow instance +# --------------------------------------------------------------------------- + +def _schema_graph(level: Any = "warn") -> Plottable: # hygiene-ok: explicit-any -- level is bool | str by design + schema = GraphSchema( + node_types=[NodeType("Person", properties={"id": int, "t": str, "city": str})], + edge_types=[EdgeType("KNOWS", source="Person", destination="Person", + properties={"s": int, "d": int})], + strict=level, + ) + return _graph().bind(schema=schema) + + +def test_schema_declared_names_collects_properties_and_labels() -> None: + names = schema_declared_names(_schema_graph()) + assert names is not None + assert {"city", "id", "t", "Person", "label__Person"} <= names + + +def test_schema_declared_names_none_without_a_schema() -> None: + assert schema_declared_names(_graph()) is None + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_name_in_schema_absent_from_instance_is_served(level: str) -> None: + # the narrow-subgraph case the owner ruled for + assert _rows(_schema_graph(level).gfql("MATCH (n) WHERE n.city = 'x' RETURN n.id AS id", strict=level)) == [] + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_name_absent_from_schema_is_a_typo_and_still_raises(level: str) -> None: + with pytest.raises(GFQLValidationError): + _schema_graph(level).gfql("MATCH (n) WHERE n.ciyt = 'x' RETURN n.id AS id", strict=level) + + +@pytest.mark.parametrize("level", ["warn", "quiet"]) +def test_label_absent_from_schema_is_a_typo_and_still_raises(level: str) -> None: + with pytest.raises(GFQLValidationError): + _schema_graph(level).gfql("MATCH (n:Persno) RETURN n.id AS id", strict=level) + + +def test_declared_label_absent_from_instance_is_served() -> None: + assert _rows(_schema_graph("quiet").gfql("MATCH (n:Person) RETURN n.id AS id", strict="quiet")) == [] + + +# --------------------------------------------------------------------------- +# chain() surface +# --------------------------------------------------------------------------- + +def test_chain_honors_the_level() -> None: + from graphistry.compute.ast import n + + g = _graph() + assert len(g.chain([n(filter_dict={"nope_col": 1})], strict="quiet")._nodes) == 0 + with pytest.raises(GFQLSchemaError): + g.chain([n(filter_dict={"nope_col": 1})], strict="strict") + + +def test_chain_default_is_warn() -> None: + from graphistry.compute.ast import n + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + out = _graph().chain([n(filter_dict={"nope_col": 1})]) + assert len(out._nodes) == 0 + assert any("GFQL" in str(w.message) for w in caught) + + +# --------------------------------------------------------------------------- +# the warn level needs the process's warning filters intact +# --------------------------------------------------------------------------- + +def test_lazy_cudf_import_leaves_the_global_warning_filters_intact() -> None: + pytest.importorskip("cudf") + from graphistry.utils.lazy_import import lazy_cudf_import + + before = list(warnings.filters) + lazy_cudf_import() + assert list(warnings.filters) == before + + +# --------------------------------------------------------------------------- +# 3VL helper +# --------------------------------------------------------------------------- + +def test_absent_column_matches_only_is_na() -> None: + from graphistry.compute.predicates.comparison import gt, isna, notna + + assert absent_column_matches(isna()) is True + assert absent_column_matches(notna()) is False + assert absent_column_matches(gt(1)) is False + assert absent_column_matches(1) is False + + +# --------------------------------------------------------------------------- +# remote: honor the level in preflight + ship it on the wire +# --------------------------------------------------------------------------- + +class _FakeResponse: + ok = True + status_code = 200 + headers = {"Content-Type": "application/json"} + text = '{"nodes": [], "edges": []}' + + def json(self) -> Dict[str, Any]: # hygiene-ok: explicit-any -- JSON payload + return {"nodes": [], "edges": []} + + +def _remote_body(level: Any, query: str = "MATCH (n) RETURN n", validate: bool = False, # hygiene-ok: explicit-any -- level is bool | str by design + g: Optional[Plottable] = None) -> Dict[str, Any]: # hygiene-ok: explicit-any -- JSON payload + captured: Dict[str, Any] = {} + + def _post(url: str, headers: Any = None, json: Any = None, verify: Any = None, **kwargs: Any) -> _FakeResponse: # hygiene-ok: explicit-any -- requests passthrough + captured["body"] = json + return _FakeResponse() + + graph = g if g is not None else _graph() + with mock.patch("graphistry.compute.chain_remote.requests.post", _post): + try: + chain_remote_generic(graph, query, api_token="t", dataset_id="ds", + validate=validate, strict=level) + except AttributeError: + pass # the stub response carries no body to deserialize; the request is what is pinned + return captured["body"] + + +@pytest.mark.parametrize("level,expected", [ + (None, "warn"), ("strict", "strict"), ("warn", "warn"), ("quiet", "quiet"), + (True, "strict"), (False, "quiet"), +]) +def test_remote_sends_strictness_field(level: Any, expected: str) -> None: # hygiene-ok: explicit-any -- level is bool | str by design + assert _remote_body(level)["strictness"] == expected + + +def test_remote_default_sends_no_warning() -> None: + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + _remote_body(None) + assert [w for w in caught if "strictness" in str(w.message)] == [] + + +@pytest.mark.parametrize("level", ["strict", "quiet", True, False]) +def test_remote_non_default_level_warns_once(level: Any) -> None: # hygiene-ok: explicit-any -- level is bool | str by design + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + _remote_body(level) + assert len([w for w in caught if "strictness" in str(w.message)]) == 1 + + +def test_remote_preflight_no_longer_hardcodes_loose() -> None: + # master preflighted with strict=False regardless of the caller's choice + with pytest.raises(GFQLValidationError): + _remote_body("strict", query="MATCH (n) WHERE n.ciyt = 1 RETURN n.id", + validate=True, g=_schema_graph("strict")) + + +def test_remote_preflight_serves_a_declared_schema_without_local_frames() -> None: + # a dataset_id-only client holds no frames, but bind(schema=...) is names without data + schema = GraphSchema(node_types=[NodeType("Person", properties={"id": int, "city": str})], + strict="strict") + g = graphistry.bind().bind(schema=schema) + g._dataset_id = "ds" + with pytest.raises(GFQLValidationError): + _remote_body("strict", query="MATCH (n) WHERE n.ciyt = 1 RETURN n.id", validate=True, g=g) + + +def test_remote_preflight_accepts_declared_name_absent_from_the_instance() -> None: + schema = GraphSchema(node_types=[NodeType("Person", properties={"id": int, "city": str})], + strict="strict") + g = graphistry.bind().bind(schema=schema) + g._dataset_id = "ds" + body = _remote_body("strict", query="MATCH (n) WHERE n.city = 1 RETURN n.id", validate=True, g=g) + assert body["strictness"] == "strict" + + +def test_leniency_does_not_swallow_errors_unrelated_to_absence() -> None: + """Only an absent column is leniency-eligible. + + A frame with integer column names makes ``resolve_filter_column`` raise + ``TypeError`` while building its own suggestion string. That is a real error, + not an absent key, so it must surface at every level rather than be reported + to the caller as "column is absent". + """ + import warnings + + import pandas as pd + + from graphistry.compute.filter_by_dict import resolve_filter_column_or_absent + from graphistry.compute.gfql.strictness import strictness_scope + + df = pd.DataFrame({0: [1, 2], 1: [3, 4]}) + + for level in ("strict", "warn", "quiet"): + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + with pytest.raises(TypeError): + with strictness_scope(level): # type: ignore[arg-type] + resolve_filter_column_or_absent(df, "missing", 1, context="nodes") + absent_warnings = [w for w in caught if "is absent" in str(w.message)] + assert absent_warnings == [], ( + f"level={level} reported an unrelated TypeError as an absent column" + ) diff --git a/graphistry/tests/compute/gfql/test_temporal_leak_family_1915.py b/graphistry/tests/compute/gfql/test_temporal_leak_family_1915.py new file mode 100644 index 0000000000..978a7531e9 --- /dev/null +++ b/graphistry/tests/compute/gfql/test_temporal_leak_family_1915.py @@ -0,0 +1,479 @@ +"""Pins for the temporal/error-leak family: #1915 B-5/B-7/B-8 + A-4, #1880 temporal half. + +Oracles are hand-computed from openCypher CIP2016-06-14 ("Comparability and equality"): + +- "Temporal instant values with timezone (`DateTime` and `LocalTime`) are compared on a + global timeline, as if the instants were normalized to UTC." -> same instant under + different offsets IS equal (B-5). +- "Two given instants `a` and `b` are equal if any only if they are of the same type and + neither of them is _before_ or _after_ the other." and "Temporal instant values are + only comparable within types." -> `datetime(...) = localdatetime(...)` is false and + their ordering is null (B-5, the both-engines-wrong case). + +B-7/#1880: temporal-vs-string comparisons must never leak raw backend errors +(polars InvalidOperationError, numpy ufunc TypeError) — they either answer with the +pandas-parity row set or raise a typed GFQL error / typed engine decline. + +B-8: non-reserved keywords are valid property names (`n.when`, `n.order`, ...). + +A-4: UNION branches projecting the SAME names in a different order align by name +(Neo4j semantics; output keeps the first branch's order); different name sets stay +a typed decline. +""" +from __future__ import annotations + +import pandas as pd +import pytest + +import graphistry +from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError, GFQLSyntaxError, GFQLValidationError + +try: + import polars as pl +except ImportError: # pragma: no cover - polars-lane file also runs in core lane + pl = None # type: ignore[assignment] + +try: + import cudf +except ImportError: + cudf = None # type: ignore[assignment] + + +def _nodes_pd() -> pd.DataFrame: + ts = pd.to_datetime([ + "2020-06-15T08:30:00", "2021-06-15T08:30:00", + "2022-01-01T00:00:00", "2019-01-01T00:00:00", None, + ]) + return pd.DataFrame({ + "id": ["p", "q", "r", "s", "t"], + "ts": ts, + "ts_aw": ts.tz_localize("UTC"), + "ts_aw_lag": ts.tz_localize("UTC") - pd.Timedelta(hours=1), + "dur": pd.to_timedelta(["1 days", "2 days", "3 days", "4 days", None]), + "when": [1, 2, 3, 4, 5], + "i": [7, 8, 9, 10, 11], + }) + + +def _edges_pd() -> pd.DataFrame: + return pd.DataFrame({"s": ["p"], "d": ["q"]}) + + +def _graph(engine: str): + nodes, edges = _nodes_pd(), _edges_pd() + if engine == "polars": + assert pl is not None + return graphistry.nodes(pl.from_pandas(nodes), "id").edges(pl.from_pandas(edges), "s", "d") + if engine == "cudf": + assert cudf is not None + return graphistry.nodes(cudf.from_pandas(nodes), "id").edges(cudf.from_pandas(edges), "s", "d") + return graphistry.nodes(nodes, "id").edges(edges, "s", "d") + + +def _rows(g, query: str, engine: str) -> pd.DataFrame: + out = g.gfql(query, engine=engine)._nodes + if pl is not None and isinstance(out, pl.LazyFrame): + out = out.collect() + if hasattr(out, "to_pandas"): + out = out.to_pandas() + return out.reset_index(drop=True) + + +def _ids(g, query: str, engine: str) -> list: + out = _rows(g, query, engine) + col = "n.id" if "n.id" in out.columns else out.columns[0] + return sorted(out[col].tolist()) + + +ENGINES = [ + "pandas", + pytest.param("polars", marks=pytest.mark.skipif(pl is None, reason="polars not installed")), + pytest.param("cudf", marks=pytest.mark.skipif(cudf is None, reason="cudf not installed")), +] + + +# --------------------------------------------------------------------------- +# B-5: literal temporal comparisons +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("engine", ENGINES) +class TestB5LiteralTemporalComparison: + def test_same_instant_different_offsets_equal(self, engine): + """CIP: zoned values compare 'on a global timeline'. Red-at-master on polars + (text equality on rendered literals gave False).""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T05:00:00+05:00') = datetime('2020-01-02T00:00:00Z') AS eq", engine) + assert out["eq"].tolist() == [True] + + def test_same_instant_different_offsets_not_unequal(self, engine): + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T05:00:00+05:00') <> datetime('2020-01-02T00:00:00Z') AS eq", engine) + assert out["eq"].tolist() == [False] + + def test_offset_vs_offset_same_instant_equal(self, engine): + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T05:00:00+05:00') = datetime('2020-01-01T19:00:00-05:00') AS eq", engine) + assert out["eq"].tolist() == [True] + + def test_distinct_instants_stay_unequal(self, engine): + """Anti-vacuity: the fold is a real instant comparison, not a constant True.""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T05:00:00+05:00') = datetime('2020-01-02T00:00:01Z') AS eq", engine) + assert out["eq"].tolist() == [False] + + def test_same_type_ordering_answers(self, engine): + """Ordering two zoned literals compares instants (polars declined this NIE before).""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T05:00:00+05:00') < datetime('2020-01-02T00:00:01Z') AS eq", engine) + assert out["eq"].tolist() == [True] + + def test_zoned_vs_local_equality_is_false(self, engine): + """CIP: equal 'if any only if they are of the same type ...'. Red-at-master on + BOTH pandas and polars (each answered True).""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T00:00:00Z') = localdatetime('2020-01-02T00:00:00') AS eq", engine) + assert out["eq"].tolist() == [False] + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "localdatetime('2020-01-02T00:00:00') = datetime('2020-01-02T00:00:00Z') AS eq", engine) + assert out["eq"].tolist() == [False] + + def test_zoned_vs_local_inequality_is_true(self, engine): + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T00:00:00Z') <> localdatetime('2020-01-02T00:00:00') AS eq", engine) + assert out["eq"].tolist() == [True] + + def test_zoned_vs_local_ordering_is_null(self, engine): + """CIP: 'Temporal instant values are only comparable within types.'""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN " + "datetime('2020-01-02T00:00:00Z') < localdatetime('2020-01-03T00:00:00') AS eq", engine) + assert out["eq"].isna().tolist() == [True] + + def test_where_form_is_row_set_visible(self, engine): + """The audit's B-5 wrong answer changed row sets, not just projections.""" + g = _graph(engine) + ids = _ids(g, "MATCH (n) WHERE datetime('2020-01-02T05:00:00+05:00') = datetime('2020-01-02T00:00:00Z') " + "RETURN n.id", engine) + assert ids == ["p", "q", "r", "s", "t"] # anti-vacuity: all 5 rows survive a true WHERE + ids = _ids(g, "MATCH (n) WHERE datetime('2020-01-02T00:00:00Z') = localdatetime('2020-01-02T00:00:00') " + "RETURN n.id", engine) + assert ids == [] + + def test_plain_string_equality_untouched(self, engine): + """Mutation guard: non-temporal string literals keep plain string semantics.""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id = 'p' RETURN 'a' = 'b' AS eq, 'a' = 'a' AS eq2", engine) + assert out["eq"].tolist() == [False] + assert out["eq2"].tolist() == [True] + + +class TestB5FoldUnits: + """Direct pins on the fold — the arms end-to-end queries cannot isolate.""" + + def _fold(self, op: str, left: str, right: str): + from graphistry.compute.gfql.expr_parser import BinaryOp, Literal + from graphistry.compute.gfql.temporal.folding import _fold_temporal_comparison + return _fold_temporal_comparison(BinaryOp(op=op, left=Literal(left), right=Literal(right))) + + def test_same_type_instant_comparison(self): + assert self._fold("=", "2020-01-02T05:00:00+05:00", "2020-01-02T00:00:00Z").value is True + assert self._fold("<", "2020-01-01", "2020-01-02").value is True + assert self._fold(">=", "12:00:00", "12:00:01").value is False + + def test_cross_type_matrix(self): + """Every distinct-kind pair: eq false, neq true, ordering null.""" + exemplars = { + "datetime": "2020-01-02T00:00:00Z", + "localdatetime": "2020-01-02T00:00:00", + "date": "2020-01-02", + "time": "12:00:00Z", + "localtime": "12:00:00", + } + kinds = list(exemplars) + checked = 0 + for i, a in enumerate(kinds): + for b in kinds[i + 1:]: + assert self._fold("=", exemplars[a], exemplars[b]).value is False, (a, b) + assert self._fold("<>", exemplars[a], exemplars[b]).value is True, (a, b) + assert self._fold("<", exemplars[a], exemplars[b]).value is None, (a, b) + checked += 1 + assert checked == 10 # anti-vacuity: all C(5,2) pairs exercised + + def test_zone_name_without_offset_declines_fold(self): + from graphistry.compute.gfql.temporal.values import _parse_temporal_value + from graphistry.compute.gfql.temporal.folding import _temporal_instant_key + value = _parse_temporal_value("2020-01-02T00:00:00[Europe/Paris]") + assert value is not None and _temporal_instant_key(value) is None + assert self._fold("=", "2020-01-02T00:00:00[Europe/Paris]", "2020-01-02T00:00:00Z") is None + + def test_non_string_literals_do_not_fold(self): + from graphistry.compute.gfql.expr_parser import BinaryOp, Literal + from graphistry.compute.gfql.temporal.folding import _fold_temporal_comparison + assert _fold_temporal_comparison(BinaryOp(op="=", left=Literal(5), right=Literal("2020-01-02"))) is None + assert _fold_temporal_comparison(BinaryOp(op="and", left=Literal("2020-01-02"), right=Literal("2020-01-02"))) is None + + def test_offset_with_zone_name_still_folds(self): + assert self._fold("=", "2020-01-02T02:00:00+02:00[Europe/Paris]", "2020-01-02T00:00:00Z").value is True + + def test_non_temporal_strings_do_not_fold(self): + assert self._fold("=", "hello", "2020-01-02") is None + assert self._fold("=", "2020-01-02", "world") is None + + def test_durations_do_not_fold(self): + assert self._fold("=", "P1D", "PT24H") is None + + def test_seconds_offset_parsed(self): + assert self._fold("=", "2020-01-02T00:00:30+00:00:30", "2020-01-02T00:00:00Z").value is True + + +# --------------------------------------------------------------------------- +# B-7 / #1880: temporal-vs-string never leaks raw backend errors +# --------------------------------------------------------------------------- + +class TestB7TemporalStringLeaks: + def test_pandas_zoned_string_vs_naive_column_answers(self): + """Red-at-master: raw numpy `ufunc 'bitwise_and'` TypeError from the pushdown.""" + g = _graph("pandas") + assert _ids(g, "MATCH (n) WHERE n.ts > '2021-01-01T00:00:00Z' RETURN n.id", "pandas") == ["q", "r"] + + def test_pandas_zoned_string_equality_matches_instant(self): + """Red-at-master: the pushed equality silently matched ZERO rows.""" + g = _graph("pandas") + assert _ids(g, "MATCH (n) WHERE n.ts = '2021-06-15T08:30:00Z' RETURN n.id", "pandas") == ["q"] + + @pytest.mark.skipif(cudf is None, reason="cudf not installed") + def test_cudf_zoned_string_vs_naive_column_answers(self): + g = _graph("cudf") + assert _ids(g, "MATCH (n) WHERE n.ts > '2021-01-01T00:00:00Z' RETURN n.id", "cudf") == ["q", "r"] + + @pytest.mark.skipif(pl is None, reason="polars not installed") + def test_polars_zoned_string_vs_naive_column_typed_decline(self): + """The where_rows residual declines typed on polars — never the raw + InvalidOperationError this leaked at master.""" + g = _graph("polars") + with pytest.raises(NotImplementedError, match="where_rows"): + g.gfql("MATCH (n) WHERE n.ts > '2021-01-01T00:00:00Z' RETURN n.id", engine="polars") + + @pytest.mark.parametrize("engine", [ + "pandas", + pytest.param("polars", marks=pytest.mark.skipif(pl is None, reason="polars not installed")), + ]) + def test_naive_string_vs_datetime_column_parity(self, engine): + """#1880: `a.dt = '...'` leaked a raw polars InvalidOperationError; both engines + must answer the same rows.""" + g = _graph(engine) + assert _ids(g, "MATCH (n) WHERE n.ts = '2021-06-15T08:30:00' RETURN n.id", engine) == ["q"] + assert _ids(g, "MATCH (n) WHERE n.ts > '2021-01-01T00:00:00' RETURN n.id", engine) == ["q", "r"] + + @pytest.mark.parametrize("engine", [ + "pandas", + pytest.param("polars", marks=pytest.mark.skipif(pl is None, reason="polars not installed")), + ]) + def test_string_vs_duration_column_parity(self, engine): + """#1880: `a.dur = '1 days'` leaked a raw polars cast InvalidOperationError.""" + g = _graph(engine) + assert _ids(g, "MATCH (n) WHERE n.dur = '1 days' RETURN n.id", engine) == ["p"] + assert _ids(g, "MATCH (n) WHERE n.dur > '2 days' RETURN n.id", engine) == ["r", "s"] + + @pytest.mark.skipif(pl is None, reason="polars not installed") + def test_polars_unparseable_string_typed_schema_error(self): + """Fail-closed (#1880): what cannot compare declines with the SAME typed error + family the scalar half raises (E302), never a raw polars exception.""" + from graphistry.compute.ast import n + from graphistry.compute.predicates.comparison import gt + g = _graph("polars") + for chain in ([n({"ts": "not-a-timestamp"})], [n({"ts": gt("not-a-timestamp")})], + [n({"dur": "not-a-duration"})], [n({"ts": gt("2021-01-01T00:00:00Z")})]): + with pytest.raises(GFQLSchemaError) as excinfo: + g.gfql(chain, engine="polars") + assert excinfo.value.code == ErrorCode.E302 + + @pytest.mark.skipif(pl is None, reason="polars not installed") + def test_polars_chain_predicate_parity_rows(self): + """Anti-vacuity for the parse-and-compare lowering: real rows, pandas-identical.""" + from graphistry.compute.ast import n + from graphistry.compute.predicates.comparison import gt, le + for chain, expect in ( + ([n({"ts": gt("2021-01-01T00:00:00")})], ["q", "r"]), + ([n({"ts": le("2020-06-15T08:30:00")})], ["p", "s"]), + ([n({"dur": "2 days"})], ["q"]), + ): + got = {} + for engine in ("pandas", "polars"): + out = _graph(engine).gfql(chain, engine=engine)._nodes + if isinstance(out, pl.LazyFrame): + out = out.collect() + if hasattr(out, "to_pandas"): + out = out.to_pandas() + got[engine] = sorted(out["id"].tolist()) + assert got["pandas"] == got["polars"] == expect + + def test_pandas_connected_join_zoned_string_answers(self): + """The connected-join lowering must also keep tz-suffixed text a residual.""" + g = _graph("pandas") + out = _rows(g, "MATCH (a)-[]->(b) WHERE b.ts > '2021-01-01T00:00:00Z' RETURN a.id, b.id", + "pandas") + assert out.to_dict("records") == [{"a.id": "p", "b.id": "q"}] + + def test_pandas_optional_match_zoned_string_answers(self): + """Red-at-master: the connected OPTIONAL MATCH lowering pushed the tz-suffixed + literal into a filter dict too, leaking the same raw numpy TypeError.""" + g = _graph("pandas") + out = _rows(g, "MATCH (a) WHERE a.id IN ['p', 'q'] OPTIONAL MATCH (a)-[]->(b) " + "WHERE b.ts > '2021-01-01T00:00:00Z' RETURN a.id, b.id", "pandas") + got = sorted(out.to_dict("records"), key=lambda r: r["a.id"]) + got = [{k: (None if isinstance(v, float) and v != v else v) for k, v in r.items()} for r in got] + assert got == [{"a.id": "p", "b.id": "q"}, {"a.id": "q", "b.id": None}] + + @pytest.mark.parametrize("engine", [ + "pandas", + pytest.param("cudf", marks=pytest.mark.skipif(cudf is None, reason="cudf not installed")), + ]) + def test_naive_vs_aware_columns_answer(self, engine): + """Red-at-master: raw pandas TypeError ('Invalid comparison ...') from the + same-path WHERE. GFQL's extension reads naive datetimes as UTC, so the + equal-instant pair yields no rows and the lagged pair yields all non-null rows.""" + g = _graph(engine) + assert _ids(g, "MATCH (n) WHERE n.ts > n.ts_aw RETURN n.id", engine) == [] + # mutation guard: the comparison is real, not constant-empty + assert _ids(g, "MATCH (n) WHERE n.ts > n.ts_aw_lag RETURN n.id", engine) == ["p", "q", "r", "s"] + + @pytest.mark.skipif(pl is None, reason="polars not installed") + def test_polars_naive_vs_aware_columns_typed_decline(self): + g = _graph("polars") + with pytest.raises(NotImplementedError, match="polars engine does not yet natively support"): + g.gfql("MATCH (n) WHERE n.ts > n.ts_aw RETURN n.id", engine="polars") + + +class TestB7Units: + def test_zoned_iso_regex_accepts_and_rejects(self): + from graphistry.compute.gfql.cypher.lowering import _ZONED_ISO_TEMPORAL_TEXT_RE as rx + for text in ("2021-01-01T00:00:00Z", "2021-01-01T00:00:00+05:00", "2021-01-01 00:00:00-0500", + "2021-01-01T00:00Z", "12:30:00Z", "12:30:00.5+02:00"): + assert rx.match(text), text + for text in ("2021-01-01T00:00:00", "2021-01-01", "1 days", "P1D", "hello", + "12:30:00", "2021-01-01T00:00:00Zx"): + assert not rx.match(text), text + + def test_align_mixed_tz_converts_only_mixed_datetime_pairs(self): + from graphistry.compute.gfql.same_path.df_utils import _align_mixed_tz_datetimes + naive = pd.Series(pd.to_datetime(["2021-01-01"])) + aware = pd.Series(pd.to_datetime(["2021-01-01"]).tz_localize("US/Eastern")) + left, right = _align_mixed_tz_datetimes(naive, aware) + assert right.dt.tz is None and right.iloc[0] == pd.Timestamp("2021-01-01T05:00:00") + assert left is naive + left, right = _align_mixed_tz_datetimes(aware, naive) + assert left.dt.tz is None and right is naive + ints = pd.Series([1]) + assert _align_mixed_tz_datetimes(ints, naive) == (ints, naive) + left, right = _align_mixed_tz_datetimes(naive, naive) + assert left is naive and right is naive + + @pytest.mark.skipif(pl is None, reason="polars not installed") + def test_parse_temporal_filter_scalar_safe_subset(self): + import datetime as dt + from graphistry.compute.gfql.lazy.engine.polars.predicates import _parse_temporal_filter_scalar + assert _parse_temporal_filter_scalar("2021-01-01T00:00:00", pl.Datetime("ns")) == dt.datetime(2021, 1, 1) + assert _parse_temporal_filter_scalar("2021-01-01T00:00:00Z", pl.Datetime("ns")) is None # tz-suffixed + assert _parse_temporal_filter_scalar("2021-01-01T00:00:00.000000001", pl.Datetime("ns")) is None # sub-us + assert _parse_temporal_filter_scalar("junk", pl.Datetime("ns")) is None + assert _parse_temporal_filter_scalar("2021-01-01", pl.Datetime("ns", "UTC")) is None # aware column + assert _parse_temporal_filter_scalar("1 days", pl.Duration("ns")) == dt.timedelta(days=1) + assert _parse_temporal_filter_scalar("junk", pl.Duration("ns")) is None + assert _parse_temporal_filter_scalar("0 days 00:00:00.000000001", pl.Duration("ns")) is None # sub-us + assert _parse_temporal_filter_scalar("2021-01-01", pl.Date) == dt.date(2021, 1, 1) + assert _parse_temporal_filter_scalar("12:30:00", pl.Time) == dt.time(12, 30) + assert _parse_temporal_filter_scalar("2021-01-01", pl.Int64()) is None + + +# --------------------------------------------------------------------------- +# B-8: non-reserved keywords as property names +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("engine", ENGINES) +class TestB8KeywordPropertyNames: + def test_where_on_keyword_property(self, engine): + """Red-at-master: `n.when > 3` raised GFQLSyntaxError.""" + g = _graph(engine) + assert _ids(g, "MATCH (n) WHERE n.when > 3 RETURN n.id", engine) == ["s", "t"] + + def test_return_and_order_by_keyword_property(self, engine): + g = _graph(engine) + out = _rows(g, "MATCH (n) RETURN n.when AS w ORDER BY n.when DESC LIMIT 2", engine) + assert out["w"].tolist() == [5, 4] + + def test_property_map_keyword_key(self, engine): + g = _graph(engine) + assert _ids(g, "MATCH (n {when: 4}) RETURN n.id", engine) == ["s"] + + +class TestB8Grammar: + def test_composite_root_property_access_accepts_keywords(self): + """The expr grammar's property_access rule (composite roots) also takes PROP_NAME.""" + from graphistry.compute.gfql.expr_parser import PropertyAccessExpr, parse_expr + node = parse_expr("(n).when") + assert isinstance(node, PropertyAccessExpr) and node.property == "when" + + @pytest.mark.parametrize("prop", ["when", "then", "end", "order", "is", "all", "any", "contains"]) + def test_issue_keywords_parse_in_where(self, prop): + """Every keyword the audit listed parses as a property name (pandas run).""" + nodes = pd.DataFrame({"id": ["a", "b"], prop: [1, 5]}) + g = graphistry.nodes(nodes, "id").edges(_edges_pd(), "s", "d") + out = g.gfql(f"MATCH (n) WHERE n.{prop} > 3 RETURN n.id", engine="pandas")._nodes + assert out["n.id"].tolist() == ["b"] + + def test_keyword_property_keeps_filter_pushdown(self): + """The WHERE-chain grammar (property_ref) must also accept keyword property + names, or the conjunct silently loses its filter_dict pushdown.""" + import warnings + from graphistry.compute.ast import ASTNode + from graphistry.compute.gfql.cypher.api import compile_cypher + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + compiled = compile_cypher("MATCH (n) WHERE n.when > 3 RETURN n.id") + node_ops = [op for op in compiled.chain.chain if isinstance(op, ASTNode)] + assert node_ops and node_ops[0].filter_dict and "when" in node_ops[0].filter_dict + + def test_keywords_stay_reserved_outside_property_position(self): + """Mutation guard: only the dot/map-key position was unreserved.""" + g = _graph("pandas") + with pytest.raises(GFQLSyntaxError): + g.gfql("MATCH (n) RETURN when", engine="pandas") + + +# --------------------------------------------------------------------------- +# A-4: UNION name alignment +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("engine", ENGINES) +class TestA4UnionNameAlignment: + def test_same_names_different_order_align(self, engine): + """Red-at-master: typed decline. Neo4j aligns by name; the output keeps the + first branch's column order.""" + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id='p' RETURN n.id AS a, n.i AS b " + "UNION MATCH (n) WHERE n.id='q' RETURN n.i AS b, n.id AS a", engine) + assert list(out.columns) == ["a", "b"] + assert sorted(out.to_dict("records"), key=lambda r: r["a"]) == [ + {"a": "p", "b": 7}, {"a": "q", "b": 8}, + ] + + def test_union_all_alignment(self, engine): + g = _graph(engine) + out = _rows(g, "MATCH (n) WHERE n.id='p' RETURN n.id AS a, n.i AS b " + "UNION ALL MATCH (n) WHERE n.id='p' RETURN n.i AS b, n.id AS a", engine) + assert list(out.columns) == ["a", "b"] + assert out.to_dict("records") == [{"a": "p", "b": 7}, {"a": "p", "b": 7}] + + def test_different_name_sets_stay_typed_decline(self, engine): + """The genuine error half of the old message survives, by name.""" + g = _graph(engine) + with pytest.raises(GFQLValidationError, match="must project the same output names"): + g.gfql("MATCH (n) RETURN n.id AS a UNION MATCH (n) RETURN n.i AS c", engine=engine) diff --git a/graphistry/tests/compute/gfql/test_validate_execute_agreement_1889.py b/graphistry/tests/compute/gfql/test_validate_execute_agreement_1889.py new file mode 100644 index 0000000000..fcb3c2539a --- /dev/null +++ b/graphistry/tests/compute/gfql/test_validate_execute_agreement_1889.py @@ -0,0 +1,195 @@ +"""#1889 validate-vs-execute agreement matrix. + +``gfql_validate`` sold itself as preflight ("validate without executing") while returning +``{ok: True, diagnostics: []}`` on graph shapes whose execution then died with a bare +``ValueError: Missing edges`` (pandas/cuDF, ``ComputeMixin.materialize_nodes``) or an +empty-message ``AssertionError`` (polars ``ensure_nodes_polars``). Verified live at master +``0c3f3a1fa`` for the both-frames-None-after-bind shape on both query languages. + +The contract pinned here, per combo and per engine: the validator verdict and the +execution outcome AGREE. Either + * both admit -- validator ok AND execution serves values, or + * both diagnose -- validator raises a typed GFQL diagnostic AND execution declines typed. +No combo may validate clean and then raise a bare (untyped / empty-message) error. + +Master reds (the drift): both-None x {cypher, chain} -- validator ok, execution bare. +Master greens kept red-proof here: nodes-only x {cypher, chain} answers on pandas (#1942), +and an edge pattern against unbound edges declines typed on BOTH surfaces. +""" +from __future__ import annotations + +import math +from typing import Any, Dict, List + +import pandas as pd +import pytest + +import graphistry +from graphistry.compute.ast import e_forward, n +from graphistry.compute.exceptions import ErrorCode, GFQLValidationError + +try: + import polars # noqa: F401 + HAS_POLARS = True +except ImportError: + HAS_POLARS = False + +polars_only = pytest.mark.skipif(not HAS_POLARS, reason="polars not installed") + +ENGINES = ["pandas", pytest.param("polars", marks=polars_only)] + +CYPHER = "MATCH (a) RETURN a" +CHAIN = [n({"v": 20})] + +# Bare = the crash classes #1889 filed: no GFQL code, no remedy (and often no message). +BARE_ERRORS = (ValueError, AssertionError, TypeError, AttributeError, KeyError, IndexError) + + +def _both_none(): + """Bindings set, frames never attached -- graphistry.bind() only names columns.""" + return graphistry.bind(source="s", destination="d", node="id") + + +def _nodes_only(): + return graphistry.nodes(pd.DataFrame({"id": [0, 1], "v": [10, 20]}), "id") + + +def _norm(value: Any) -> Any: + """py3.13 keeps NaN out of record equality: compare at VALUE level, never via notna().""" + return None if isinstance(value, float) and math.isnan(value) else value + + +def _records(df) -> List[Dict[str, Any]]: + pdf = df.to_pandas() if hasattr(df, "to_pandas") else df + return [{k: _norm(v) for k, v in row.items()} for row in pdf.to_dict("records")] + + +def _validator_verdict(g, query) -> Dict[str, Any]: + try: + out = g.gfql_validate(query) + except GFQLValidationError as e: + return {"verdict": "diagnose", "code": e.code} + assert out["ok"] is True and out["diagnostics"] == [], out + return {"verdict": "admit", "code": None} + + +def _execution_verdict(g, query, engine) -> Dict[str, Any]: + try: + out = g.gfql(query, engine=engine) + except GFQLValidationError as e: + return {"verdict": "diagnose", "code": e.code, "typed": True} + except NotImplementedError as e: + # Honest engine-capability decline (named limitation + remedy), not a crash. + assert str(e).strip() != "", "NotImplementedError with no message is a bare crash" + return {"verdict": "diagnose", "code": None, "typed": True} + except BARE_ERRORS as e: # pragma: no cover - the #1889 defect; red at master only + return {"verdict": "bare", "error": f"{type(e).__name__}: {e}", "typed": False} + return {"verdict": "admit", "nodes": None if out._nodes is None else _records(out._nodes)} + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize( + "shape,query", + [ + ("both-none", CYPHER), + ("both-none", CHAIN), + ("nodes-only", CYPHER), + ("nodes-only", CHAIN), + ], + ids=["bothnone-cypher", "bothnone-chain", "nodesonly-cypher", "nodesonly-chain"], +) +def test_validate_execute_agreement_matrix(shape, query, engine): + """The #1889 matrix: no cell validates clean and then crashes raw.""" + g = _both_none() if shape == "both-none" else _nodes_only() + + validation = _validator_verdict(g, query) + execution = _execution_verdict(g, query, engine) + + assert execution["verdict"] != "bare", ( + f"{shape}/{engine}: execution raised a bare error ({execution.get('error')}) " + f"while the validator said {validation['verdict']}" + ) + if validation["verdict"] == "admit": + assert execution["verdict"] in ("admit", "diagnose") + else: + assert execution["verdict"] == "diagnose", ( + f"{shape}/{engine}: validator diagnosed {validation['code']} but execution admitted" + ) + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("query", [CYPHER, CHAIN], ids=["cypher", "chain"]) +def test_both_none_diagnoses_typed_on_both_surfaces(query, engine): + """Master red: validator returned ok:True; execution raised bare ValueError/AssertionError.""" + g = _both_none() + + with pytest.raises(GFQLValidationError) as validate_exc: + g.gfql_validate(query) + assert validate_exc.value.code == ErrorCode.E305 + assert validate_exc.value.context.get("suggestion") + + with pytest.raises(GFQLValidationError) as exec_exc: + g.gfql(query, engine=engine) + assert exec_exc.value.code == ErrorCode.E305 + # Same shape, same verdict, same words on both surfaces. + assert exec_exc.value.message == validate_exc.value.message + + +@pytest.mark.parametrize("engine", ENGINES) +def test_edge_pattern_without_edges_diagnoses_on_both_surfaces(engine): + """Execution already declined typed (#1942); the validator now says the same thing.""" + g = _nodes_only() + + for query in ([n(), e_forward(), n()], "MATCH (a)-[r]->(b) RETURN a"): + with pytest.raises(GFQLValidationError) as validate_exc: + g.gfql_validate(query) + assert validate_exc.value.code == ErrorCode.E304 + + with pytest.raises(GFQLValidationError) as exec_exc: + g.gfql(query, engine=engine) + assert exec_exc.value.code == ErrorCode.E304 + + +@pytest.mark.parametrize("query,expected", [(CYPHER, [{"a.id": 0, "a.v": 10}, {"a.id": 1, "a.v": 20}]), + (CHAIN, [{"id": 1, "v": 20}])], + ids=["cypher", "chain"]) +def test_nodes_only_admits_on_both_surfaces_with_values(query, expected): + """Anti-vacuity: the shape the validator MAY admit must really answer, with these values.""" + g = _nodes_only() + + assert g.gfql_validate(query) == { + "ok": True, + "query_type": "chain", + "language": "cypher" if isinstance(query, str) else "gfql", + "diagnostics": [], + **({"compiled_kind": "query"} if isinstance(query, str) else {}), + } + assert _records(g.gfql(query, engine="pandas")._nodes) == expected + + +def test_graph_with_data_still_validates_and_executes_clean(): + """Anti-vacuity: the shape guard must not fire on an ordinary bound graph.""" + g = ( + graphistry + .nodes(pd.DataFrame({"id": [0, 1], "v": [10, 20]}), "id") + .edges(pd.DataFrame({"s": [0], "d": [1]}), "s", "d") + ) + + assert g.gfql_validate([n(), e_forward(), n()])["ok"] is True + assert _records(g.gfql([n(), e_forward(), n()], engine="pandas")._nodes) == [ + {"id": 0, "v": 10}, {"id": 1, "v": 20} + ] + + +@pytest.mark.parametrize("query", [CYPHER, CHAIN], ids=["cypher", "chain"]) +def test_schema_false_skips_the_shape_guard_for_unbound_graphs(query): + """Remote preflight validates with schema=False against a graph whose frames live server-side.""" + assert _both_none().gfql_validate(query, schema=False)["ok"] is True + + +def test_edges_only_graph_is_not_flagged_by_the_shape_guard(): + """Nodes are synthesizable from edges, so an edges-only graph stays answerable.""" + g = graphistry.edges(pd.DataFrame({"s": [0], "d": [1]}), "s", "d") + + assert g.gfql_validate([n()])["ok"] is True + assert _records(g.gfql([n()], engine="pandas")._nodes) == [{"id": 0}, {"id": 1}] diff --git a/graphistry/tests/compute/gfql/test_whole_entity_projection_bag_1994.py b/graphistry/tests/compute/gfql/test_whole_entity_projection_bag_1994.py new file mode 100644 index 0000000000..9a5d900e9b --- /dev/null +++ b/graphistry/tests/compute/gfql/test_whole_entity_projection_bag_1994.py @@ -0,0 +1,159 @@ +"""Whole-entity endpoint projection preserves relationship-match bags.""" +from __future__ import annotations + +import typing + +import pandas as pd +import pytest +from typing_extensions import Literal + +import graphistry +from graphistry.Engine import Engine, df_to_engine +from graphistry.Plottable import Plottable +from graphistry.tests.compute.gfql.polars_test_utils import engine_skip_reason, to_pandas_any + +_GFQLEngine = Literal["pandas", "polars", "cudf", "polars-gpu"] +_ENGINES: typing.Tuple[_GFQLEngine, ...] = ("pandas", "polars", "cudf", "polars-gpu") +_VARIABLE_LENGTH_ENGINES: typing.Tuple[_GFQLEngine, ...] = ("pandas", "polars") + +_NODES = pd.DataFrame( + {"id": [1, 2, 3, 4, 5], "name": ["Ann", "Bob", "Cat", "Dan", "Eve"]} +) +_EDGES = pd.DataFrame({"s": [1, 1, 2, 3], "d": [2, 3, 3, 4]}) +_PARALLEL_NODES = pd.DataFrame( + {"id": [1, 2, 3], "name": ["Ann", "Bob", "Cat"]} +) +_PARALLEL_EDGES = pd.DataFrame({"s": [1, 1, 2], "d": [2, 2, 3]}) + + +def _bind(nodes: pd.DataFrame, edges: pd.DataFrame, engine: _GFQLEngine) -> Plottable: + resolved_engine = Engine(engine) + return graphistry.nodes(df_to_engine(nodes, resolved_engine), "id").edges( + df_to_engine(edges, resolved_engine), "s", "d" + ) + + +def _smoke(engine: _GFQLEngine) -> Plottable: + return _bind(_NODES.iloc[:2], _EDGES.iloc[:1], engine).gfql( + "MATCH (n) RETURN n.id AS id", engine=engine + ) + +def _require_engine(engine: _GFQLEngine) -> None: + skip_reason = engine_skip_reason(engine, lambda: _smoke(engine)) + if skip_reason is not None: + pytest.skip(skip_reason) + + + +def _run(query: str, engine: _GFQLEngine, *, parallel: bool = False) -> pd.DataFrame: + _require_engine(engine) + graph = ( + _bind(_PARALLEL_NODES, _PARALLEL_EDGES, engine) + if parallel + else _bind(_NODES, _EDGES, engine) + ) + result_frame = graph.gfql(query, engine=engine)._nodes + pandas_frame = to_pandas_any(result_frame) + assert isinstance(pandas_frame, pd.DataFrame) + return pandas_frame.reset_index(drop=True) + + +def _bag(df: pd.DataFrame, column: str) -> typing.List[typing.Optional[int]]: + numeric_values = pd.to_numeric(df[column], errors="coerce") + values = [ + None if pd.isna(value) else int(value) + for value in numeric_values.tolist() + ] + return sorted(values, key=lambda value: (value is None, value)) + + +@pytest.mark.parametrize("engine", _ENGINES) +@pytest.mark.parametrize("query,column,expected", [ + ("MATCH (a)-->(b) RETURN b", "b.id", [2, 3, 3, 4]), + ("MATCH (a)-->(b) RETURN a", "a.id", [1, 1, 2, 3]), + ("MATCH (a)-->(b) RETURN b AS n", "n.id", [2, 3, 3, 4]), + ("MATCH (a)-->(b)-->(c) RETURN c", "c.id", [3, 4, 4]), +], ids=["dst", "src", "aliased", "two_hop_dst"]) +def test_whole_entity_endpoint_projection_keeps_bag(query: str, column: str, expected: typing.Sequence[int], engine: _GFQLEngine) -> None: + assert _bag(_run(query, engine), column) == expected + + +@pytest.mark.parametrize("engine", _ENGINES) +@pytest.mark.parametrize("query,column,expected", [ + ("MATCH (a)-->(b) RETURN b", "b.id", [2, 2, 3]), + ("MATCH (a)-->(b) RETURN a", "a.id", [1, 1, 2]), +], ids=["dst", "src"]) +def test_whole_entity_projection_counts_parallel_edges(query: str, column: str, expected: typing.Sequence[int], engine: _GFQLEngine) -> None: + assert _bag(_run(query, engine, parallel=True), column) == expected + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_whole_entity_projection_with_sibling_property_output(engine: _GFQLEngine) -> None: + df = _run("MATCH (a)-->(b) RETURN b, b.id AS x", engine) + assert _bag(df, "b.id") == [2, 3, 3, 4] + assert _bag(df, "x") == [2, 3, 3, 4] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_multi_alias_whole_entity_projection_renders(engine: _GFQLEngine) -> None: + df = _run("MATCH (a)-->(b) RETURN a, b", engine) + got = sorted((int(r["a.id"]), int(r["b.id"])) for r in df.to_dict("records")) + assert got == [(1, 2), (1, 3), (2, 3), (3, 4)] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_whole_entity_projection_carries_every_field(engine: _GFQLEngine) -> None: + df = _run("MATCH (a)-->(b) RETURN b", engine) + got = sorted((int(r["b.id"]), str(r["b.name"])) for r in df.to_dict("records")) + assert got == [(2, "Bob"), (3, "Cat"), (3, "Cat"), (4, "Dan")] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_whole_entity_projection_ordered_bag(engine: _GFQLEngine) -> None: + df = _run("MATCH (a)-->(b) RETURN b ORDER BY b.id", engine) + assert [int(v) for v in df["b.id"]] == [2, 3, 3, 4] + + + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_distinct_whole_entity_still_dedupes(engine: _GFQLEngine) -> None: + assert _bag(_run("MATCH (a)-->(b) RETURN DISTINCT b", engine), "b.id") == [2, 3, 4] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_whole_entity_projection_without_relationship_unchanged(engine: _GFQLEngine) -> None: + assert _bag(_run("MATCH (a) RETURN a", engine), "a.id") == [1, 2, 3, 4, 5] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_whole_entity_projection_after_where(engine: _GFQLEngine) -> None: + assert _bag(_run("MATCH (a)-->(b) WHERE b.id >= 3 RETURN a", engine), "a.id") == [1, 2, 3] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_property_projection_bag_unchanged(engine: _GFQLEngine) -> None: + assert _bag(_run("MATCH (a)-->(b) RETURN b.id AS x", engine), "x") == [2, 3, 3, 4] + + +@pytest.mark.parametrize("engine", _VARIABLE_LENGTH_ENGINES) +def test_variable_length_whole_entity_projection_unchanged(engine: _GFQLEngine) -> None: + edges = pd.DataFrame({"s": ["p0", "p1", "p2", "p1"], "d": ["p1", "p2", "p4", "p0"]}) + _require_engine(engine) + resolved_engine = Engine(engine) + graph = graphistry.edges(df_to_engine(edges, resolved_engine), "s", "d").materialize_nodes( + engine=engine + ) + result_frame = graph.gfql("MATCH (a {id: 'p0'})-[*1..2]-(b) RETURN b", engine=engine)._nodes + pandas_frame = to_pandas_any(result_frame) + assert isinstance(pandas_frame, pd.DataFrame) + assert sorted(pandas_frame["b.id"].tolist()) == ["p1", "p2"] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_whole_entity_carry_into_reentry_unchanged(engine: _GFQLEngine) -> None: + df = _run( + "MATCH (a)-->(c) WITH a AS p OPTIONAL MATCH (p)-->(z) RETURN p.id AS pid, z.id AS zid", + engine, + ) + assert len(df) == 4 diff --git a/graphistry/tests/compute/predicates/test_str.py b/graphistry/tests/compute/predicates/test_str.py index 0faaba70ce..0a577966e6 100644 --- a/graphistry/tests/compute/predicates/test_str.py +++ b/graphistry/tests/compute/predicates/test_str.py @@ -964,3 +964,110 @@ def test_string_predicate_on_string_and_mixed_unchanged(): r = Contains("a", regex=False)(pd.Series(["a1", 2, None], dtype="object")) assert r.iloc[0] == True # noqa: E712 assert pd.isna(r.iloc[1]) and pd.isna(r.iloc[2]) + + +# --- Regression: a CATEGORICAL-of-strings column is string-VALUED on every engine, but only +# pandas lends it a `.str` accessor. cuDF raises on `.str`, which routed the whole column into +# the non-string (null/False) result: pandas answered rows, cuDF answered NOTHING, silently. --- +class _NoStrAccessorCategorical: + """cuDF-shaped categorical stand-in: string categories, no working ``.str``. + + Lets the CPU lane pin the decode contract that only a GPU can otherwise exercise. + """ + + def __init__(self, values): + self._s = pd.Series(values, dtype='object') + self.dtype = pd.CategoricalDtype(pd.Index(['ab', 'cd'], dtype='object')) + + @property + def str(self): + raise AttributeError("Can only use .str accessor with string values") + + def astype(self, _t): + return self._s + + +class _NoStrAccessorFloatCategorical(_NoStrAccessorCategorical): + def __init__(self, values): + super().__init__(values) + self.dtype = pd.CategoricalDtype(pd.Index([1.5, 2.5], dtype='float64')) + + +def test_str_ops_series_passes_through_usable_accessor(): + from graphistry.compute.predicates.str import _str_ops_series + s = pd.Series(['a', 'b']) + assert _str_ops_series(s) is s + + +def test_str_ops_series_decodes_string_categories_without_accessor(): + from graphistry.compute.predicates.str import _str_ops_series + out = _str_ops_series(_NoStrAccessorCategorical(['ab', None, 'cd'])) + assert out is not None + assert list(out) == ['ab', None, 'cd'] + + +def test_str_ops_series_declines_non_string_categories(): + """Stringifying a numeric categorical would render engine-divergently, so it stays a + non-string column (null result) rather than becoming a wrong match.""" + from graphistry.compute.predicates.str import _str_ops_series + assert _str_ops_series(_NoStrAccessorFloatCategorical([1.5, 2.5])) is None + assert _str_ops_series(pd.Series([1, 2, 3])) is None + + +def test_pandas_categorical_string_predicates_match_values(): + """The pandas oracle every non-pandas engine is held to.""" + s = pd.Series(pd.Categorical(['Xa', None, 'yb', 'XC'])) + assert list(Contains('x', case=False, regex=False, na=False)(s)) == [True, False, False, True] + assert list(Startswith('X', na=False)(s)) == [True, False, False, True] + assert list(Match('^X', na=False)(s)) == [True, False, False, True] + + +@requires_cudf +class TestCudfCategoricalStringPredicateParity: + """Every string predicate on a cuDF categorical-of-str must equal the pandas answer.""" + + @staticmethod + def _pair(values): + import cudf + p = pd.Series(pd.Categorical(values)) + return p, cudf.from_pandas(p) + + @pytest.mark.parametrize("pred", [ + Contains('x', case=False, regex=False, na=False), + Contains('X', case=True, regex=False, na=False), + Startswith('X', na=False), + Endswith('a', na=False), + Match('^X', na=False), + Fullmatch('Xa', na=False), + ]) + def test_matches_pandas(self, pred): + p, c = self._pair(['Xa', None, 'yb', 'XC', 'zz']) + assert list(pred(c).to_pandas()) == list(pred(p)), f"{pred!r} diverged on cuDF" + + def test_callable_predicate_matches_pandas(self): + """``isalpha`` & co. take an unguarded ``.str``: on a cuDF categorical that was a raw + AttributeError where pandas answered.""" + from graphistry.compute.predicates.str import IsAlpha, IsUpper + p, c = self._pair(['Xa', None, 'yb', 'XC', 'zz']) + for pred in [IsAlpha(), IsUpper()]: + got = list(pred(c).to_pandas().fillna(-1)) + want = list(pd.Series(pred(p)).fillna(-1)) + assert got == want, f"{pred!r} diverged on cuDF" + + def test_searchany_explicit_categorical_column_matches_pandas(self): + from graphistry.compute.gfql.search_any import search_any_mask + import cudf + pdf = pd.DataFrame({'id': list(range(5)), + 'cat': pd.Categorical(['Xa', None, 'yb', 'XC', 'zz'])}) + cdf = cudf.from_pandas(pdf) + want = list(search_any_mask(pdf, 'x', columns=['cat'])) + assert want == [True, False, False, True, False], "pandas oracle drift" + assert list(search_any_mask(cdf, 'x', columns=['cat']).to_pandas()) == want + + def test_searchany_explicit_float_categorical_still_declines(self): + """Numeric-categorical stringification stays an honest NIE, not a silent divergence.""" + from graphistry.compute.gfql.search_any import search_any_mask + import cudf + cdf = cudf.from_pandas(pd.DataFrame({'catf': pd.Categorical([1.5, 2.5, 1.5])})) + with pytest.raises(NotImplementedError): + search_any_mask(cdf, '1', columns=['catf']) diff --git a/graphistry/tests/compute/test_chain_remote_v2.py b/graphistry/tests/compute/test_chain_remote_v2.py index b392f9b3ea..ea7c6379d4 100644 --- a/graphistry/tests/compute/test_chain_remote_v2.py +++ b/graphistry/tests/compute/test_chain_remote_v2.py @@ -302,5 +302,5 @@ def test_validate_true_uses_remote_safe_local_preflight(self) -> None: ) kwargs = mock_validate.call_args.kwargs - assert kwargs["strict"] is False + assert kwargs["strict"] == "warn" assert kwargs["schema"] is False diff --git a/graphistry/tests/compute/test_chain_schema_validation.py b/graphistry/tests/compute/test_chain_schema_validation.py index 7239a97ad2..23d444050e 100644 --- a/graphistry/tests/compute/test_chain_schema_validation.py +++ b/graphistry/tests/compute/test_chain_schema_validation.py @@ -41,23 +41,23 @@ def test_valid_schema_operations(self): assert len(result._nodes) > 0 def test_nonexistent_node_column(self): - """Reference to non-existent node column fails.""" + """Reference to non-existent node column fails under strict.""" with pytest.raises(GFQLSchemaError) as exc_info: self.g.gfql([ n({'missing_column': 'value'}) - ]) + ], strict=True) assert exc_info.value.code == ErrorCode.E301 assert 'missing_column' in str(exc_info.value) assert 'does not exist' in str(exc_info.value) def test_nonexistent_edge_column(self): - """Reference to non-existent edge column fails.""" + """Reference to non-existent edge column fails under strict.""" with pytest.raises(GFQLSchemaError) as exc_info: self.g.gfql([ n(), e_forward({'missing_edge_col': 'value'}) - ]) + ], strict=True) assert exc_info.value.code == ErrorCode.E301 assert 'missing_edge_col' in str(exc_info.value) @@ -83,11 +83,14 @@ def test_empty_graph_warning(self): result = empty_g.gfql([n()]) assert len(result._nodes) == 0 - # But filtering on non-existent column should still fail + # But filtering on non-existent column should still fail under strict with pytest.raises(GFQLSchemaError) as exc_info: - empty_g.gfql([n({'any_col': 'value'})]) - + empty_g.gfql([n({'any_col': 'value'})], strict=True) + assert exc_info.value.code == ErrorCode.E301 + + # and resolve to null -- so match nothing -- at the warn default + assert len(empty_g.gfql([n({'any_col': 'value'})])._nodes) == 0 def test_collect_all_schema_errors(self): """Can collect multiple schema errors.""" diff --git a/graphistry/tests/compute/test_filter_dict_none_serialization.py b/graphistry/tests/compute/test_filter_dict_none_serialization.py new file mode 100644 index 0000000000..2e5749a8bf --- /dev/null +++ b/graphistry/tests/compute/test_filter_dict_none_serialization.py @@ -0,0 +1,142 @@ +"""Pins for #1954: a ``None`` filter value must survive GFQL serialization. + +``_filter_dict_to_json`` used to drop every entry whose value was ``None``. A filter the +local engine evaluates as "matches nothing" therefore serialized to *no filter at all*, +i.e. "matches everything" -- so ``chain_remote``/``gfql_remote``, saved query JSON, and any +``to_json``/``from_json`` round trip silently returned the whole graph where the in-process +call returned the empty graph. + +The oracle here is the in-process answer: whatever ``filter_dict={'x': None}`` means locally, +the wire form must mean the same thing. Assertions are on row counts and id lists rather than +``to_dict('records')`` so that the pandas 3.13 ``None``/``nan`` cell rendering split cannot +make them vacuous. +""" +from typing import Any, Dict, List, Sequence, Tuple + +import json +import pandas as pd +import pytest + +import graphistry +from graphistry.Plottable import Plottable +from graphistry.compute.ast import ASTEdge, ASTNode, ASTObject, e, n +from graphistry.compute.chain import Chain + + +def _graph_with_nulls() -> Plottable: + nodes = pd.DataFrame({'id': [0, 1, 2], 'x': [None, 'a', 'b']}) + edges = pd.DataFrame({'s': [0, 1], 'd': [1, 2], 'w': [None, 'k']}) + return graphistry.edges(edges, 's', 'd').nodes(nodes, 'id') + + +def _ids(g: Plottable) -> List[int]: + return sorted(g._nodes['id'].tolist()) + + +def _edge_pairs(g: Plottable) -> List[Tuple[int, int]]: + return sorted(zip(g._edges['s'].tolist(), g._edges['d'].tolist())) + + +def test_node_filter_dict_none_value_is_serialized() -> None: + assert n({'x': None}).to_json()['filter_dict'] == {'x': None} + + +def test_node_filter_dict_none_value_among_non_null_is_serialized() -> None: + assert n({'x': None, 'y': 'a'}).to_json()['filter_dict'] == {'x': None, 'y': 'a'} + + +@pytest.mark.parametrize('key', ['edge_match', 'source_node_match', 'destination_node_match']) +def test_edge_match_family_none_value_is_serialized(key: str) -> None: + op = e(**{key: {'w': None}}) + + assert op.to_json()[key] == {'w': None} + + +def test_node_filter_dict_none_value_round_trips_through_json_text() -> None: + op = n({'x': None}) + + revived = ASTNode.from_json(json.loads(json.dumps(op.to_json()))) + + assert revived.filter_dict == {'x': None} + + +@pytest.mark.parametrize('key', ['edge_match', 'source_node_match', 'destination_node_match']) +def test_edge_match_family_none_value_round_trips_through_json_text(key: str) -> None: + op = e(**{key: {'w': None}}) + + revived = ASTEdge.from_json(json.loads(json.dumps(op.to_json()))) + + assert getattr(revived, key) == {'w': None} + + +_MATCH_ANYTHING_WOULD_RETURN_ALL_NODES = 3 + + +@pytest.mark.parametrize( + 'label,ops', + [ + ('node', [n({'x': None})]), + ('node_mixed', [n({'x': None, 'y': 'a'})]), + ('edge_match', [e(edge_match={'w': None})]), + ('source_node_match', [e(source_node_match={'x': None})]), + ('destination_node_match', [e(destination_node_match={'x': None})]), + ], +) +def test_chain_with_none_filter_value_is_round_trip_equivalent(label: str, ops: Sequence[ASTObject]) -> None: + g = _graph_with_nulls() + if label == 'node_mixed': + g = g.nodes(g._nodes.assign(y=['a', 'a', 'a']), 'id') + + local = g.gfql(Chain(ops)) + wire = g.gfql(Chain.from_json(json.loads(json.dumps(Chain(ops).to_json())))) + + assert _ids(wire) == _ids(local) + assert _edge_pairs(wire) == _edge_pairs(local) + assert len(local._nodes) != _MATCH_ANYTHING_WOULD_RETURN_ALL_NODES + + +@pytest.mark.parametrize( + 'label,ops', + [ + ('node', [n({'x': None})]), + ('edge_match', [e(edge_match={'w': None})]), + ('source_node_match', [e(source_node_match={'x': None})]), + ('destination_node_match', [e(destination_node_match={'x': None})]), + ], +) +def test_chain_with_none_filter_value_does_not_widen_to_whole_graph(label: str, ops: Sequence[ASTObject]) -> None: + g = _graph_with_nulls() + + wire = g.gfql(Chain.from_json(json.loads(json.dumps(Chain(ops).to_json())))) + + assert _ids(wire) == [] + assert _edge_pairs(wire) == [] + + +def test_cypher_null_param_lowers_to_a_serializable_filter() -> None: + from graphistry.compute.gfql.cypher.api import cypher_to_gfql + + g = _graph_with_nulls() + params: Dict[str, Any] = {'p': None} + compiled = cypher_to_gfql('MATCH (a {x: $p}) RETURN a', params=params) + + local = g.gfql(compiled) + wire = g.gfql(Chain.from_json(json.loads(json.dumps(compiled.to_json())))) + + assert _ids(local) == [] + assert _ids(wire) == _ids(local) + + +@pytest.mark.parametrize( + 'query', + [ + 'MATCH (a {x: null}) RETURN a.id AS i', + 'MATCH (a) WHERE a.x = null RETURN a.id AS i', + 'MATCH (a {x: null})-[r]->(b) RETURN a.id AS i', + 'MATCH (a {x: null}), (b) RETURN a.id AS i', + ], +) +def test_cypher_null_property_matches_nothing_on_every_pattern_shape(query: str) -> None: + g = _graph_with_nulls() + + assert len(g.gfql(query)._nodes) == 0 diff --git a/graphistry/tests/compute/test_gfql.py b/graphistry/tests/compute/test_gfql.py index 002d127ee1..0a676d7e58 100644 --- a/graphistry/tests/compute/test_gfql.py +++ b/graphistry/tests/compute/test_gfql.py @@ -299,14 +299,19 @@ def test_gfql_validate_false_skips_preflight(self): result = g.gfql([n()]) assert result is not None - def test_gfql_validate_true_catches_cypher_schema_errors_by_default(self): + def test_gfql_validate_true_catches_cypher_schema_errors_under_strict(self): g = _mk_people_company_graph3() with pytest.raises(GFQLValidationError) as exc_info: - g.gfql("MATCH (p:Employee) RETURN p.id AS id", validate=True) + g.gfql("MATCH (p:Employee) RETURN p.id AS id", validate=True, strict=True) assert exc_info.value.code == ErrorCode.E301 + def test_gfql_validate_true_serves_an_absent_label_by_default(self): + g = _mk_people_company_graph3() + + assert len(g.gfql("MATCH (p:Employee) RETURN p.id AS id", validate=True)._nodes) == 0 + def test_gfql_validate_true_treats_all_strings_as_cypher(self): g = _mk_people_company_graph3() diff --git a/graphistry/tests/compute/test_gfql_hypergraph.py b/graphistry/tests/compute/test_gfql_hypergraph.py index 9fd78b74e3..040a7febbf 100644 --- a/graphistry/tests/compute/test_gfql_hypergraph.py +++ b/graphistry/tests/compute/test_gfql_hypergraph.py @@ -556,7 +556,7 @@ def mock_server_hypergraph(g, chain, api_token=None, dataset_id=None, output_type='all', format=None, df_export_args=None, node_col_subset=None, edge_col_subset=None, engine=None, validate=True, persist=False, - params=None, output=None): + df_import_args=None, params=None, output=None, strict=None): """Mock server that executes hypergraph locally.""" from graphistry.compute.ast import ASTCall diff --git a/graphistry/tests/compute/test_gfql_validate_only.py b/graphistry/tests/compute/test_gfql_validate_only.py index f30c83617b..49718508b0 100644 --- a/graphistry/tests/compute/test_gfql_validate_only.py +++ b/graphistry/tests/compute/test_gfql_validate_only.py @@ -58,10 +58,16 @@ def test_gfql_validate_cypher_success(): assert report["diagnostics"] == [] -def test_gfql_validate_cypher_default_reports_schema_errors(): +def test_gfql_validate_cypher_default_is_warn_not_error(): + g = _mk_graph() + report = g.gfql_validate("MATCH (p:Employee) RETURN p.name AS name") + assert report["ok"] is True + + +def test_gfql_validate_cypher_strict_reports_schema_errors(): g = _mk_graph() with pytest.raises(GFQLValidationError) as exc_info: - g.gfql_validate("MATCH (p:Employee) RETURN p.name AS name") + g.gfql_validate("MATCH (p:Employee) RETURN p.name AS name", strict=True) assert exc_info.value.code == ErrorCode.E301 @@ -146,9 +152,17 @@ def test_gfql_validate_exception_payload_is_llm_friendly(): assert diagnostics[0]["code"] == ErrorCode.E301 -def test_gfql_validate_chain_without_bound_tables_is_structural_only(): +def test_gfql_validate_chain_without_bound_tables_diagnoses_the_unqueryable_graph(): + """Was structural-only (#1321) until execution parity: a graph with no frames answers nothing (#1889).""" + g = CGFull() + with pytest.raises(GFQLValidationError) as exc_info: + g.gfql_validate([n({"missing_col": "x"})]) + assert exc_info.value.code == ErrorCode.E305 + + +def test_gfql_validate_chain_without_bound_tables_is_structural_only_when_schema_off(): g = CGFull() - report = g.gfql_validate([n({"missing_col": "x"})]) + report = g.gfql_validate([n({"missing_col": "x"})], schema=False) assert report["ok"] is True assert report["language"] == "gfql" assert report["query_type"] == "chain" diff --git a/graphistry/tests/compute/test_let_binding_contracts.py b/graphistry/tests/compute/test_let_binding_contracts.py index 6fc6c797a4..dbcacc0221 100644 --- a/graphistry/tests/compute/test_let_binding_contracts.py +++ b/graphistry/tests/compute/test_let_binding_contracts.py @@ -292,9 +292,11 @@ def test_nested_let_cycle_through_an_enclosing_binding_is_a_coded_error(engine: @pytest.mark.parametrize("engine", ENGINES) def test_binding_schema_failure_keeps_its_gfql_error_type(engine: str) -> None: + # strict= selects the level that still rejects an absent column; warn resolves it to null with pytest.raises(GFQLSchemaError) as exc_info: _graph(engine).gfql( - ASTLet({"x": n({"nosuchcol": 1}), "y": n({})}), output="x", engine=engine + ASTLet({"x": n({"nosuchcol": 1}), "y": n({})}), output="x", engine=engine, + strict=True, ) assert exc_info.value.code == ErrorCode.E301 diff --git a/graphistry/tests/compute/test_python_remote_code_normalization.py b/graphistry/tests/compute/test_python_remote_code_normalization.py new file mode 100644 index 0000000000..f6276594ca --- /dev/null +++ b/graphistry/tests/compute/test_python_remote_code_normalization.py @@ -0,0 +1,84 @@ +"""Contract: python_remote accepts the code forms its own documentation prescribes. + +``code`` may arrive as a callable or as a source string. Both are normalized to a +top-level ``def task`` source string before validation, so neither the function's name +nor the indentation of the literal it was written in can decide whether the call works. +""" + +import pytest + +from graphistry.compute.python_remote import normalize_task_code, validate_python_str + + +def task(g): + return {"n": len(g._edges)} + + +def helper(g): + return {"n": len(g._edges)} + + +def test_callable_named_task_normalizes_to_a_string() -> None: + out = normalize_task_code(task) + assert isinstance(out, str) + assert validate_python_str(out) is True + + +def test_callable_named_other_than_task_normalizes_to_a_string() -> None: + out = normalize_task_code(helper) + assert isinstance(out, str) + assert "def task(g)" in out + assert validate_python_str(out) is True + + +def test_callable_named_task_and_renamed_callable_agree() -> None: + assert normalize_task_code(task) == normalize_task_code(helper) + + +def test_nested_callable_source_is_dedented() -> None: + def outer(): + def task(g): + return {"n": len(g._edges)} + return task + + out = normalize_task_code(outer()) + assert not out.startswith(" ") + assert validate_python_str(out) is True + + +def test_indented_source_literal_is_accepted() -> None: + code = """ + from typing import Any, Dict + + def task(g): + return {'num_edges': len(g._edges)} + """ + assert validate_python_str(normalize_task_code(code)) is True + + +def test_unindented_source_literal_is_unchanged() -> None: + code = "def task(g):\n return {'n': 1}\n" + assert normalize_task_code(code) == code + assert validate_python_str(normalize_task_code(code)) is True + + +def test_relative_indentation_inside_the_body_survives_dedent() -> None: + code = """ + def task(g): + if g is None: + return {'n': 0} + return {'n': 1} + """ + out = normalize_task_code(code) + assert " return {'n': 0}" in out + assert validate_python_str(out) is True + + +def test_missing_task_function_still_declines() -> None: + with pytest.raises(ValueError, match="No top-level function 'task'"): + validate_python_str(normalize_task_code("def other(g):\n return {}\n")) + + +def test_task_with_wrong_arity_still_declines() -> None: + with pytest.raises(ValueError, match="exactly one parameter"): + validate_python_str(normalize_task_code("def task(g, extra):\n return {}\n")) diff --git a/graphistry/tests/compute/test_remote_csv_fidelity.py b/graphistry/tests/compute/test_remote_csv_fidelity.py new file mode 100644 index 0000000000..5a95f206ca --- /dev/null +++ b/graphistry/tests/compute/test_remote_csv_fidelity.py @@ -0,0 +1,393 @@ +"""Remote result decoding must preserve server-side values or decline loudly.""" +from io import BytesIO +from unittest.mock import MagicMock, patch +import os +import zipfile + +import pandas as pd +import pytest + +import graphistry +from graphistry.compute.ast import n + + +skip_gpu = pytest.mark.skipif( + not ("TEST_CUDF" in os.environ and os.environ["TEST_CUDF"] == "1"), + reason="cudf tests need TEST_CUDF=1" +) + + +# Server-side truth: leading-zero ids, pandas NA-vocabulary strings, int64 beyond float53 +NODES = pd.DataFrame({ + 'id': ['007', '08', 'NA'], + 'name': ['', 'null', 'x'], + 'big': [4611686018427387904, 4611686018427387905, 3], +}) +EDGES = pd.DataFrame({'s': ['007'], 'd': ['08']}) + +FAITHFUL_ARGS = { + 'dtype': {'id': str, 'name': str, 's': str, 'd': str}, + 'keep_default_na': False, + 'na_values': [], +} + + +def build_zip(fmt: str) -> bytes: + buf = BytesIO() + with zipfile.ZipFile(buf, 'w') as z: + if fmt == 'csv': + z.writestr('nodes.csv', NODES.to_csv(index=False)) + z.writestr('edges.csv', EDGES.to_csv(index=False)) + else: + nb = BytesIO() + NODES.to_parquet(nb, index=False) + eb = BytesIO() + EDGES.to_parquet(eb, index=False) + z.writestr('nodes.parquet', nb.getvalue()) + z.writestr('edges.parquet', eb.getvalue()) + return buf.getvalue() + + +def build_table(fmt: str) -> bytes: + if fmt == 'csv': + return NODES.to_csv(index=False).encode('utf-8') + buf = BytesIO() + NODES.to_parquet(buf, index=False) + return buf.getvalue() + + +def mock_response(content: bytes) -> MagicMock: + resp = MagicMock() + resp.ok = True + resp.content = content + resp.headers = {} + resp.raise_for_status.return_value = None + return resp + + +def bound_graph(): + g = graphistry.edges(EDGES, 's', 'd').nodes(NODES, 'id') + g._dataset_id = 'ds_test' + return g + + +def lossy_warning(rec) -> str: + hits = [str(w.message) for w in rec if 'df_import_args' in str(w.message)] + assert len(hits) == 1, [str(w.message) for w in rec] + return hits[0] + + +def norm(df: pd.DataFrame): + return [ + {k: (None if isinstance(v, float) and v != v else v) for k, v in rec.items()} + for rec in df.to_dict('records') + ] + + +class TestGfqlRemoteCsvFidelity: + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_warns_and_serves_when_no_import_args(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + with pytest.warns(UserWarning) as rec: + out = bound_graph().gfql_remote([n()], format='csv', api_token='t') + msg = str(rec[0].message) + assert 'df_import_args' in msg + assert 'parquet' in msg + assert mock_post.called + assert out._nodes is not None + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_warns_and_serves_for_nodes_output_type(self, mock_post): + mock_post.return_value = mock_response(build_table('csv')) + with pytest.warns(UserWarning): + out = bound_graph().gfql_remote([n()], output_type='nodes', format='csv', api_token='t') + assert mock_post.called + assert out._nodes is not None + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_shape_csv_warns_and_serves(self, mock_post): + mock_post.return_value = mock_response(build_table('csv')) + with pytest.warns(UserWarning): + out = bound_graph().gfql_remote_shape([n()], format='csv', api_token='t') + assert mock_post.called + assert out is not None + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_warns_and_serves_when_import_args_govern_nothing(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + with pytest.warns(UserWarning) as rec: + out = bound_graph().gfql_remote( + [n()], format='csv', api_token='t', df_import_args={'sep': ','} + ) + assert 'dtype inference' in lossy_warning(rec) + assert mock_post.called + assert pd.api.types.is_numeric_dtype(out._nodes['id']) + assert out._nodes['name'].isna().sum() == 2 + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_warns_and_serves_on_empty_import_args(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + with pytest.warns(UserWarning) as rec: + out = bound_graph().gfql_remote( + [n()], format='csv', api_token='t', df_import_args={} + ) + assert 'NA substitution' in lossy_warning(rec) + assert mock_post.called + assert list(out._nodes['id'])[:2] == [7.0, 8.0] + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_rejects_non_dict_import_args(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + with pytest.raises(ValueError): + bound_graph().gfql_remote( + [n()], format='csv', api_token='t', + df_import_args='dtype=str', # type: ignore[arg-type] + ) + assert not mock_post.called + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_opt_in_preserves_string_ids(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + out = bound_graph().gfql_remote( + [n()], format='csv', api_token='t', df_import_args=FAITHFUL_ARGS + ) + assert not pd.api.types.is_numeric_dtype(out._nodes['id']) + assert list(out._nodes['id']) == ['007', '08', 'NA'] + assert list(out._nodes['name']) == ['', 'null', 'x'] + assert list(out._nodes['big']) == [4611686018427387904, 4611686018427387905, 3] + assert out._nodes['name'].isna().sum() == 0 + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_opt_in_keeps_node_edge_join_coherent(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + out = bound_graph().gfql_remote( + [n()], format='csv', api_token='t', df_import_args=FAITHFUL_ARGS + ) + assert str(out._nodes['id'].dtype) == str(out._edges['s'].dtype) + assert set(out._edges['s']).issubset(set(out._nodes['id'])) + assert set(out._edges['d']).issubset(set(out._nodes['id'])) + joined = out._edges.merge(out._nodes, left_on='s', right_on='id', how='inner') + assert len(joined) == len(out._edges) + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_opt_in_matches_parquet_values(self, mock_post): + mock_post.return_value = mock_response(build_zip('parquet')) + ref = bound_graph().gfql_remote([n()], format='parquet', api_token='t') + + mock_post.return_value = mock_response(build_zip('csv')) + out = bound_graph().gfql_remote( + [n()], format='csv', api_token='t', df_import_args=FAITHFUL_ARGS + ) + assert norm(out._nodes) == norm(ref._nodes) + assert norm(out._edges) == norm(ref._edges) + + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_parquet_needs_no_import_args(self, mock_post): + mock_post.return_value = mock_response(build_zip('parquet')) + out = bound_graph().gfql_remote([n()], format='parquet', api_token='t') + assert list(out._nodes['id']) == ['007', '08', 'NA'] + + @skip_gpu + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_warns_and_serves_on_cudf_graph(self, mock_post): + import cudf + mock_post.return_value = mock_response(build_zip('csv')) + g = graphistry.edges(cudf.from_pandas(EDGES), 's', 'd').nodes(cudf.from_pandas(NODES), 'id') + g._dataset_id = 'ds_test' + with pytest.warns(UserWarning) as rec: + out = g.gfql_remote([n()], format='csv', api_token='t') + msg = lossy_warning(rec) + assert 'parquet' in msg + assert mock_post.called + assert out._nodes is not None + assert len(out._nodes) == len(NODES) + + @skip_gpu + @patch('graphistry.compute.chain_remote.requests.post') + def test_gfql_remote_csv_opt_in_preserves_string_ids_on_cudf_graph(self, mock_post): + import cudf + mock_post.return_value = mock_response(build_zip('csv')) + g = graphistry.edges(cudf.from_pandas(EDGES), 's', 'd').nodes(cudf.from_pandas(NODES), 'id') + g._dataset_id = 'ds_test' + out = g.gfql_remote( + [n()], format='csv', api_token='t', + df_import_args={'dtype': {'id': 'str', 'name': 'str', 's': 'str', 'd': 'str'}, + 'keep_default_na': False, 'na_values': []}, + ) + assert list(out._nodes['id'].to_pandas()) == ['007', '08', 'NA'] + assert str(out._nodes['id'].dtype) == str(out._edges['s'].dtype) + + +class TestPythonRemoteCsvFidelity: + + @patch('graphistry.compute.python_remote.requests.post') + def test_python_remote_table_csv_warns_and_serves(self, mock_post): + mock_post.return_value = mock_response(build_table('csv')) + code = "def task(g):\n return g._nodes\n" + with pytest.warns(UserWarning) as rec: + out = bound_graph().python_remote_table(code, format='csv', api_token='t') + assert 'df_import_args' in str(rec[0].message) + assert mock_post.called + assert out is not None + + @patch('graphistry.compute.python_remote.requests.post') + def test_python_remote_g_csv_warns_and_serves(self, mock_post): + mock_post.return_value = mock_response(build_zip('csv')) + code = "def task(g):\n return g\n" + with pytest.warns(UserWarning): + out = bound_graph().python_remote_g(code, format='csv', api_token='t') + assert mock_post.called + assert out._nodes is not None + + @patch('graphistry.compute.python_remote.requests.post') + def test_python_remote_table_csv_opt_in_preserves_string_ids(self, mock_post): + mock_post.return_value = mock_response(build_table('csv')) + code = "def task(g):\n return g._nodes\n" + out = bound_graph().python_remote_table( + code, format='csv', api_token='t', df_import_args=FAITHFUL_ARGS + ) + assert list(out['id']) == ['007', '08', 'NA'] + assert list(out['name']) == ['', 'null', 'x'] + assert list(out['big']) == [4611686018427387904, 4611686018427387905, 3] + + @patch('graphistry.compute.python_remote.requests.post') + def test_python_remote_table_parquet_needs_no_import_args(self, mock_post): + mock_post.return_value = mock_response(build_table('parquet')) + code = "def task(g):\n return g._nodes\n" + out = bound_graph().python_remote_table(code, format='parquet', api_token='t') + assert list(out['id']) == ['007', '08', 'NA'] + + +def test_missing_import_args_warns_and_yields_inferring_reader() -> None: + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with pytest.warns(UserWarning) as rec: + args = resolve_csv_import_args(None, "gfql_remote") + assert args == {} + assert 'parquet' in str(rec[0].message) + + +def test_import_args_governing_both_axes_do_not_warn() -> None: + import warnings as _w + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with _w.catch_warnings(): + _w.simplefilter("error") + assert resolve_csv_import_args(FAITHFUL_ARGS, "gfql_remote") == FAITHFUL_ARGS + + +def test_converters_govern_both_axes_and_do_not_warn() -> None: + import warnings as _w + from graphistry.compute.remote_df_io import resolve_csv_import_args + + args = {'converters': {'id': str}} + with _w.catch_warnings(): + _w.simplefilter("error") + assert resolve_csv_import_args(args, "gfql_remote") == args + + +@pytest.mark.parametrize('args', [{}, {'sep': ','}, {'nrows': 10, 'engine': 'c'}]) +def test_import_args_governing_neither_axis_warn_about_both(args) -> None: + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with pytest.warns(UserWarning) as rec: + assert resolve_csv_import_args(dict(args), "gfql_remote") == args + msg = lossy_warning(rec) + assert 'dtype inference' in msg + assert 'NA substitution' in msg + assert 'parquet' in msg + + +def test_dtype_only_import_args_still_warn_about_na_substitution() -> None: + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with pytest.warns(UserWarning) as rec: + resolve_csv_import_args({'dtype': str}, "gfql_remote") + msg = lossy_warning(rec) + assert 'NA substitution' in msg + assert 'dtype inference' not in msg + + +def test_explicitly_restating_a_default_counts_as_governing_that_axis() -> None: + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with pytest.warns(UserWarning) as rec: + resolve_csv_import_args({'na_filter': True}, "gfql_remote") + msg = lossy_warning(rec) + assert 'NA substitution' not in msg + assert 'dtype inference' in msg + + +def test_na_only_import_args_still_warn_about_dtype_inference() -> None: + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with pytest.warns(UserWarning) as rec: + resolve_csv_import_args({'keep_default_na': False, 'na_values': []}, "gfql_remote") + msg = lossy_warning(rec) + assert 'dtype inference' in msg + assert 'NA substitution' not in msg + + +def test_each_warned_axis_names_a_real_rewrite_pandas_performs() -> None: + from io import StringIO + + csv = pd.DataFrame({'id': ['007', '08'], 'name': ['NA', 'x']}).to_csv(index=False) + + dtype_only = pd.read_csv(StringIO(csv), dtype=str) + assert list(dtype_only['id']) == ['007', '08'] + assert dtype_only['name'].isna().sum() == 1 + + na_only = pd.read_csv(StringIO(csv), keep_default_na=False, na_values=[]) + assert list(na_only['name']) == ['NA', 'x'] + assert list(na_only['id']) == [7, 8] + + both = pd.read_csv(StringIO(csv), dtype=str, keep_default_na=False, na_values=[]) + assert list(both['id']) == ['007', '08'] + assert list(both['name']) == ['NA', 'x'] + + +def test_non_dict_import_args_is_a_typed_gfql_error() -> None: + from graphistry.compute.exceptions import ( + ErrorCode, GFQLRemoteError, GFQLValidationError + ) + from graphistry.compute.remote_df_io import resolve_csv_import_args + + with pytest.raises(GFQLRemoteError) as excinfo: + resolve_csv_import_args("nope", "gfql_remote") # type: ignore[arg-type] + assert excinfo.value.code == ErrorCode.E403 + + # Catchable the documented GFQL way, and still as ValueError. + with pytest.raises(GFQLValidationError): + resolve_csv_import_args("nope", "gfql_remote") # type: ignore[arg-type] + with pytest.raises(ValueError): + resolve_csv_import_args("nope", "gfql_remote") # type: ignore[arg-type] + + +def test_polars_frames_decline_before_the_request() -> None: + pl = pytest.importorskip("polars") + from unittest.mock import MagicMock, patch as _patch + from graphistry.compute.exceptions import ErrorCode, GFQLRemoteError + + g = graphistry.nodes(pl.DataFrame({"id": [0, 1]}), "id").edges( + pl.DataFrame({"s": [0], "d": [1]}), "s", "d") + g._dataset_id = "ds" + resp = MagicMock() + resp.status_code = 200 + resp.content = b"id\n1\n" + resp.headers = {"content-type": "text/csv"} + + with _patch("graphistry.compute.chain_remote.requests.post", return_value=resp) as mp: + with pytest.raises(GFQLRemoteError) as excinfo: + g.gfql_remote([n()], format="parquet", api_token="t") + assert excinfo.value.code == ErrorCode.E404 + assert "polars" in str(excinfo.value).lower() + assert not mp.called + + +def test_supported_frame_library_resolves_pandas_and_none() -> None: + from graphistry.compute.remote_df_io import require_supported_frame_library + + assert require_supported_frame_library(None, None, "gfql_remote") == "pandas" + assert require_supported_frame_library( + pd.DataFrame({"id": [0]}), pd.DataFrame({"s": [0]}), "gfql_remote") == "pandas" diff --git a/graphistry/tests/compute/test_remote_engine_contract.py b/graphistry/tests/compute/test_remote_engine_contract.py new file mode 100644 index 0000000000..db1025e1f2 --- /dev/null +++ b/graphistry/tests/compute/test_remote_engine_contract.py @@ -0,0 +1,127 @@ +"""Contract tests for explicit engines on remote compute calls.""" + +import typing +from typing import Optional +from unittest.mock import MagicMock, patch +from typing_extensions import Literal + +import pandas as pd +import pytest + +from graphistry.Engine import EngineAbstractType +from graphistry.Plottable import Plottable +from graphistry.compute.ast import ASTNode +from graphistry.compute.chain import Chain +from graphistry.compute.chain_remote import chain_remote_generic +from graphistry.compute.exceptions import ErrorCode, GFQLRemoteError +from graphistry.compute.python_remote import python_remote_generic +from graphistry.compute.remote_df_io import RemoteAPIName + + +TASK = "def task(g):\n return g\n" +QUERY = Chain([ASTNode(filter_dict={"type": "Person"})]) +_PostTarget = Literal[ + "graphistry.compute.chain_remote.requests.post", + "graphistry.compute.python_remote.requests.post", +] + + + +class Posted(Exception): + """Stop a test after the request reaches the mocked transport.""" + + +def mock_plottable(dataset_id: Optional[str] = None) -> MagicMock: + """Build the minimum graph state used by both remote entry points.""" + graph = MagicMock() + graph._dataset_id = dataset_id + graph._edges = pd.DataFrame({"s": [0], "d": [1]}) + graph._nodes = pd.DataFrame({"id": [0, 1]}) + graph._privacy = None + graph._url_params = {} + graph.session.api_token = "refreshed-token" + graph.session.certificate_validation = True + graph.base_url_server.return_value = "https://test.graphistry.com" + + def upload(*, validate: bool) -> MagicMock: + graph._dataset_id = "uploaded-dataset" + return graph + + graph.upload.side_effect = upload + return graph + + +def call_remote( + api_name: RemoteAPIName, + graph: Plottable, + engine: EngineAbstractType, + *, + with_creds: bool, +) -> typing.NoReturn: + """Call one remote entry point with matching mock credentials.""" + api_token = "token" if with_creds else None + dataset_id = "dataset" if with_creds else None + if api_name == "gfql_remote": + chain_remote_generic( + graph, + QUERY, + api_token=api_token, + dataset_id=dataset_id, + engine=engine, + format="json", + validate=False, + ) + else: + python_remote_generic( + graph, + TASK, + api_token=api_token, + dataset_id=dataset_id, + engine=engine, + format="json", + output_type="json", + validate=False, + ) + raise AssertionError("remote call returned before transport") + + +@pytest.mark.parametrize( + ("api_name", "post_target"), + [ + ("gfql_remote", "graphistry.compute.chain_remote.requests.post"), + ("python_remote", "graphistry.compute.python_remote.requests.post"), + ], +) +@pytest.mark.parametrize("engine", ["pandas", "cudf"]) +def test_explicit_supported_engine_is_sent_unchanged( + api_name: RemoteAPIName, post_target: _PostTarget, engine: EngineAbstractType +) -> None: + graph = mock_plottable("dataset") + with patch(post_target, side_effect=Posted) as post: + with pytest.raises(Posted): + call_remote(api_name, graph, engine, with_creds=True) + assert post.call_args.kwargs["json"]["engine"] == engine + + +@pytest.mark.parametrize( + ("api_name", "post_target"), + [ + ("gfql_remote", "graphistry.compute.chain_remote.requests.post"), + ("python_remote", "graphistry.compute.python_remote.requests.post"), + ], +) +@pytest.mark.parametrize("engine", ["polars", "polars-gpu"]) +def test_explicit_unsupported_engine_declines_before_side_effects( + api_name: RemoteAPIName, post_target: _PostTarget, engine: EngineAbstractType +) -> None: + graph = mock_plottable() + with patch(post_target) as post: + with pytest.raises(GFQLRemoteError) as excinfo: + call_remote(api_name, graph, engine, with_creds=False) + + assert excinfo.value.code == ErrorCode.E405 + assert excinfo.value.context["field"] == "engine" + assert excinfo.value.context["value"] == engine + graph._pygraphistry.refresh.assert_not_called() + graph.upload.assert_not_called() + post.assert_not_called() diff --git a/graphistry/tests/compute/test_remote_error_surface.py b/graphistry/tests/compute/test_remote_error_surface.py new file mode 100644 index 0000000000..40151c9db7 --- /dev/null +++ b/graphistry/tests/compute/test_remote_error_surface.py @@ -0,0 +1,382 @@ +"""A user of the remote GFQL/Python APIs gets a typed GFQL error, never raw plumbing.""" +import io +import json +import zipfile +from unittest.mock import patch + +import pandas as pd +import pytest +import requests + +import graphistry +from graphistry.compute.ast import n +from graphistry.compute.chain_remote import chain_remote_generic +from graphistry.compute.exceptions import GFQLRemoteError, GFQLSchemaError, GFQLSyntaxError, GFQLTypeError +from graphistry.compute.predicates.numeric import gt +from graphistry.compute.python_remote import python_remote_generic + + +TASK = 'def task(g):\n return g\n' +CREDS = {'api_token': 'tok', 'dataset_id': 'ds-1'} + +NODES = pd.DataFrame({'id': [0, 1], 'x': ['a', 'b']}) +EDGES = pd.DataFrame({'s': [0], 'd': [1]}) + + +def resp(status: int, content: bytes, ctype: str) -> requests.Response: + r = requests.models.Response() + r.status_code = status + r._content = content + r.headers['content-type'] = ctype + r.url = 'https://t/x' + return r + + +def bound_graph(): + g = graphistry.edges(EDGES, 's', 'd').nodes(NODES, 'id') + g._dataset_id = 'ds-1' + return g + + +class Transport: + """Patches requests at Session.send so real body preparation and real + Response decoding both run.""" + + def __init__(self, response=None): + self.response = response + self.bodies = [] + + def __enter__(self): + outer = self + + def send(self, request, **kwargs): + body = request.body + if isinstance(body, (bytes, bytearray)): + body = body.decode('utf-8') + outer.bodies.append(json.loads(body)) + return outer.response + + self._patch = patch('requests.sessions.Session.send', new=send) + self._patch.start() + return self + + def __exit__(self, *exc): + self._patch.stop() + return False + + +def parquet_bytes(df: pd.DataFrame) -> bytes: + buf = io.BytesIO() + df.to_parquet(buf, index=False) + return buf.getvalue() + + +def zip_of(members) -> bytes: + buf = io.BytesIO() + with zipfile.ZipFile(buf, 'w') as z: + for name, payload in members: + z.writestr(name, payload) + return buf.getvalue() + + +def gfql(g, **kwargs): + kwargs.setdefault('format', 'json') + kwargs.setdefault('output_type', 'all') + return chain_remote_generic(g, kwargs.pop('chain', [n()]), **CREDS, **kwargs) + + +def pyrem(g, **kwargs): + kwargs.setdefault('format', 'json') + kwargs.setdefault('output_type', 'json') + return python_remote_generic(g, TASK, **CREDS, **kwargs) + + +# --- #1956 1: a JSON content-type with a non-JSON payload ------------------- + + +@pytest.mark.parametrize('call', [gfql, pyrem]) +def test_json_ctype_non_json_error_body_is_typed_not_jsondecodeerror(call): + with Transport(resp(500, b'gateway error', 'application/json')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph()) + assert not isinstance(ei.value, requests.exceptions.RequestException) + assert 'gateway error' in str(ei.value) + assert '500' in str(ei.value) + assert ei.value.context['status_code'] == 500 + + +# --- #1956 2: python_remote must not leak a raw HTTPError ------------------- + + +@pytest.mark.parametrize('call', [gfql, pyrem]) +def test_non_json_error_status_keeps_body_and_stays_typed(call): + with Transport(resp(502, b'bad gateway', 'text/html')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph()) + assert not isinstance(ei.value, requests.exceptions.RequestException) + assert 'bad gateway' in str(ei.value) + assert '502' in str(ei.value) + + +# --- #1956 3: the server's own message survives the zip handler ------------- + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_zip_path_error_body_keeps_server_message(call, kwargs): + body = json.dumps({'error': "GFQL validation failed: unknown column 'foo'"}).encode() + with Transport(resp(200, body, 'application/json')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph(), **kwargs) + assert "unknown column 'foo'" in str(ei.value) + + +# --- #1956 4: a 200 whose JSON body is an error document -------------------- + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'json', 'output_type': 'all'}), + (pyrem, {'format': 'json', 'output_type': 'all'}), +]) +def test_json_200_error_document_is_typed_not_keyerror(call, kwargs): + with Transport(resp(200, json.dumps({'error': 'boom'}).encode(), 'application/json')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph(), **kwargs) + assert 'boom' in str(ei.value) + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'json', 'output_type': 'all'}), + (pyrem, {'format': 'json', 'output_type': 'all'}), +]) +def test_json_200_missing_result_key_is_typed_not_keyerror(call, kwargs): + with Transport(resp(200, json.dumps({'nodes': []}).encode(), 'application/json')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph(), **kwargs) + assert 'edges' in str(ei.value) + + +# --- #1956 5: a zip missing an expected member ------------------------------ + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_zip_missing_member_is_typed_not_indexerror(call, kwargs): + payload = zip_of([('edges.parquet', parquet_bytes(EDGES))]) + with Transport(resp(200, payload, 'application/zip')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph(), **kwargs) + assert 'nodes' in str(ei.value) + + +# --- #1956 6: substring member selection silently bound the WRONG table ----- + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_zip_member_selection_binds_nodes_to_the_nodes_table(call, kwargs): + payload = zip_of([ + ('nodes_and_edges.parquet', parquet_bytes(EDGES)), + ('nodes.parquet', parquet_bytes(NODES)), + ('edges.parquet', parquet_bytes(EDGES)), + ]) + with Transport(resp(200, payload, 'application/zip')): + out = call(bound_graph(), **kwargs) + assert out._nodes.to_dict('records') == NODES.to_dict('records') + assert out._edges.to_dict('records') == EDGES.to_dict('records') + assert list(out._nodes.columns) == ['id', 'x'] + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_zip_ambiguous_member_is_typed_not_arbitrary(call, kwargs): + payload = zip_of([ + ('a/nodes_1.parquet', parquet_bytes(NODES)), + ('a/nodes_2.parquet', parquet_bytes(NODES)), + ('edges.parquet', parquet_bytes(EDGES)), + ]) + with Transport(resp(200, payload, 'application/zip')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph(), **kwargs) + assert 'nodes' in str(ei.value) + + +def test_compound_member_is_never_bound_to_either_table(): + from graphistry.compute.remote_response import select_zip_member + from graphistry.compute.exceptions import GFQLRemoteError as _E + for kind in ('nodes', 'edges'): + with pytest.raises(_E): + select_zip_member(['nodes_and_edges.parquet'], kind, 'api') + + +def test_compound_member_does_not_stand_in_for_a_missing_member(): + from graphistry.compute.remote_response import select_zip_member + from graphistry.compute.exceptions import GFQLRemoteError as _E + names = ['nodes_and_edges.parquet', 'edges.parquet'] + assert select_zip_member(names, 'edges', 'api') == 'edges.parquet' + with pytest.raises(_E): + select_zip_member(names, 'nodes', 'api') + + +def test_prefixed_member_names_still_resolve(): + from graphistry.compute.remote_response import select_zip_member + names = ['graph_nodes.parquet', 'graph_edges.parquet'] + assert select_zip_member(names, 'nodes', 'api') == 'graph_nodes.parquet' + assert select_zip_member(names, 'edges', 'api') == 'graph_edges.parquet' + + +def test_exact_member_still_wins_over_a_compound_decoy(): + from graphistry.compute.remote_response import select_zip_member + names = ['nodes_and_edges.parquet', 'nodes.parquet', 'edges.parquet'] + assert select_zip_member(names, 'nodes', 'api') == 'nodes.parquet' + assert select_zip_member(names, 'edges', 'api') == 'edges.parquet' + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_compound_only_zip_declines_instead_of_binding_one_table_twice(call, kwargs): + payload = zip_of([('nodes_and_edges.parquet', parquet_bytes(EDGES))]) + with Transport(resp(200, payload, 'application/zip')): + with pytest.raises(GFQLRemoteError) as ei: + call(bound_graph(), **kwargs) + assert 'nodes_and_edges.parquet' in str(ei.value) + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_prefixed_member_zip_still_round_trips(call, kwargs): + payload = zip_of([ + ('out/graph_nodes.parquet', parquet_bytes(NODES)), + ('out/graph_edges.parquet', parquet_bytes(EDGES)), + ]) + with Transport(resp(200, payload, 'application/zip')): + out = call(bound_graph(), **kwargs) + assert out._nodes.to_dict('records') == NODES.to_dict('records') + assert out._edges.to_dict('records') == EDGES.to_dict('records') + + +@pytest.mark.parametrize('call,kwargs', [ + (gfql, {'format': 'parquet', 'output_type': 'all'}), + (pyrem, {'format': 'parquet', 'output_type': 'all'}), +]) +def test_well_formed_zip_still_round_trips(call, kwargs): + payload = zip_of([ + ('nodes.parquet', parquet_bytes(NODES)), + ('edges.parquet', parquet_bytes(EDGES)), + ]) + with Transport(resp(200, payload, 'application/zip')): + out = call(bound_graph(), **kwargs) + assert out._nodes.to_dict('records') == NODES.to_dict('records') + assert out._edges.to_dict('records') == EDGES.to_dict('records') + + +# --- #1960 1: NaN/inf get the same typed decline as other non-JSON values --- + + +@pytest.mark.parametrize('flt', [[n({'x': float('nan')})], + [n({'x': float('inf')})], + [n({'x': float('-inf')})], + [n({'x': gt(float('nan'))})]]) +def test_non_finite_filter_value_is_typed_and_never_reaches_the_wire(flt): + t = Transport(resp(200, json.dumps({'nodes': [], 'edges': []}).encode(), 'application/json')) + with t: + with pytest.raises(GFQLTypeError) as ei: + gfql(bound_graph(), chain=flt) + assert not isinstance(ei.value, requests.exceptions.RequestException) + assert t.bodies == [] + + +def test_finite_filter_value_still_goes_on_the_wire(): + t = Transport(resp(200, json.dumps({'nodes': [], 'edges': []}).encode(), 'application/json')) + with t: + gfql(bound_graph(), chain=[n({'x': 1.5})]) + assert t.bodies[0]['gfql_operations'][0]['filter_dict'] == {'x': 1.5} + + +# --- #1960 2: output= is honored on a Let, declined typed on a flat chain --- + + +def test_output_on_flat_chain_is_declined_typed_not_dropped(): + t = Transport(resp(200, json.dumps({'nodes': [], 'edges': []}).encode(), 'application/json')) + with t: + with pytest.raises(GFQLSyntaxError) as ei: + gfql(bound_graph(), output='foo') + assert 'output' in str(ei.value) + assert t.bodies == [] + + +def test_output_on_let_still_reaches_the_wire(): + let = {'type': 'Let', 'bindings': {'foo': {'type': 'Chain', 'chain': [{'type': 'Node'}]}}} + t = Transport(resp(200, json.dumps({'nodes': [], 'edges': []}).encode(), 'application/json')) + with t: + gfql(bound_graph(), chain=let, output='foo') + assert t.bodies[0]['gfql_output'] == 'foo' + + +# --- #1960 3: the shape variant accepts params/output ----------------------- + + +def test_shape_variant_accepts_cypher_params(): + t = Transport(resp(200, json.dumps({'nodes': [1], 'edges': [1]}).encode(), 'application/json')) + with t: + out = bound_graph().gfql_remote_shape( + "MATCH (a) WHERE a.x > $cut RETURN a", params={'cut': 1}, **CREDS) + assert isinstance(out, pd.DataFrame) + assert t.bodies[0]['gfql_operations'] + + +# --- #1960 4: a column subset must not strand the result's own bindings ----- + + +@pytest.mark.parametrize('fmt,payload_key', [('json', 'json'), ('parquet', 'zip')]) +def test_node_col_subset_dropping_the_bound_id_is_typed(fmt, payload_key): + if payload_key == 'json': + r = resp(200, json.dumps({'nodes': [{'x': 'a'}], 'edges': [{'s': 0, 'd': 1}]}).encode(), + 'application/json') + else: + r = resp(200, zip_of([('nodes.parquet', parquet_bytes(NODES[['x']])), + ('edges.parquet', parquet_bytes(EDGES))]), 'application/zip') + with Transport(r): + with pytest.raises(GFQLSchemaError) as ei: + gfql(bound_graph(), format=fmt, node_col_subset=['x']) + assert "'id'" in str(ei.value) + + +def test_edge_col_subset_dropping_the_bound_destination_is_typed(): + r = resp(200, json.dumps({'nodes': [{'id': 0}], 'edges': [{'s': 0}]}).encode(), 'application/json') + with Transport(r): + with pytest.raises(GFQLSchemaError) as ei: + gfql(bound_graph(), edge_col_subset=['s']) + assert "'d'" in str(ei.value) + + +def test_col_subset_keeping_the_bound_columns_is_accepted(): + r = resp(200, json.dumps({'nodes': [{'id': 0, 'x': 'a'}], 'edges': [{'s': 0, 'd': 1}]}).encode(), + 'application/json') + with Transport(r): + out = gfql(bound_graph(), node_col_subset=['id', 'x'], edge_col_subset=['s', 'd']) + assert out._node == 'id' + assert list(out._nodes.columns) == ['id', 'x'] + + +def test_no_col_subset_leaves_server_supplied_bindings_alone(): + r = resp(200, json.dumps({ + 'nodes': [{'id': 0}], + 'edges': [{'src': 0, 'dst': 1}], + 'metadata': {'bindings': {'source': 'new_src', 'destination': 'new_dst'}}, + }).encode(), 'application/json') + with Transport(r): + out = gfql(bound_graph()) + assert out._source == 'new_src' diff --git a/graphistry/tests/layout/test_circle_sort_by_inert.py b/graphistry/tests/layout/test_circle_sort_by_inert.py new file mode 100644 index 0000000000..4f385e10fe --- /dev/null +++ b/graphistry/tests/layout/test_circle_sort_by_inert.py @@ -0,0 +1,103 @@ +"""Characterization pin: circle_layout's sort parameters do not affect positions. + +The docstring on ``circle_layout`` states plainly that ``sort_by`` / ``ascending`` / +``na_position`` / ``ignore_index`` have no effect on the layout. That is a consequence of +the unconditional re-sort by node id which fixes ring order before any angle is assigned. +These tests lock the documented behavior so a future change that makes ``sort_by`` real +must update the docs in the same commit. +""" + +from typing import Any, Dict, List + +import pandas as pd +import pytest + +import graphistry + + +def _graph() -> Any: + nodes = pd.DataFrame({ + 'id': ['a', 'b', 'c', 'd', 'e'], + 'k': [5, 4, 3, 2, 1], + 'grp': ['x', 'x', 'y', 'y', 'y'], + }) + edges = pd.DataFrame({'s': ['a', 'b', 'c', 'd'], 'd': ['b', 'c', 'd', 'e']}) + return graphistry.edges(edges, 's', 'd').nodes(nodes, 'id') + + +def _positions(g: Any) -> Dict[str, Any]: + nodes = g._nodes + return { + row['id']: (round(float(row['x']), 9), round(float(row['y']), 9)) + for _, row in nodes.iterrows() + } + + +_VARIANTS: List[Dict[str, Any]] = [ + {'sort_by': 'k'}, + {'sort_by': 'k', 'ascending': False}, + {'sort_by': ['k'], 'ascending': [False]}, + {'sort_by': 'k', 'na_position': 'first'}, + {'sort_by': 'k', 'ignore_index': False}, + {'sort_by': ['grp', 'k'], 'ascending': False}, +] + + +@pytest.mark.parametrize('variant', _VARIANTS) +def test_sort_params_do_not_change_positions(variant: Dict[str, Any]) -> None: + g = _graph() + baseline = _positions(g.circle_layout(bounding_box=(0, 0, 10, 10))) + got = _positions(g.circle_layout(bounding_box=(0, 0, 10, 10), **variant)) + assert got == baseline, f'{variant} changed positions; docs say it cannot' + + +# Under partition_by the discarded sort prepends the partition columns to `by`, so a +# list-valued `ascending` of the caller's own length is rejected by pandas before it is +# thrown away. Excluded here; the raising behavior is covered separately. +_PARTITIONED_VARIANTS = [v for v in _VARIANTS if not isinstance(v.get('ascending'), list)] + + +@pytest.mark.parametrize('variant', _PARTITIONED_VARIANTS) +def test_sort_params_do_not_change_positions_when_partitioned( + variant: Dict[str, Any] +) -> None: + g = _graph() + bbox = pd.DataFrame({ + 'grp': ['x', 'y'], + 'cx': [0.0, 20.0], + 'cy': [0.0, 0.0], + 'w': [10.0, 10.0], + 'h': [10.0, 10.0], + }) + kwargs: Dict[str, Any] = {'bounding_box': bbox, 'partition_by': 'grp'} + baseline = _positions(g.circle_layout(**kwargs)) + got = _positions(g.circle_layout(**kwargs, **variant)) + assert got == baseline, f'{variant} changed positions; docs say it cannot' + + +def test_ring_order_is_by_node_id() -> None: + """The documented ordering rule: position follows node id, not any sort key.""" + g = _graph() + # Reversing the node table must not move any node. + g_rev = g.nodes(g._nodes.iloc[::-1].reset_index(drop=True), 'id') + assert _positions(g.circle_layout(bounding_box=(0, 0, 10, 10))) == _positions( + g_rev.circle_layout(bounding_box=(0, 0, 10, 10)) + ) + + +def test_sort_by_none_attaches_degree_columns() -> None: + """The one documented residual effect of sort_by: degree columns on the output.""" + g = _graph() + default_cols = set(g.circle_layout(bounding_box=(0, 0, 10, 10))._nodes.columns) + sorted_cols = set( + g.circle_layout(bounding_box=(0, 0, 10, 10), sort_by='k')._nodes.columns + ) + assert {'degree', 'degree_in', 'degree_out'} <= default_cols + assert not ({'degree', 'degree_in', 'degree_out'} & sorted_cols) + + +def test_unknown_sort_by_still_raises() -> None: + """Also documented: the discarded sort still validates its column.""" + g = _graph() + with pytest.raises(KeyError): + g.circle_layout(bounding_box=(0, 0, 10, 10), sort_by='nope') diff --git a/graphistry/tests/layout/test_gib.py b/graphistry/tests/layout/test_gib.py index ab0615cddf..f67d7eda80 100644 --- a/graphistry/tests/layout/test_gib.py +++ b/graphistry/tests/layout/test_gib.py @@ -1,5 +1,7 @@ import logging, os, pandas as pd, pytest, warnings +from contextlib import contextmanager from graphistry.compute import ComputeMixin +from graphistry.Engine import Engine from graphistry.layouts import LayoutsMixin from graphistry.plotter import PlotterBase from graphistry.tests.common import NoAuthTestCase @@ -9,6 +11,54 @@ test_cudf = "TEST_CUDF" in os.environ and os.environ["TEST_CUDF"] == "1" + +# nodes 0-2: connected triangle; 3-4: connected pair; 5: singleton; 6-8: edgeless partition +MIXED_SIZES_NODES = {'id': [0, 1, 2, 3, 4, 5, 6, 7, 8], 'partition': [0, 0, 0, 1, 1, 2, 3, 3, 3]} +MIXED_SIZES_EDGES = {'s': [0, 1, 2, 3], 'd': [1, 2, 0, 4]} + + +NA_FILL_MESSAGE = 'filling layout-returned NAs as random' + + +@contextmanager +def capture_gib_layout_logs(): + """Collect partitioned_layout debug records so a pin can tell which branch positioned a node""" + messages: list = [] + + class _Collect(logging.Handler): + def emit(self, record): + messages.append(record.getMessage()) + + target = logging.getLogger('graphistry.layout.gib.partitioned_layout') + handler = _Collect() + prior_level = target.level + target.setLevel(logging.DEBUG) + target.addHandler(handler) + try: + yield messages + finally: + target.removeHandler(handler) + target.setLevel(prior_level) + + +def grid_layout(g): + """Deterministic layout callable so pins do not depend on igraph/cugraph being installed""" + n = g._nodes + idx = n[g._node].astype('float64') + return g.nodes(n.assign(x=idx % 3.0, y=idx // 3.0)) + + +def partial_layout(g): + """Layout callable leaving one x and one y unpositioned, exercising both NA-fill branches""" + n = g._nodes + idx = n[g._node].astype('float64') + out = n.assign(x=idx % 3.0, y=idx // 3.0) + ids = out[g._node] + out['x'] = out['x'].where(ids != ids.iloc[0], float('nan')) + out['y'] = out['y'].where(ids != ids.iloc[-1], float('nan')) + return g.nodes(out) + + class LG(LayoutsMixin): def __init__(self, *args, **kwargs): super().__init__() @@ -217,6 +267,159 @@ def test_circle_layout_with_partition_cudf(self): assert not result._nodes.y.isna().any(), "circle_layout produced NaN y coordinates" assert len(result._nodes) == 5 + def test_gib_node_id_multiset_preserved_on_mixed_partition_sizes(self): + lg = LGFull() + nodes = pd.DataFrame(MIXED_SIZES_NODES) + edges = pd.DataFrame(MIXED_SIZES_EDGES) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = ( + lg + .nodes(nodes, 'id') + .edges(edges, 's', 'd') + .group_in_a_box_layout(layout_alg=grid_layout) + ) + + assert sorted(g._nodes['id'].to_numpy().tolist()) == sorted(nodes['id'].tolist()) + assert int(g._nodes['id'].duplicated().sum()) == 0 + assert not g._nodes.x.isna().any() + assert not g._nodes.y.isna().any() + + def test_gib_node_id_multiset_preserved_on_edgeless_graph(self): + lg = LGFull() + nodes = pd.DataFrame({'id': [0, 1, 2, 5], 'partition': [0, 1, 2, 3]}) + edges = pd.DataFrame({'s': pd.Series([], dtype='int64'), 'd': pd.Series([], dtype='int64')}) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = ( + lg + .nodes(nodes, 'id') + .edges(edges, 's', 'd') + .group_in_a_box_layout(layout_alg=grid_layout) + ) + + assert sorted(g._nodes['id'].to_numpy().tolist()) == [0, 1, 2, 5] + assert int(g._nodes['id'].duplicated().sum()) == 0 + + def test_gib_node_id_multiset_preserved_with_default_layout(self): + pytest.importorskip('igraph') + lg = LGFull() + nodes = pd.DataFrame({'id': [0, 1, 2, 3, 4, 5, 6, 7, 8]}) + edges = pd.DataFrame({'s': [0, 1, 2, 3, 4, 5, 6], 'd': [1, 2, 0, 4, 5, 3, 7]}) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = ( + lg + .nodes(nodes, 'id') + .edges(edges, 's', 'd') + .group_in_a_box_layout() + ) + + assert sorted(g._nodes['id'].to_numpy().tolist()) == sorted(nodes['id'].tolist()) + assert int(g._nodes['id'].duplicated().sum()) == 0 + assert not g._nodes.x.isna().any() + assert not g._nodes.y.isna().any() + + def test_gib_unpositioned_nodes_filled_instead_of_asserting(self): + lg = LGFull() + # every partition has 3 connected nodes, so the small-partition branches stay out of this pin + nodes = pd.DataFrame({'id': [0, 1, 2, 3, 4, 5], 'partition': [0, 0, 0, 1, 1, 1]}) + edges = pd.DataFrame({'s': [0, 1, 2, 3, 4, 5], 'd': [1, 2, 0, 4, 5, 3]}) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = ( + lg + .nodes(nodes, 'id') + .edges(edges, 's', 'd') + .group_in_a_box_layout(layout_alg=partial_layout) + ) + + assert len(g._nodes) == 6 + assert not g._nodes.x.isna().any() + assert not g._nodes.y.isna().any() + + def test_partitioned_layout_non_bulk_positions_edgeless_partitions_on_both_axes(self): + from graphistry.layout.gib.partitioned_layout import partitioned_layout + from graphistry.layout.gib.treemap import treemap + + lg = LGFull() + # partition 0 is a connected pair, partition 1 is a 3-node edgeless group + nodes = pd.DataFrame({'id': [0, 1, 2, 3, 4], 'partition': [0, 0, 1, 1, 1]}) + edges = pd.DataFrame({'s': [0], 'd': [1]}) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = lg.nodes(nodes, 'id').edges(edges, 's', 'd') + offsets = treemap(g, x=0, y=0, w=None, h=None, partition_key='partition', engine=Engine.PANDAS) + with capture_gib_layout_logs() as messages: + out = partitioned_layout( + g, + partition_offsets=offsets, + partition_key='partition', + bulk_mode=False, + engine=Engine.PANDAS, + ) + + assert sorted(out._nodes['id'].to_numpy().tolist()) == [0, 1, 2, 3, 4] + assert int(out._nodes['id'].duplicated().sum()) == 0 + assert not out._nodes.x.isna().any() + assert not out._nodes.y.isna().any() + # the edgeless branch must position both axes itself, not lean on the NA backstop + assert [m for m in messages if 'edgeless-community' in m], messages + assert not [m for m in messages if NA_FILL_MESSAGE in m], messages + + @pytest.mark.skipif( + not ("TEST_CUDF" in os.environ and os.environ["TEST_CUDF"] == "1"), + reason="cudf tests need TEST_CUDF=1") + def test_gib_cudf_node_id_multiset_preserved_on_mixed_partition_sizes(self): + import cudf + + lg = LGFull() + nodes = cudf.DataFrame(MIXED_SIZES_NODES) + edges = cudf.DataFrame(MIXED_SIZES_EDGES) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = ( + lg + .nodes(nodes, 'id') + .edges(edges, 's', 'd') + .group_in_a_box_layout(layout_alg=grid_layout, engine='cudf') + ) + + assert isinstance(g._nodes, cudf.DataFrame) + assert sorted(g._nodes['id'].to_arrow().to_pylist()) == sorted(MIXED_SIZES_NODES['id']) + assert int(g._nodes['id'].duplicated().sum()) == 0 + assert not g._nodes.x.isna().any() + assert not g._nodes.y.isna().any() + + @pytest.mark.skipif( + not ("TEST_CUDF" in os.environ and os.environ["TEST_CUDF"] == "1"), + reason="cudf tests need TEST_CUDF=1") + def test_gib_cudf_unpositioned_nodes_filled_instead_of_asserting(self): + import cudf + + lg = LGFull() + nodes = cudf.DataFrame({'id': [0, 1, 2, 3, 4, 5], 'partition': [0, 0, 0, 1, 1, 1]}) + edges = cudf.DataFrame({'s': [0, 1, 2, 3, 4, 5], 'd': [1, 2, 0, 4, 5, 3]}) + + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=FutureWarning) + g = ( + lg + .nodes(nodes, 'id') + .edges(edges, 's', 'd') + .group_in_a_box_layout(layout_alg=partial_layout, engine='cudf') + ) + + assert len(g._nodes) == 6 + assert not g._nodes.x.isna().any() + assert not g._nodes.y.isna().any() + @pytest.mark.skipif( not ("TEST_CUDF" in os.environ and os.environ["TEST_CUDF"] == "1"), reason="cudf tests need TEST_CUDF=1") diff --git a/graphistry/tests/test_engine_frame_helpers.py b/graphistry/tests/test_engine_frame_helpers.py index 8c6fca9346..d16313ebc4 100644 --- a/graphistry/tests/test_engine_frame_helpers.py +++ b/graphistry/tests/test_engine_frame_helpers.py @@ -216,9 +216,10 @@ def test_remote_surfaces_resolve_to_supported_engine(self): @polars_only @pytest.mark.parametrize("surface", ["circle", "fa2", "chain_remote", "python_remote", "cluster"]) def test_pandas_computing_surfaces_use_input_resolver(self, surface): - """The convention itself, pinned per module: a pandas-computing surface - must not call resolve_engine (modern AUTO) -- migrating one to native - polars means deliberately flipping it back and deleting its row here.""" + """Pin pandas-computing surfaces to the input resolver, directly or via + the remote-only wrapper. They must not call modern resolve_engine; + migrating one to native polars means deliberately flipping it back and + deleting its row here.""" import importlib mod = importlib.import_module({ "circle": "graphistry.layout.circle", @@ -229,5 +230,11 @@ def test_pandas_computing_surfaces_use_input_resolver(self, surface): }[surface]) import inspect src = inspect.getsource(mod) - assert "resolve_input_engine" in src + if surface in ("chain_remote", "python_remote"): + from graphistry.compute.remote_df_io import resolve_remote_engine + + assert "resolve_remote_engine" in src + assert "resolve_input_engine" in inspect.getsource(resolve_remote_engine) + else: + assert "resolve_input_engine" in src assert "resolve_engine(" not in src.replace("resolve_input_engine(", "") diff --git a/graphistry/tests/utils/test_lazy_import.py b/graphistry/tests/utils/test_lazy_import.py new file mode 100644 index 0000000000..fd81296620 --- /dev/null +++ b/graphistry/tests/utils/test_lazy_import.py @@ -0,0 +1,125 @@ +"""Arity + surfaced-exception pins for the lazy dependency probes.""" + +import builtins +from typing import Any, Callable, List, Tuple + +import pytest + +from graphistry.utils.lazy_import import ( + assert_imported, + assert_imported_text, + lazy_import_has_min_dependancy, + lazy_sentence_transformers_import, + lazy_umap_import, +) + + +class _Boom(RuntimeError): + """A non-ModuleNotFoundError import failure (e.g. a broken/ABI-mismatched wheel).""" + + +def _raise_on(monkeypatch: pytest.MonkeyPatch, prefixes: Tuple[str, ...]) -> None: + """Make ``import ...`` raise ``_Boom`` instead of ModuleNotFoundError.""" + real_import = builtins.__import__ + + def fake_import(name: str, *args: Any, **kwargs: Any) -> Any: + if any(name == p or name.startswith(p + '.') for p in prefixes): + raise _Boom(f'broken install: {name}') + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, '__import__', fake_import) + + +def test_min_dependancy_generic_failure_returns_two_tuple( + monkeypatch: pytest.MonkeyPatch +) -> None: + _raise_on(monkeypatch, ('scipy', 'sklearn')) + out = lazy_import_has_min_dependancy() + assert isinstance(out, tuple) + # Every caller unpacks exactly two values. + assert len(out) == 2, f'arity drift: {out!r}' + ok, exn = out + assert ok is False + assert isinstance(exn, _Boom) + + +def test_assert_imported_surfaces_the_dependency_error( + monkeypatch: pytest.MonkeyPatch +) -> None: + _raise_on(monkeypatch, ('scipy', 'sklearn')) + # Must be the underlying import failure, not ValueError from a bad unpack. + with pytest.raises(_Boom): + assert_imported() + + +def test_min_dependancy_module_not_found_returns_two_tuple( + monkeypatch: pytest.MonkeyPatch +) -> None: + real_import = builtins.__import__ + + def fake_import(name: str, *args: Any, **kwargs: Any) -> Any: + if name == 'scipy' or name.startswith('scipy.'): + raise ModuleNotFoundError(f"No module named {name!r}") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, '__import__', fake_import) + out = lazy_import_has_min_dependancy() + assert len(out) == 2 + assert out[0] is False + assert isinstance(out[1], ModuleNotFoundError) + + +@pytest.mark.parametrize( + 'probe,prefixes', + [ + (lazy_umap_import, ('umap',)), + (lazy_sentence_transformers_import, ('sentence_transformers',)), + ], +) +def test_three_tuple_probes_keep_their_arity_on_generic_failure( + monkeypatch: pytest.MonkeyPatch, + probe: Callable[[], Any], + prefixes: Tuple[str, ...], +) -> None: + _raise_on(monkeypatch, prefixes) + out = probe() + assert len(out) == 3, f'arity drift: {out!r}' + assert out[0] is False + assert isinstance(out[1], _Boom) + assert out[2] is None + + +def test_assert_imported_text_surfaces_the_dependency_error( + monkeypatch: pytest.MonkeyPatch +) -> None: + _raise_on(monkeypatch, ('sentence_transformers',)) + with pytest.raises(_Boom): + assert_imported_text() + + +def test_min_dependancy_success_returns_two_tuple() -> None: + pytest.importorskip('scipy') + pytest.importorskip('sklearn') + out = lazy_import_has_min_dependancy() + assert len(out) == 2 + ok, msg = out + assert ok is True + assert msg == 'ok' + + +def test_all_return_paths_of_min_dependancy_have_equal_arity() -> None: + """Mutation guard: a new early-return with a different arity breaks callers.""" + import ast + import inspect + import textwrap + + src = textwrap.dedent(inspect.getsource(lazy_import_has_min_dependancy)) + fn = ast.parse(src).body[0] + assert isinstance(fn, ast.FunctionDef) + arities: List[int] = [ + len(node.value.elts) + for node in ast.walk(fn) + if isinstance(node, ast.Return) and isinstance(node.value, ast.Tuple) + ] + assert arities, 'expected tuple returns' + assert set(arities) == {2}, f'mixed return arity: {arities}' diff --git a/graphistry/tests/utils/test_lazy_import_warning_filters.py b/graphistry/tests/utils/test_lazy_import_warning_filters.py new file mode 100644 index 0000000000..e97842a5eb --- /dev/null +++ b/graphistry/tests/utils/test_lazy_import_warning_filters.py @@ -0,0 +1,33 @@ +"""Probing for an optional dependency must not silence the caller's warnings.""" +import os +import warnings + +import pytest + +from graphistry.utils.lazy_import import lazy_cudf_import, lazy_cuml_import + + +skip_gpu = pytest.mark.skipif( + not ("TEST_CUDF" in os.environ and os.environ["TEST_CUDF"] == "1"), + reason="cudf tests need TEST_CUDF=1" +) + + +@pytest.mark.parametrize('probe', [lazy_cudf_import, lazy_cuml_import]) +def test_gpu_lazy_import_leaves_global_warning_filters_intact(probe) -> None: + with warnings.catch_warnings(): + warnings.resetwarnings() + warnings.simplefilter("always") + before = list(warnings.filters) + probe() + assert warnings.filters == before + + +@skip_gpu +def test_cudf_probe_leaves_a_user_warning_deliverable() -> None: + with warnings.catch_warnings(record=True) as rec: + warnings.simplefilter("always") + has_cudf, _, _ = lazy_cudf_import() + assert has_cudf + warnings.warn("still audible", UserWarning) + assert [str(w.message) for w in rec] == ["still audible"] diff --git a/graphistry/text_utils.py b/graphistry/text_utils.py index 05fc46dc59..00f835f6ef 100644 --- a/graphistry/text_utils.py +++ b/graphistry/text_utils.py @@ -155,14 +155,14 @@ def search( Args: :query (str): natural language query. :cols (list or str, optional): if fuzzy=False, select which column to query. - Defaults to None since fuzzy=True by defaul. + Defaults to None since fuzzy=True by default. :thresh (float, optional): distance threshold from query vector to returned results. Defaults to 5000, set large just in case, but could be as low as 10. :fuzzy (bool, optional): if True, uses embedding + annoy index for recall, otherwise does string matching over given `cols` Defaults to True. - :top_n (int, optional): how many results to return. Defaults to 100. + :top_n (int, optional): how many results to return. Defaults to 10. Returns: **pd.DataFrame, vector_encoding_of_query:** diff --git a/graphistry/utils/json.py b/graphistry/utils/json.py index 99926e3bcc..d230cc9a34 100644 --- a/graphistry/utils/json.py +++ b/graphistry/utils/json.py @@ -1,6 +1,7 @@ import json -from typing import Any, Dict, List, Union +from math import isfinite +from typing import Any, Dict, List, Optional, Union # For mypy 0.942, we need to handle recursive types more explicitly # Using a simple base type that mypy can resolve @@ -25,6 +26,31 @@ def is_json_serializable(data): def assert_json_serializable(data): assert is_json_serializable(data), f"Data is not JSON-serializable: {data}" + +def find_non_finite(data: Any, path: str = '') -> Optional[str]: # hygiene-ok: explicit-any -- scans an arbitrary JSON-shaped document + """Locate the first NaN/infinity in a JSON-shaped value. + + ``json.dumps`` emits these as the non-standard ``NaN``/``Infinity`` literals, + so ``is_json_serializable`` accepts them while a strict encoder rejects them. + + :param data: Value to scan. + :param path: Dotted path of ``data`` within the enclosing document. + :return: Path of the first non-finite float, or ``None`` when there is none. + """ + if isinstance(data, float) and not isfinite(data): + return path or '' + if isinstance(data, dict): + for k, v in data.items(): + hit = find_non_finite(v, f'{path}.{k}' if path else str(k)) + if hit is not None: + return hit + elif isinstance(data, (list, tuple)): + for i, v in enumerate(data): + hit = find_non_finite(v, f'{path}[{i}]') + if hit is not None: + return hit + return None + def serialize_to_json_val(obj: Any) -> JSONVal: if isinstance(obj, (str, int, float, bool, type(None))): return obj diff --git a/graphistry/utils/lazy_import.py b/graphistry/utils/lazy_import.py index 7081a53aed..a7e9f7bf5d 100644 --- a/graphistry/utils/lazy_import.py +++ b/graphistry/utils/lazy_import.py @@ -7,8 +7,10 @@ #TODO use new importer when it lands (this is copied from umap_utils) def lazy_cudf_import(): try: - warnings.filterwarnings("ignore") - import cudf # type: ignore + # scoped, not global: a bare filterwarnings here mutes every later warning in the process + with warnings.catch_warnings(): + warnings.filterwarnings("ignore") + import cudf # type: ignore # cudf >= 26.02 removed DataFrame.from_pandas() and Series.from_pandas(). # Restore them so existing call sites keep working across RAPIDS versions. @@ -27,7 +29,6 @@ def lazy_cudf_import(): def lazy_cuml_import(): try: - warnings.filterwarnings("ignore") with warnings.catch_warnings(): warnings.filterwarnings("ignore") import cuml # type: ignore @@ -185,7 +186,8 @@ def lazy_import_has_min_dependancy(): return False, e except Exception as e: logger.warn('Unexpected exn during lazy import', exc_info=e) - return False, e, None + # 2-tuple on every path: callers unpack two values. + return False, e def assert_imported_text(): has_dependancy_text_, import_text_exn, _ = lazy_sentence_transformers_import()