feat(promql+sql): unified positional L3 intent algebra (PromQL & SQL lowering)

[zzylol]

⚠️ Update — reconciled onto the unified positional L3 IR

This supersedes the "consistency fixes" notes in the original description below.

Since this PR was opened, #4 (SQL lowering) merged to main with a name-based L3 IR, which conflicted structurally with this PR's positional design (ColumnId + Binder). Per the agreed direction — workload-level reuse / CSE needs positional identity to prove row-stability — this branch now reconciles both onto one positional IR: PromQL and SQL lower through the same convert_root. origin/main is merged in; the IR conflicts were resolved in favour of the positional design (the name-based expr.rs is deleted).

What landed on top of the original PromQL work

Area	Change
Scalar IR	Split into L2Expr (name-based, Column(ColumnRef), front-end-emitted) and L3Expr (positional, Column(ColumnId)) — both the SQL ∪ PromQL superset (Arith / Cast / InList / FunctionCall / Case / IsNull + Like/ILike + Regex/NotRegex). The converter resolves every L2 column ref to a ColumnId, so all L3 column identity — filters, projections, sort keys, join predicates — is positional, not just group keys.
AggIntent	col-carrying Sum/Min/Max/Avg/StdDev/Variance { col: Option<ColumnId> } (None = the time-series sample value); StdDev/Variance kept first-class
Converter	convert is bottom-up schema-aware: keys + per-aggregate input columns resolve against the converted child's output_schema (so a JOIN's concatenated schema binds correctly); relational::SourceSpec carries Option<Schema> for SQL leaves
Schema flow	Join concatenates left+right (per-JoinKind nullability); Project recomputes its schema from its items; Aggregate.output_names lets a Project-over-Aggregate resolve DataFusion's generated names; both SQL GROUP BY and PromQL by(...) (over instant selectors + per-series rate/increase) now land the key in a positional Aggregate.by — one shape across the two front ends (see "L3 canonicality cleanup" below)
SQL front end	#4's DataFusion lowerer re-targeted to emit relational L2 (Scan / Filter / Project / Aggregate / Sort / Limit / Distinct / Union / Join / analytic WindowFunc) + convert_root; new SqlCatalog; dropped the name-based schema_pass.rs

Status

PromQL and SQL both lower onto the positional L3 IR — 133 tests pass, clippy --all-targets -D warnings clean, fmt --check clean. See docs/intent-algebra-reconciliation.md (the ASAP ⇄ control_plane plan; this intra-repo #4⇄#5 step is now done).

L3 canonicality cleanup (post-merge review)

A review of the fused IR found redundancy and multiple spellings of the same thing; fixed in three commits:

• A — one operator vocabulary. BinaryOpKind no longer re-declares the arithmetic/comparison operators the scalar IR already has; it now reuses Arith(ArithOp) / Compare(CompareOp) and keeps only the PromQL-vector ops (And/Or/Unless/Pow/Atan2). Display lives once on ArithOp/CompareOp. So a comparison has exactly one representation + rendering (they had already drifted — ILike/NotILike were on only one copy).
• E — no name-based residue in the positional IR. Distinct.cols is now Vec<ColumnId> (was Vec<ColumnRef>, resolved lazily); ColumnRef (incl. the SampleValue/Wildcard conventions) moved out of the L3 query_expr module into expr_ir, since it's a purely name-based L2 concept the converter always resolves away.
• D — one grouped-aggregate shape. PromQL sum by (job) (…) previously parked its key in a name-based Partition (documented as a droppable L5 hint, yet load-bearing) over an empty Aggregate.by, diverging from SQL's positional Aggregate.by. Range-vector funcs are now modeled as per-series, label-preserving reductions, so the outer cross-series aggregate resolves its keys positionally — instant-vector and rate/increase grouped aggregates emit the same positional shape as SQL (better for workload CSE / Wire workload-level CSE into a cost model #6). Grouped *_over_time still falls back to Partition pending Converge grouped *_over_time onto positional Aggregate.by (residual of review D) #8.

L2 cleanup (post-merge review)

A second review of the L2 relational IR + its producers:

• docs: clean up L3 QueryExpr; add schema flow + L4 SketchExpr + system I/O #1 (correctness) — qualified group keys. Aggregate.keys / TopK.by / WindowFunc.partition_by were unqualified Vec<String>, so the issue-L3 join-predicate disambiguation via alias-qualified schema columns (3b follow-up to #5) #7 qualifier fix didn't reach GROUP BY / PARTITION BY: a self-join's GROUP BY b.k and GROUP BY a.k both bound to the first k (wrong result). They now carry ColumnRef and resolve via column_id_qualified with the bare-name fallback, like the scalar path. Pinned by a self-join test (GROUP BY b.service → by=[5], a.service → [1]).
• docs: QuerySpec definition + batched-queries DAG-reuse example #2 (scalar) — one Expr<C>. The byte-identical L2Expr/L3Expr 13-variant twins are now one generic Expr<C> with L2Expr = Expr<ColumnRef> / L3Expr = Expr<ColumnId> and one set of helpers. (Node-level QueryExpr<C> was assessed and not pursued — see limitations.)
• feat(core): interface types for L1 input, L3 intent algebra, L4 sketch algebra #3/feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4/Wire workload-level CSE into a cost model #6 (hygiene). Removed the write-only AggItem.distinct field (redundant with AggFunc::CountDistinct); aligned AggItem.alias to Option<String>; doc-marked the L2 nodes no front end produces (Ref/LetBinding/Merge/Partition, PartitionKeys::Without) as Reserved.

Known limitations (follow-up)

• SQL semi/anti/mark joins and subqueries are rejected (no L3 node yet) rather than mislowered. Analytic window functions are supported, but window frames (ROWS/RANGE BETWEEN …) are not modelled (the default frame is assumed).
• PromQL unary negation (-rate(...), -metric) is rejected — the L2 PromQL path has no negate/scalar node to express the sign flip (no -1 * x, since bare scalar operands are themselves rejected). Pinned by unary_negation_is_rejected__GAP. (SQL's -x lowers correctly as -1 * x.)
• Node-level QueryExpr<C> parameterization is intentionally not done. Unlike the scalar Expr<C> (clean — only the column type varies), the L2 and L3 node trees diverge structurally, not just in column type: L2 Source(SourceSpec) vs L3 Scan{predicates,schema}; L2 TopK folds into L3 Aggregate{TopK intent} (no L3 TopK); AggItem (func + alias) vs AggIntent (intent + accuracy); output_names is L3-only; Window fields differ. A single QueryExpr<C> would need a multi-associated-type Layer trait and still couldn't unify the divergent variant sets — net more complexity for little gain, while the converter stays a real transformation (accuracy threading, TopK/window folds, Filter→Scan fold). Left as two enums by design.
• Join predicates over duplicate column names are now disambiguated (closes L3 join-predicate disambiguation via alias-qualified schema columns (3b follow-up to #5) #7). Column carries an optional table qualifier; the SQL front end qualifies a scan's columns with the table name (and re-qualifies them with the alias for a SubqueryAlias), and ColumnRef::Qualified { table, name } resolves via Schema::column_id_qualified with the bare-name lookup as fallback. So metrics JOIN hosts ON metrics.service = hosts.service binds the key to distinct positions Compare(Column(1), Eq, Column(4)), and a true self-join metrics a JOIN metrics b ON a.service = b.service resolves to Compare(Column(1), Eq, Column(5)). The predicate still flows through DataFusion's filter (no switch to the optimized plan), so existing SQL lowering behaviour is unchanged.

Original PR description (PromQL L1–L3 design notes)

Summary

Adds an asap-control-lower crate that lowers PromQL through L1→L2→L3, ending at the shared intent_algebra::QueryExpr. This is the PromQL sibling of the SQL work in #4 and follows the same shape: a thin per-language front end over a free parser AST, lowering to the language- and deployment-independent intent algebra.

• L1 (parse) — delegated to promql-parser 0.8 (Expr AST).
• L2 (per-language tree) — the parser's own AST. Like feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4 (which reuses DataFusion's LogicalPlan), promql-parser already hands us a typed language tree, so no separate L2 struct is materialised.
• L3 (intent algebra) — QueryExpr with intent-only AggIntents over a Source::TimeSeries leaf, in the canonical Window-over-Aggregate shape. No sketch types/params (those are L4).

PromQL → L3 mapping

PromQL	L3
quantile_over_time(φ, m{f}[w])	TimeWindow{w} → Aggregate{[Quantile{φ}]} → Scan
histogram_quantile(φ, rate(m[w]))	substituted to TimeWindow{w} → Aggregate{[Quantile{φ}]}
avg/min/max/sum_over_time	TimeWindow → Aggregate{[Avg/Min/Max/Sum]}
stddev/stdvar_over_time	Aggregate{[StdDev/Variance]}
count_over_time, changes, resets	Aggregate{[Count]}
rate/irate/increase(m[w])	Aggregate{[Rate{w}/Increase{w}]} — window lives in the intent, no TimeWindow node
OUTER by/without (dims) (…)	dims flow onto the inner Aggregate.by
count by (d) (…)	Aggregate{by:d, [Cardinality]}
topk(k, count_over_time(…))	Aggregate{[TopK{k}]} — heavy-hitter, one pass
topk(k, <non-count>) / all bottomk	generic Sort{value} → Limit{k}
m{f}	Scan{TimeSeries, predicates}
a OP b	BinaryOp{vector_match}
expr[r:res]	TimeWindow{Sliding, r, res}

The TopK-vs-Sort+Limit split directly implements the L3 design rule: only frequency ranking has a heavy-hitter sketch primitive, so only topk over count earns AggIntent::TopK; everything else is generic ordering.

Core types filled in (were stubs on main)

To lower into real L3 the following moved from empty stubs to concrete types: MetricRef(String), ColumnRef(String), GroupKey(String), Predicate(L3Expr), plus a new language-independent scalar IR expr_ir::{L3Expr, L3Scalar, CompareOp}, a metric-aware SchemaCatalog, and Source::TimeSeries schema derivation (HasSchema).

Input/output struct notes & proposed consistency fixes

Per the request, here are the struct decisions and the inconsistencies found relative to #4 — flagged for reconciliation when both land:

1. Label matchers are language-independent filters, not a PromQL leaf field. The doc's Source::TimeSeries { …, labels: LabelFilter } puts a PromQL-flavored type on the leaf. Since L3 is meant to be language-independent, this PR drops LabelFilter and lowers matchers to Scan.predicates: Vec<Predicate> (the same home SQL WHERE conjuncts would use). Every node above the leaf stays data-model-agnostic. Note: feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4's SQL path emits a separate Filter node for non-time predicates while leaving Scan.predicates empty — recommend standardizing on one (this PR uses Scan.predicates).

2. Source::TimeSeries.time should be Option<TimeRange>. PromQL strings carry no absolute range (it comes from the /query_range API), so time = None. This matches feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4 making Source::Table.time_range: Option<TimeRange>; the two leaves were inconsistent (time: TimeRange required vs optional). TimeRange { start_ms, end_ms } matches feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4.

3. New CompareOp::Regex / NotRegex for PromQL =~ / !~. feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4's expr_ir already has Like/ILike for SQL; these are the regex analogues. expr_ir.rs overlaps with feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4 — recommend converging on feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4's richer superset (Arith/Case/InList/Cast/FunctionCall) plus these two regex ops.

4. AggIntent additions: Avg, StdDev{population}, Variance{population}. main's 9-variant set has no mean/stddev. feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4 adds Avg{col} / Stddev{col, population} (column-carrying, for arbitrary SQL columns). PromQL always reduces the sample value, so this PR uses unit variants. Recommend converging on a column-carrying shape (e.g. Avg{col: Option<ColumnRef>}) where None = the time-series sample value.

5. Aggregate has no output_names. feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4 added output_names: Vec<String> to thread DataFusion's column names through. PromQL doesn't have that problem (output is value), so this PR leaves Aggregate as {child, by, aggs, having} and names agg outputs synthetically. Merge will need to add the field back; PromQL can pass vec![].

6. Arc<L3Node> (was Rc on main). Switched to match feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4 (its async DataFusion path needs Send). Identical change in both PRs → trivial merge.

7. VectorMatch is now concrete (on/ignoring + group_left/right) instead of an empty stub, so PromQL binary-op matching survives into L3.

8. SchemaCatalog gains metrics: HashMap<String, MetricSchema> alongside feat(SQL): L1 to L3 SQL lowering via DataFusion with TDD #4's tables. Additive; the two halves merge cleanly.

Test plan

• cargo test --workspace --locked — 24 lowering tests (every pattern above + schema population + batch entry point + without catalog resolution + accuracy propagation)
• cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
• cargo fmt --all -- --check

Design doc — Binder / positional ColumnId / CSE

Added docs/promql-lowering.md explaining the two-IR + Binder architecture this PR adopts from the asapquery-backend control plane, and why it's shaped that way:

• Positional ColumnId — why the canonical IR resolves column names to usize positions (identity settled once → downstream refs can't dangle; Scan-carried schema makes sub-trees self-describing; matches the backend's unique_keys: Vec<Vec<usize>> wire format).
• The Binder as an explicit pass — name resolution happens once in Binder::bind, so the L2→canonical converter is purely structural and total; SchemaCatalog policy (incl. why without(...) is rejected under UsageDerivedCatalog) swaps without touching lowering.
• unique_keys / CSE — why cse_reuse_is_legal gates shared-producer reuse on a provable unique key (structural identity finds candidates; unique keys prove row-identity stability), and how that connects to the Wire workload-level CSE into a cost model #6 cost-model follow-up.

It ends with a full trace of topk by (service) (10, count_over_time(requests{env="prod"}[1m])) through all three layers — L1 (Expr AST) → L2 (relational::QueryExpr, names) → L3 (query_expr::QueryExpr, positions) — with the Binder shown explicitly as the L2→L3 pass (it resolves names → positional ColumnIds; it is not a layer of its own). Includes the schema-flow output and the CSE gate refusing to share the windowed scan under the default usage-derived catalog.

Consolidation — unifying L3 toward L4/L5 (follow-up planning)

The intent_algebra in this PR is a slimmed fork of the canonical L3 in ASAPQuery-backend/control_plane. Toward consolidating onto one shared core (and porting control_plane's L4 optimizer + L5 physical/emit), added docs/intent-algebra-reconciliation.md: a file-by-file plan to unify the two L3 copies (base = control_plane's richer IR + this PR's fixes), with drift evidence, three sign-off decisions (StdDev/Variance representation, scalar-IR location, shared types home), per-file tasks/effort/risk, and a recommended order. Verdict: a contained merge, not a rewrite — schema.rs is already byte-identical, ASAP's node set is a subset of control_plane's, and AggIntent is a union; cost concentrates in lower.rs (port per-branch binding + accuracy threading) and expr_ir.rs (scalar superset).

Borrowable optimizer design — Thanos promql-engine

For the L4 optimizer framework, thanos-io/promql-engine (Apache-2.0) is a strong reference — its logicalplan/ is a near-match for our L2→L3 + rule-engine shape:

• Optimizer trait + pipeline + generic traversal (logicalplan/plan.go) — a clean shape for our L4 rule engine.
• MergeSelectsOptimizer (logicalplan/merge_selects.go) — matcher-subsumption reuse (metric{a,c} shares the base select of metric{a} + a residual filter) — a generalization of our CSE beyond identical subtrees.
• PropagateSelectorsOptimizer / sort_matchers.go — predicate propagation + matcher canonicalization (we already added matcher sorting).
• projection.go (label/column pushdown) and DistributeOptimizer (distribute.go, sharded planning) — map to our schema projection and the L5 stage-allocation / topology work.
• Tests: promqlsmith-driven generative fuzzing + differential vs the Prometheus reference engine (engine/enginefuzz_test.go) — a strong upgrade over our fixed corpus.

🤖 Generated with Claude Code

[zzylol]

Update: restructured to mirror the ASAPQuery-backend control-plane IR (commit fb17ce9)

Per review feedback, the L1–L3 plumbing now matches the backend's control_plane/src/intent_algebra/ architecture, while keeping this PR's PromQL semantics and intent vocabulary.

Adopted from the backend:

• Two IRs + Binder. The PromQL parser now emits a Layer-2 relational::QueryExpr; lower::convert_root runs the Binder (name → positional ColumnId resolution against a self-contained Schema) and folds single-statistic aggregates into canonical shapes.
• Positional Schema = { columns, time_index, unique_keys }, ColumnId = usize, Aggregate.by: Vec<ColumnId>; cse_reuse_is_legal gates shared-producer reuse on unique_keys.
• Source::TimeSeries { metric: String } with the schema carried on the Scan node (Binder-built), not on an L3Node edge wrapper.
• Box<QueryExpr> tree (DAG fan-in via LetBinding/Ref) replaces Arc<L3Node>.
• Full workload-level CSE (cse::dedupe_subtrees + LetBinding/Ref).

Kept from the earlier commit (per "others stay as current PR #5"): the AggIntent vocabulary (adds StdDev/Variance; no Frequency/archive intents), the heavy-hitter TopK vs generic Sort + Limit split, no redundant Window node for rate/increase, L3Expr label-matcher predicates with CompareOp::Regex, and the typed LoweringError.

Notes / deltas vs. the backend:

• Grouping (by (...)) now lands on a Partition wrapper for the single-statistic path (backend model), with Aggregate.by: Vec<ColumnId> used on the multi-agg path.
• without (...) is now rejected with a clear error — the usage-derived schema (default UsageDerivedCatalog) can't enumerate a metric's full label set; a registry-backed SchemaCatalog would lift this.
• serde + thiserror added to asap-control-core (the backend IR derives both).

Tests: 22 core + 22 lower pass under --locked; clippy -D warnings and fmt --check clean.

[zzylol]

Follow-up tracked in #6 — wiring the workload-level CSE machinery (dedupe_subtrees / cse_reuse_is_legal / Schema::unique_keys) into a cost model. This PR lands the CSE scaffolding; #6 covers the cost-model integration that makes it influence planning.