diff --git a/Cargo.lock b/Cargo.lock index 3e71391b..7e86ddeb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -315,6 +315,7 @@ name = "asap-frontend-promql" version = "0.1.0" dependencies = [ "asap-ir", + "asap-l2", "promql-parser", ] @@ -323,6 +324,7 @@ name = "asap-frontend-sql" version = "0.1.0" dependencies = [ "asap-ir", + "asap-l2", "datafusion", "tokio", ] @@ -336,6 +338,14 @@ dependencies = [ "thiserror", ] +[[package]] +name = "asap-l2" +version = "0.1.0" +dependencies = [ + "asap-ir", + "thiserror", +] + [[package]] name = "asap-lower" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index 5759b045..c9ae4cca 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,7 @@ [workspace] members = [ "crates/ir", + "crates/l2", "crates/sketch", "crates/plan", "crates/frontend-promql", diff --git a/README.md b/README.md index 0e8283a4..0b7d89ad 100644 --- a/README.md +++ b/README.md @@ -39,38 +39,40 @@ runtime coupling. | Crate | Role | Depends on | ~LOC | |---|---|---|---| -| **`asap-ir`** | L2 relational + L3 canonical IR + L2→L3 converter, binder, resolution, schema, expr, workload types | *(nothing — no query-language deps)* | 3,900 | +| **`asap-ir`** | **L3 canonical IR only** — `QueryExpr` + `AggIntent` + scalar expr IR + schema + names | *(nothing — no query-language deps)* | 2,300 | +| `asap-l2` | L2 per-language relational algebra + the L2→L3 converter (`convert_root`, binder, column resolution) | `asap-ir` | 1,600 | | `asap-sketch` | L4 sketch-bound IR (`SketchExpr`) | `asap-ir` | 240 | | `asap-plan` | optimizer layer — CSE (landed); cost-model / boundary / canonicalize (stubs) | `asap-ir` | 290 | -| `asap-frontend-promql` | PromQL L1→L2 | `asap-ir`, **promql-parser** | 1,030 | -| `asap-frontend-sql` | SQL L1→L2 | `asap-ir`, **datafusion** | 1,130 | +| `asap-frontend-promql` | PromQL L1→L2 | `asap-ir`, `asap-l2`, **promql-parser** | 1,030 | +| `asap-frontend-sql` | SQL L1→L2 | `asap-ir`, `asap-l2`, **datafusion** | 1,130 | | `asap-lower` | facade re-exporting both front ends | the two `frontend-*` crates | 15 | | `asap-e2e` | cross-language integration tests | `asap-frontend-promql` | 40 | -Splitting the front ends **quarantines their parsers**: a caller that needs -only PromQL depends on `asap-frontend-promql` and never compiles DataFusion, -and vice-versa (verified with `cargo tree`). `asap-lower` is the convenience -facade for callers that want both. +Two isolation wins fall out of this: +- **The front ends quarantine their parsers** — a caller that needs only PromQL depends on `asap-frontend-promql` and never compiles DataFusion, and vice-versa (verified with `cargo tree`). `asap-lower` is the facade for callers that want both. +- **L3-only consumers stay lean** — `asap-sketch` and `asap-plan` depend on `asap-ir` alone, so they never pull the L2 relational tree, the converter, or the binder (`asap-l2`). Only the front ends, which actually *lower* queries, need `asap-l2`. ### Directory structure ``` crates/ -├── ir/ # asap-ir — the shared IR (largest crate) +├── ir/ # asap-ir — L3 canonical IR (the shared vocabulary) │ └── src/ │ ├── lib.rs │ ├── types.rs # AccuracyTarget, … │ ├── workload.rs # QueryWorkload / QueryLanguage / SqlDialect (front-end input) │ └── intent_algebra/ -│ ├── relational.rs # L2: per-language relational tree the front ends emit │ ├── query_expr.rs # L3: canonical QueryExpr — the IR everything pivots on │ ├── agg_intent.rs # L3: AggIntent vocabulary (Sum/Quantile/Rate/TopK/…) -│ ├── expr_ir.rs # scalar expr IR (L2Expr / L3Expr) -│ ├── lower.rs # L2→L3 converter (convert_root) -│ ├── binder.rs # positional name-resolution seed -│ ├── column_resolution.rs +│ ├── expr_ir.rs # scalar expr IR (L2Expr / L3Expr / ColumnRef) │ ├── schema.rs # per-edge Schema + unique-keys -│ └── names.rs +│ └── names.rs # BindingName / QueryId +├── l2/ # asap-l2 — L2 relational algebra + L2→L3 converter +│ └── src/ +│ ├── relational.rs # L2: per-language relational tree the front ends emit +│ ├── lower.rs # L2→L3 converter (convert_root) +│ ├── binder.rs # positional name-resolution seed +│ └── column_resolution.rs ├── sketch/ # asap-sketch — L4 sketch IR: expr / schema / sketch ├── plan/ # asap-plan — optimizer: cse.rs (landed) │ └── src/ # + cost_model.rs / boundary.rs / canonicalize.rs (stubs) @@ -83,13 +85,14 @@ crates/ # deployment-model-* crates, control-proto, and the bin/ entrypoints. ``` -**Why `asap-ir` holds the most.** L2 (the per-language relational tree) and L3 -(the canonical intent algebra) live in one crate because the **L2→L3 converter -needs both** — front ends only *emit* L2, they don't own it. Its modules -(`relational` / `query_expr` / `agg_intent` / `schema` / `binder` / …) are -cleanly separated and could split into their own crates later if the crate -grows unwieldy; for now they share one compilation unit to keep the converter's -tight coupling in-crate. +**Why L2 and L3 are separate crates.** `asap-ir` is the canonical L3 IR — the +vocabulary every downstream layer pivots on. The L2 relational tree and the +L2→L3 converter live in `asap-l2` because only the *front ends* need them: they +emit L2 and call `convert_root`. Keeping them out of `asap-ir` means the +optimizer (`asap-plan`), the sketch IR (`asap-sketch`), and any future +L3-consuming layer compile against a lean core without the converter/binder +machinery. (The converter co-locates with L2 rather than L3 because it owns the +L2 tree definition and only *reads* L3.) *(Planned.)* A new deployment model will land by adding one crate with `rules.rs` (pick L4 rules) + `topology.rs` + an emitter, plus one line in `bin/asap-controller/main.rs` — no changes to the IR crates. diff --git a/crates/frontend-promql/Cargo.toml b/crates/frontend-promql/Cargo.toml index b5eb9d4b..0e834c0f 100644 --- a/crates/frontend-promql/Cargo.toml +++ b/crates/frontend-promql/Cargo.toml @@ -7,6 +7,7 @@ edition = "2021" # in asap-ir. Pulls the PromQL parser only — never DataFusion. [dependencies] asap-ir = { path = "../ir" } +asap-l2 = { path = "../l2" } # Private mirror of GreptimeTeam/promql-parser (Apache-2.0). `main` tracks # upstream; our `asap` branch carries the local patches. diff --git a/crates/frontend-promql/src/error.rs b/crates/frontend-promql/src/error.rs index 29aeba19..56d2aa6b 100644 --- a/crates/frontend-promql/src/error.rs +++ b/crates/frontend-promql/src/error.rs @@ -1,6 +1,6 @@ use std::fmt; -use asap_ir::intent_algebra::ConvertError; +use asap_l2::ConvertError; /// Errors from lowering a PromQL query (L1 parse → L2 → shared L2→L3 convert). /// diff --git a/crates/frontend-promql/src/lib.rs b/crates/frontend-promql/src/lib.rs index 84069763..30ddd8f0 100644 --- a/crates/frontend-promql/src/lib.rs +++ b/crates/frontend-promql/src/lib.rs @@ -1,16 +1,17 @@ //! PromQL front end: L1 (parse via `promql-parser`) → L2 relational, then the -//! shared L2→L3 [`convert_root`](asap_ir::intent_algebra::convert_root). +//! shared L2→L3 [`convert_root`](asap_l2::convert_root). //! //! Emits the per-language -//! [`relational::QueryExpr`](asap_ir::intent_algebra::relational); the shared -//! converter runs the [`Binder`](asap_ir::intent_algebra::Binder) for +//! [`relational::QueryExpr`](asap_l2::relational); the shared +//! converter runs the [`Binder`](asap_l2::Binder) for //! positional name resolution. Depends on the PromQL parser only — never on the //! SQL / DataFusion stack. pub mod error; pub mod promql; -use asap_ir::intent_algebra::{convert_root, QueryExpr}; +use asap_ir::intent_algebra::QueryExpr; +use asap_l2::convert_root; use asap_ir::types::AccuracyTarget; use asap_ir::workload::{QueryLanguage, QueryWorkload}; diff --git a/crates/frontend-promql/src/promql.rs b/crates/frontend-promql/src/promql.rs index ff35e551..a696f876 100644 --- a/crates/frontend-promql/src/promql.rs +++ b/crates/frontend-promql/src/promql.rs @@ -4,7 +4,7 @@ //! - **L2 (per-language tree)** is built here: the walk interprets PromQL //! semantics (range vectors, aggregate operators, label matchers) and emits //! the language-flavored [`relational::QueryExpr`] the controller's L2→L3 -//! converter ([`convert_root`](asap_ir::intent_algebra::convert_root)) +//! converter ([`convert_root`](asap_l2::convert_root)) //! consumes. Canonicalisation (window-over-aggregate fold, GROUP-BY → //! positional `Aggregate.by`, positional name binding) happens in that //! converter, not here. @@ -44,7 +44,7 @@ use promql_parser::parser::{ use asap_ir::intent_algebra::query_expr::{ BinaryOpKind, GroupSide, VectorGrouping, VectorMatch, VectorMatchKind, }; -use asap_ir::intent_algebra::relational::{ +use asap_l2::relational::{ AggFunc, AggItem, L2SortKey, QueryExpr as L2, SourceSpec, }; use asap_ir::intent_algebra::{ArithOp, ColumnRef, CompareOp, L2Expr, L3Scalar}; diff --git a/crates/frontend-sql/Cargo.toml b/crates/frontend-sql/Cargo.toml index 7dd6812e..1d1a5ed6 100644 --- a/crates/frontend-sql/Cargo.toml +++ b/crates/frontend-sql/Cargo.toml @@ -7,6 +7,7 @@ edition = "2021" # shared L2→L3 converter in asap-ir. Pulls DataFusion only — never promql-parser. [dependencies] asap-ir = { path = "../ir" } +asap-l2 = { path = "../l2" } datafusion = "43" [dev-dependencies] diff --git a/crates/frontend-sql/src/error.rs b/crates/frontend-sql/src/error.rs index 558f5fa9..012c84d4 100644 --- a/crates/frontend-sql/src/error.rs +++ b/crates/frontend-sql/src/error.rs @@ -1,6 +1,6 @@ use std::fmt; -use asap_ir::intent_algebra::ConvertError; +use asap_l2::ConvertError; /// Errors from lowering a SQL query (L1 parse + plan via DataFusion → L2 → /// shared L2→L3 convert). diff --git a/crates/frontend-sql/src/lib.rs b/crates/frontend-sql/src/lib.rs index 3ff24b0d..1e7c7cf6 100644 --- a/crates/frontend-sql/src/lib.rs +++ b/crates/frontend-sql/src/lib.rs @@ -1,16 +1,17 @@ //! SQL front end: L1 (parse + plan via DataFusion) → L2 relational, then the -//! shared L2→L3 [`convert_root`](asap_ir::intent_algebra::convert_root). +//! shared L2→L3 [`convert_root`](asap_l2::convert_root). //! //! Emits the per-language -//! [`relational::QueryExpr`](asap_ir::intent_algebra::relational); the shared -//! converter runs the [`Binder`](asap_ir::intent_algebra::Binder) for +//! [`relational::QueryExpr`](asap_l2::relational); the shared +//! converter runs the [`Binder`](asap_l2::Binder) for //! positional name resolution. Depends on DataFusion only — never on the PromQL //! parser. pub mod error; pub mod sql; -use asap_ir::intent_algebra::{convert_root, QueryExpr}; +use asap_ir::intent_algebra::QueryExpr; +use asap_l2::convert_root; use asap_ir::types::AccuracyTarget; use asap_ir::workload::{QueryLanguage, QueryWorkload, SqlDialect}; diff --git a/crates/frontend-sql/src/sql/mod.rs b/crates/frontend-sql/src/sql/mod.rs index 95358338..8537866e 100644 --- a/crates/frontend-sql/src/sql/mod.rs +++ b/crates/frontend-sql/src/sql/mod.rs @@ -2,8 +2,8 @@ //! //! Parses SQL via DataFusion (over the catalog's registered tables), then walks //! the unoptimized `LogicalPlan` and emits the language-independent -//! [`relational::QueryExpr`](asap_ir::intent_algebra::relational) that -//! [`convert_root`](asap_ir::intent_algebra::convert_root) lowers to +//! [`relational::QueryExpr`](asap_l2::relational) that +//! [`convert_root`](asap_l2::convert_root) lowers to //! canonical L3. Positional column identity, accuracy threading, and the //! window-over-aggregate fold all happen in that converter — this front end //! only interprets SQL semantics into the shared L2 algebra. @@ -17,7 +17,7 @@ use datafusion::logical_expr::{ }; use datafusion::prelude::SessionContext; -use asap_ir::intent_algebra::relational::{ +use asap_l2::relational::{ AggFunc, AggItem, L2ProjectItem, L2SortKey, QueryExpr as L2, SourceSpec, }; use asap_ir::intent_algebra::schema::Schema; @@ -36,7 +36,7 @@ use self::expr::df_expr_to_l2; use self::types::schema_to_arrow; /// Lowers SQL strings to the Layer-2 [`relational::QueryExpr`] over a table -/// [`SqlCatalog`]. Call [`convert_root`](asap_ir::intent_algebra::convert_root) +/// [`SqlCatalog`]. Call [`convert_root`](asap_l2::convert_root) /// on the result for canonical L3. pub struct SqlLowerer<'a> { catalog: &'a SqlCatalog, diff --git a/crates/ir/src/intent_algebra/mod.rs b/crates/ir/src/intent_algebra/mod.rs index db25063c..4dfabb1d 100644 --- a/crates/ir/src/intent_algebra/mod.rs +++ b/crates/ir/src/intent_algebra/mod.rs @@ -1,42 +1,33 @@ -//! Layers 2–3 of the controller pipeline. +//! Layer 3 — the canonical intent algebra IR. //! -//! - [`relational`] — the Layer-2 per-language algebra tree the parser front -//! ends emit (PromQL / SQL). -//! - [`lower`] — the L2→L3 converter ([`convert_root`]), which runs the -//! [`Binder`] for name resolution and folds single-statistic sketchable -//! aggregates into canonical shapes. //! - [`query_expr`] — the canonical, language- and deployment-independent L3 //! intent algebra ([`QueryExpr`] + [`AggIntent`]), with positional //! [`ColumnId`] schema flow. +//! - [`agg_intent`] — the L3 aggregation-intent vocabulary. +//! - [`expr_ir`] — the scalar expression IR ([`L2Expr`] / [`L3Expr`] / +//! [`ColumnRef`]) shared by L2 and L3. +//! - [`schema`] — the per-edge [`Schema`] every L3 node carries. +//! - [`names`] — binding / query identifiers. //! -//! Workload-level common-sub-expression elimination over L3 lives in the -//! optimizer layer (`asap-plan`), not here — the IR crate stays free of -//! cost-aware passes. +//! The Layer-2 relational tree and the L2→L3 converter (`convert_root`, the +//! `Binder`, column resolution) live in the `asap-l2` crate — front ends need +//! them, but L3-only consumers (optimizer, sketch) do not, so they stay out of +//! this crate. Workload-level CSE lives in `asap-plan`. pub mod agg_intent; -pub mod binder; -pub mod column_resolution; pub mod expr_ir; -pub mod lower; pub mod names; pub mod query_expr; -pub mod relational; pub mod schema; pub use agg_intent::{ agg_accuracy, agg_is_exact, agg_is_mergeable, default_cardinality, default_quantile, AggIntent, }; -pub use binder::{Binder, SchemaCatalog, UsageDerivedCatalog}; -pub use column_resolution::{ - infer_schema_for_root, infer_source_schema, output_schema_for_aggregate, resolve_column_ref, - resolve_column_refs, resolve_expr, ResolveError, -}; pub use expr_ir::{ArithOp, ColumnRef, CompareOp, Expr, L2Expr, L3Expr, L3Scalar}; -pub use lower::{convert, convert_root, ConvertError}; pub use names::{BindingName, QueryId}; pub use query_expr::{ - BinaryOpKind, BindingScope, DataModel, GroupKeys, GroupSide, JoinKind, Predicate, - ProjectItem, QueryExpr, QueryExprError, SetOpKind, SortKey, Source, VectorGrouping, - VectorMatch, VectorMatchKind, WindowFuncKind, WindowKind, + BinaryOpKind, BindingScope, DataModel, GroupKeys, GroupSide, JoinKind, Predicate, ProjectItem, + QueryExpr, QueryExprError, SetOpKind, SortKey, Source, VectorGrouping, VectorMatch, + VectorMatchKind, WindowFuncKind, WindowKind, }; pub use schema::{cse_reuse_is_legal, Column, ColumnId, CseError, DataType, Schema}; diff --git a/crates/l2/Cargo.toml b/crates/l2/Cargo.toml new file mode 100644 index 00000000..c83f3347 --- /dev/null +++ b/crates/l2/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "asap-l2" +version = "0.1.0" +edition = "2021" + +# The L2 per-language relational algebra + the L2→L3 converter (convert_root, +# binder, column resolution). Front ends emit L2 and call convert_root; this +# crate owns both. Depends only on the L3 IR crate (asap-ir). +[dependencies] +asap-ir = { path = "../ir" } +thiserror = "2" diff --git a/crates/ir/src/intent_algebra/binder.rs b/crates/l2/src/binder.rs similarity index 93% rename from crates/ir/src/intent_algebra/binder.rs rename to crates/l2/src/binder.rs index a8cb9413..945bbe5f 100644 --- a/crates/ir/src/intent_algebra/binder.rs +++ b/crates/l2/src/binder.rs @@ -2,7 +2,7 @@ //! //! [`Binder::bind`] produces the complete, self-contained [`Schema`] every //! `ColumnId` in the converted canonical tree indexes into. The converter -//! ([`super::lower::convert`]) then becomes purely structural: it threads the +//! ([`crate::lower::convert`]) then becomes purely structural: it threads the //! Binder's schema and positional resolution downstream is **total**. //! //! The default [`UsageDerivedCatalog`] knows nothing — every schema is derived @@ -11,10 +11,10 @@ //! `SchemaCatalog` is future work; the `Binder` pass does not change when it //! lands, only the catalog impl swaps. -use crate::intent_algebra::expr_ir::ColumnRef; -use crate::intent_algebra::expr_ir::L2Expr; -use crate::intent_algebra::relational::QueryExpr as LQueryExpr; -use crate::intent_algebra::schema::{Column, DataType, Schema}; +use asap_ir::intent_algebra::expr_ir::ColumnRef; +use asap_ir::intent_algebra::expr_ir::L2Expr; +use crate::relational::QueryExpr as LQueryExpr; +use asap_ir::intent_algebra::schema::{Column, DataType, Schema}; /// The DB / source-schema metadata source — resolves a source (metric / /// table) name to its known columns. @@ -24,7 +24,7 @@ use crate::intent_algebra::schema::{Column, DataType, Schema}; /// Distinct from `Scan.schema`, which is the *resolved* binding schema this /// feeds — the catalog is the input, the schema is the result. Even a /// registry-backed PromQL catalog yields an **open** schema -/// ([`Schema::closed`](super::schema::Schema::closed) `= false`): a metric's +/// ([`Schema::closed`](asap_ir::intent_algebra::schema::Schema::closed) `= false`): a metric's /// labels are per-series and time-varying, so the registry is a superset hint, /// not a per-row contract. pub trait SchemaCatalog { @@ -170,9 +170,9 @@ fn collect_referenced_columns(tree: &LQueryExpr) -> Vec { #[cfg(test)] mod tests { use super::*; - use crate::intent_algebra::expr_ir::ColumnRef; - use crate::intent_algebra::relational::{L2SortKey, QueryExpr as LQueryExpr, SourceSpec}; - use crate::intent_algebra::L2Expr; + use asap_ir::intent_algebra::expr_ir::ColumnRef; + use crate::relational::{L2SortKey, QueryExpr as LQueryExpr, SourceSpec}; + use asap_ir::intent_algebra::L2Expr; fn src(name: &str) -> LQueryExpr { LQueryExpr::Source(SourceSpec::new(name)) diff --git a/crates/ir/src/intent_algebra/column_resolution.rs b/crates/l2/src/column_resolution.rs similarity index 97% rename from crates/ir/src/intent_algebra/column_resolution.rs rename to crates/l2/src/column_resolution.rs index 904c0cc5..485e6381 100644 --- a/crates/ir/src/intent_algebra/column_resolution.rs +++ b/crates/l2/src/column_resolution.rs @@ -2,17 +2,17 @@ //! //! The Layer-2 IR uses `ColumnRef` (name-based, optionally table-qualified); //! the canonical IR uses positional [`ColumnId`] resolved against a per-node -//! [`Schema`]. These helpers bridge the two — the [`Binder`](super::binder) +//! [`Schema`]. These helpers bridge the two — the [`Binder`](crate::binder) //! builds the schema, and [`resolve_column_refs`] turns the L2 refs (group //! keys, dedup columns) into positional ids, qualifier-aware. use thiserror::Error; -use crate::intent_algebra::agg_intent::AggIntent; -use crate::intent_algebra::expr_ir::ColumnRef; -use crate::intent_algebra::expr_ir::{L2Expr, L3Expr}; -use crate::intent_algebra::relational::QueryExpr; -use crate::intent_algebra::schema::{Column, ColumnId, DataType, Schema}; +use asap_ir::intent_algebra::agg_intent::AggIntent; +use asap_ir::intent_algebra::expr_ir::ColumnRef; +use asap_ir::intent_algebra::expr_ir::{L2Expr, L3Expr}; +use crate::relational::QueryExpr; +use asap_ir::intent_algebra::schema::{Column, ColumnId, DataType, Schema}; /// Errors returned by the resolution helpers. #[derive(Debug, Error, PartialEq, Eq)] diff --git a/crates/l2/src/lib.rs b/crates/l2/src/lib.rs new file mode 100644 index 00000000..c1360b15 --- /dev/null +++ b/crates/l2/src/lib.rs @@ -0,0 +1,23 @@ +//! `asap-l2` — the Layer-2 relational algebra + the L2→L3 converter. +//! +//! The parser front ends emit the per-language [`relational::QueryExpr`] tree; +//! [`convert_root`] lowers it to the canonical L3 [`QueryExpr`] in +//! [`asap_ir`], running the [`Binder`] for positional name resolution and +//! folding single-statistic sketchable aggregates into canonical shapes. +//! +//! This crate owns L2 *and* the converter because the converter needs both L2 +//! and L3; it depends only on the L3 IR crate. Downstream consumers that reason +//! about L3 only (optimizer, sketch) depend on `asap-ir` directly and never +//! pull this lowering machinery. + +pub mod binder; +pub mod column_resolution; +pub mod lower; +pub mod relational; + +pub use binder::{Binder, SchemaCatalog, UsageDerivedCatalog}; +pub use column_resolution::{ + infer_schema_for_root, infer_source_schema, output_schema_for_aggregate, resolve_column_ref, + resolve_column_refs, resolve_expr, ResolveError, +}; +pub use lower::{convert, convert_root, ConvertError}; diff --git a/crates/ir/src/intent_algebra/lower.rs b/crates/l2/src/lower.rs similarity index 96% rename from crates/ir/src/intent_algebra/lower.rs rename to crates/l2/src/lower.rs index ce168caf..f60b9339 100644 --- a/crates/ir/src/intent_algebra/lower.rs +++ b/crates/l2/src/lower.rs @@ -6,7 +6,7 @@ //! see the `Aggregate` arm. //! //! Name resolution is an explicit pass: [`convert_root`] runs the -//! [`Binder`](super::binder) first to build the complete, self-contained +//! [`Binder`](crate::binder) first to build the complete, self-contained //! schema every `ColumnId` indexes into, so positional resolution downstream //! is total. @@ -14,19 +14,19 @@ use std::time::Duration; use thiserror::Error; -use crate::intent_algebra::agg_intent::AggIntent; -use crate::intent_algebra::binder::Binder; -use crate::intent_algebra::column_resolution::{ +use asap_ir::intent_algebra::agg_intent::AggIntent; +use crate::binder::Binder; +use crate::column_resolution::{ output_schema_for_aggregate, resolve_column_refs, resolve_expr, ResolveError, }; -use crate::intent_algebra::expr_ir::{ColumnRef, L2Expr, L3Expr, L3Scalar}; -use crate::intent_algebra::names::BindingName; -use crate::intent_algebra::query_expr::{ +use asap_ir::intent_algebra::expr_ir::{ColumnRef, L2Expr, L3Expr, L3Scalar}; +use asap_ir::intent_algebra::names::BindingName; +use asap_ir::intent_algebra::query_expr::{ GroupKeys, Predicate, ProjectItem, QueryExpr as CQueryExpr, SortKey, Source, }; -use crate::intent_algebra::relational::{AggFunc, QueryExpr as LQueryExpr, SourceSpec}; -use crate::intent_algebra::schema::{ColumnId, Schema}; -use crate::types::AccuracyTarget; +use crate::relational::{AggFunc, QueryExpr as LQueryExpr, SourceSpec}; +use asap_ir::intent_algebra::schema::{ColumnId, Schema}; +use asap_ir::types::AccuracyTarget; /// Errors produced while converting a Layer-2 tree to canonical. #[derive(Debug, Error)] @@ -38,7 +38,7 @@ pub enum ConvertError { /// Deriving the schema of an already-converted child failed (needed to /// resolve positional column references against it). #[error("schema derivation failed: {0}")] - Schema(#[from] crate::intent_algebra::query_expr::QueryExprError), + Schema(#[from] asap_ir::intent_algebra::query_expr::QueryExprError), /// Group keys landed on a per-series windowed/range reduction, which must /// stay label-preserving (`Aggregate.by` empty). The only PromQL shape that /// would do this is a generic `topk by (…)`, whose grouping is routed to @@ -540,13 +540,13 @@ fn agg_func_to_intent(func: &AggFunc, acc: &AccuracyTarget, col: Option Column { Column::new(name, dtype, false) diff --git a/crates/ir/src/intent_algebra/relational.rs b/crates/l2/src/relational.rs similarity index 93% rename from crates/ir/src/intent_algebra/relational.rs rename to crates/l2/src/relational.rs index 0f5a1979..702c9029 100644 --- a/crates/ir/src/intent_algebra/relational.rs +++ b/crates/l2/src/relational.rs @@ -1,17 +1,17 @@ //! The Layer-2 relational IR — the per-language query algebra the parser -//! front ends emit, before [`convert_root`](super::lower::convert_root) lowers -//! it to the canonical L3 [`query_expr::QueryExpr`](super::query_expr::QueryExpr). +//! front ends emit, before [`convert_root`](crate::lower::convert_root) lowers +//! it to the canonical L3 [`query_expr::QueryExpr`](asap_ir::intent_algebra::query_expr::QueryExpr). //! //! Leaf / scalar types (`ColumnRef`, `SortKey`, //! `BinaryOpKind`, `VectorMatch`) are owned by `query_expr` and re-used here so //! there is one canonical spelling. Filter / having / project expressions use -//! the shared language-independent [`L3Expr`](super::expr_ir::L3Expr). +//! the shared language-independent [`L3Expr`](asap_ir::intent_algebra::expr_ir::L3Expr). use std::time::Duration; -pub use super::expr_ir::{ColumnRef, L2Expr}; -pub use super::query_expr::{BinaryOpKind, VectorMatch, WindowFuncKind}; -use super::schema::Schema; +pub use asap_ir::intent_algebra::expr_ir::{ColumnRef, L2Expr}; +pub use asap_ir::intent_algebra::query_expr::{BinaryOpKind, VectorMatch, WindowFuncKind}; +use asap_ir::intent_algebra::schema::Schema; /// SELECT-list item at Layer 2 — a name-based [`L2Expr`] + optional alias. /// (`query_expr::ProjectItem` is the positional L3 sibling.) @@ -35,7 +35,7 @@ pub struct SourceSpec { /// Metric name (PromQL) or table name (SQL). pub name: String, /// Front-end-resolved leaf schema. `Some` for SQL tables (DataFusion knows - /// the columns); `None` for PromQL, where the [`Binder`](super::binder) + /// the columns); `None` for PromQL, where the [`Binder`](crate::binder) /// synthesises a usage-derived schema (the `(ts, value)` floor + referenced /// labels). The presence of a schema also selects the L3 `Source` variant: /// `Some` → `Source::Table`, `None` → `Source::TimeSeries`. @@ -71,7 +71,7 @@ pub struct AggItem { } /// Layer-2 aggregate functions. Mapped to canonical [`AggIntent`] by -/// [`super::lower::convert`]. +/// [`crate::lower::convert`]. #[derive(Debug, Clone, PartialEq)] pub enum AggFunc { Count, @@ -182,13 +182,13 @@ pub enum QueryExpr { Merge { inputs: Vec }, Join { - kind: super::query_expr::JoinKind, + kind: asap_ir::intent_algebra::query_expr::JoinKind, pred: Option, left: Box, right: Box, }, SetOp { - kind: super::query_expr::SetOpKind, + kind: asap_ir::intent_algebra::query_expr::SetOpKind, all: bool, left: Box, right: Box, diff --git a/docs/design.md b/docs/design.md index ea286103..0b1e977b 100644 --- a/docs/design.md +++ b/docs/design.md @@ -125,7 +125,7 @@ This is what makes the topology a *parameter* rather than an axis of code: the s This drives the core/deployment model split: -- **The shared infrastructure crates** own all 5 layers. Landed today (§5.1): the front ends (L1→L2), `asap-ir` (L3 IR + converter), `asap-sketch` (L4 IR), and `asap-plan` (L4 optimizer — CSE, with the rule engine + cost model + boundary + canonicalize as stubs). Planned: the L5 stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. (The original design put all of this in one `core/` crate; §5.1 splits it by role.) +- **The shared infrastructure crates** own all 5 layers. Landed today (§5.1): the front ends (L1→L2), `asap-l2` (L2 relational + the L2→L3 converter), `asap-ir` (L3 canonical IR), `asap-sketch` (L4 IR), and `asap-plan` (L4 optimizer — CSE, with the rule engine + cost model + boundary + canonicalize as stubs). Planned: the L5 stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. (The original design put all of this in one `core/` crate; §5.1 splits it by role.) - **Each deployment model is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any deployment-model-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. ## 4. Principles @@ -162,42 +162,46 @@ The workspace is split by pipeline role into a layer-named crate stack. The dependency arrows only ever point **up** — `asap-ir` never depends on a front end, an optimizer, or a runtime. -``` -asap-ir L2 relational + L3 canonical IR + converter + binder + - resolution + schema + expr + names + workload types - ▲ ▲ ▲ - │ │ └── asap-sketch L4 sketch algebra - │ │ - │ └─────── asap-plan optimizer layer: CSE (landed) + - │ cost model / boundary / canonicalize (stubs) - │ - ├─ asap-frontend-promql PromQL L1→L2 (dep: promql-parser only) - ├─ asap-frontend-sql SQL L1→L2 (dep: datafusion only) - │ └── asap-lower thin facade re-exporting lower_promql / lower_sql - ▲ -asap-e2e integration tests -``` +Dependency table (clearer than arrows). Every crate depends on `asap-ir`; +`asap-ir` depends on nothing above it. + +| Crate | Role | Depends on | +|---|---|---| +| `asap-ir` | **L3 canonical IR only** — `QueryExpr` + `AggIntent` + scalar expr + schema + names | *(nothing)* | +| `asap-l2` | L2 relational algebra + L2→L3 converter (`convert_root`, binder, resolution) | `asap-ir` | +| `asap-sketch` | L4 sketch-bound IR | `asap-ir` | +| `asap-plan` | optimizer — CSE (landed) + stubs | `asap-ir` | +| `asap-frontend-promql` | PromQL L1→L2 | `asap-ir`, `asap-l2`, promql-parser | +| `asap-frontend-sql` | SQL L1→L2 | `asap-ir`, `asap-l2`, datafusion | +| `asap-lower` | facade over both front ends | the two `frontend-*` | +| `asap-e2e` | integration tests | `asap-frontend-promql` | ``` crates/ - ir/ # L2 relational + L3 canonical IR + L2→L3 converter + binder - # + column resolution + schema + expr IR + names + workload - # types. No query-language dependencies. (crate: asap-ir) + ir/ # L3 canonical IR: query_expr + agg_intent + expr_ir + + # schema + names (+ types, workload). No query-language + # deps. (crate: asap-ir) + l2/ # L2 relational algebra + L2→L3 converter: relational + + # lower (convert_root) + binder + column_resolution. + # (crate: asap-l2) sketch/ # L4 sketch-bound IR (SketchExpr). (crate: asap-sketch) plan/ # optimizer layer: CSE today; cost_model / boundary / # canonicalize are stubs. (crate: asap-plan) - frontend-promql/ # PromQL L1→L2, promql-parser only. (crate: asap-frontend-promql) - frontend-sql/ # SQL L1→L2, datafusion only. (crate: asap-frontend-sql) + frontend-promql/ # PromQL L1→L2, promql-parser. (crate: asap-frontend-promql) + frontend-sql/ # SQL L1→L2, datafusion. (crate: asap-frontend-sql) lower/ # facade re-exporting both front ends. (crate: asap-lower) e2e/ # cross-language integration tests. (crate: asap-e2e) ``` -Splitting the front ends **quarantines their parsers**: a caller that needs -only PromQL depends on `asap-frontend-promql` and never compiles DataFusion, -and vice-versa (verified with `cargo tree`). This is why the old monolithic -`LoweringError` was partitioned into `PromqlError` / `SqlError` — a shared -error embedding `DataFusionError` would have leaked DataFusion into the PromQL -crate. +Two isolation wins: +- **The front ends quarantine their parsers**: a PromQL-only caller depends on + `asap-frontend-promql` and never compiles DataFusion, and vice-versa (verified + with `cargo tree`). This is why the old monolithic `LoweringError` was + partitioned into `PromqlError` / `SqlError` — a shared error embedding + `DataFusionError` would have leaked DataFusion into the PromQL crate. +- **L3-only consumers stay lean**: `asap-sketch` and `asap-plan` depend on + `asap-ir` alone and never pull the L2 relational tree / converter / binder + (`asap-l2`) — only the front ends, which actually *lower* queries, need it. **Not yet built** (see §5.2): the L4 optimizer engine + rule library, the L5 physical framework, the runtime service, the `deployment-model-*` crates, the @@ -312,8 +316,8 @@ the landed crates (§5.1) as follows: | §6 name | Landed crate / module | Status | |---|---|---| | `core::query_language` (L1) | `asap-frontend-promql`, `asap-frontend-sql` (parse step) | landed (PromQL + SQL; DataFusion/ElasticDSL planned) | -| `core::logical_plan` (L2) | `asap-ir::intent_algebra::relational` (emitted by the front ends) | landed | -| `core::lower` (L1→L2→L3) | `asap-frontend-*` (L1→L2) + `asap-ir::intent_algebra::lower` / `convert_root` (L2→L3) | landed, split per language | +| `core::logical_plan` (L2) | `asap-l2::relational` (emitted by the front ends) | landed | +| `core::lower` (L1→L2→L3) | `asap-frontend-*` (L1→L2) + `asap-l2::convert_root` (L2→L3, with the binder + column resolution) | landed, split per language | | `core::intent_algebra` (L3) | `asap-ir::intent_algebra` | landed | | `core::sketch_algebra` (L4 IR) | `asap-sketch` | landed | | `core::optimizer` (L4 framework) | `asap-plan` | placeholder (CSE only; engine/rules planned) | diff --git a/docs/migration-plan.md b/docs/migration-plan.md index 5903afcb..a3642507 100644 --- a/docs/migration-plan.md +++ b/docs/migration-plan.md @@ -5,7 +5,7 @@ Companion to `design.md`. Concrete phase-by-phase plan to get from today's three > **Status (reconciled with the landed workspace).** Phases 0–1 (skeleton + > the L1–L3 query→IR core) have substantially landed, but **not** as the single > `core` crate this plan describes: the core was built and then split by -> pipeline role into the layer-named stack — `asap-ir`, `asap-sketch`, +> pipeline role into the layer-named stack — `asap-ir`, `asap-l2`, `asap-sketch`, > `asap-plan`, `asap-frontend-promql`, `asap-frontend-sql`, `asap-lower`, > `asap-e2e` (see `design.md` §5.1). The L4/L5 framework, runtime, and > `deployment-model-*` crates (Phases 2–7) are **not yet built**. The