diff --git a/README.md b/README.md index 62b60f53..bba6a1c5 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,10 @@ This repo merges three previously-separate projects into a single workspace with ## Status -**Early implementation.** The workspace builds; the front-end crates are landing per the migration plan below. Present today: `crates/core` (shared planner core) and `crates/lower` (SQL/PromQL → L2 lowering). The two docs below are the design + migration plan that the code is following. +**Early implementation.** The workspace builds and is organized into a +layer-named crate stack (the query→sketch pipeline reads top-to-bottom). The +two docs below are the broader design + migration plan the code is following; +the deployment-model / runtime / bin crates they describe are still planned. - [`docs/design.md`](docs/design.md) — target architecture, repo layout, extension points, wire contracts. - [`docs/migration-plan.md`](docs/migration-plan.md) — phase-by-phase plan to land the merger without disrupting any running deployment. @@ -19,38 +22,52 @@ This repo merges three previously-separate projects into a single workspace with **5-layer query→sketch pipeline**: -| Layer | What | Where it lives | +| Layer | What | Where it lives (today) | |---|---|---| -| 1 | Query-language parsing (PromQL, SQL, DataFusion, ElasticDSL) | `core::query_language` | -| 2 | Per-language logical plan (relational algebra tree) | `core::logical_plan` | -| 3 | Intent algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only — no sketch type, no params). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `core::intent_algebra` | -| 4 | Cost-aware optimizer — rule engine + shared rule library; deployment models pick rules. Produces the **sketch algebra** IR `SketchExpr` (sketch-bound: kind + params committed). | **framework in `core::optimizer`**; sketch-bound IR in `core::sketch_algebra`; rule choices in each deployment model | -| 5 | Physical plan — stage allocation + emit to wire format | **framework in `core::physical`**; topology + emitter in each deployment model | +| 1 | Query-language parsing (PromQL, SQL; DataFusion, ElasticDSL planned) | `asap-frontend-promql`, `asap-frontend-sql` | +| 2 | Per-language relational algebra tree | `asap-ir::intent_algebra::relational` (emitted by the front ends) | +| 3 | Intent algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only — no sketch type, no params). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `asap-ir::intent_algebra` | +| 4 | Cost-aware optimizer — CSE + cost model + sketch-vs-exact boundary + canonicalization. Produces the **sketch algebra** IR (sketch-bound: kind + params committed). | optimizer passes in `asap-plan`; sketch-bound IR in `asap-sketch` | +| 5 | Physical plan — stage allocation + emit to wire format | *planned* (per-deployment-model) | -**Core owns the shared infrastructure across all 5 layers**, not just L1-3. Deployment models are thin: they pick which rules fire (L4), declare their deployment topology (L5), and provide an emitter for their output format. Typical deployment model crate: **500-2500 LOC**. +### Crate DAG + +The workspace splits by pipeline role; the dependency arrows only ever point +**up** — `asap-ir` never depends on a front end, an optimizer, or a runtime. + +``` +asap-ir L2 relational + L3 canonical IR + converter + binder + + resolution + schema + expr + names + workload types + ▲ ▲ ▲ + │ │ └── asap-sketch L4 sketch algebra + │ │ + │ └─────── asap-plan optimizer layer: + │ CSE + cost model + sketch-vs-exact + │ boundary + canonicalize + │ + ├─ asap-frontend-promql PromQL L1→L2 (dep: promql-parser only) + ├─ asap-frontend-sql SQL L1→L2 (dep: datafusion only) + │ └── asap-lower thin facade re-exporting lower_promql / lower_sql + ▲ +asap-e2e integration tests +``` + +Splitting the front ends quarantines their parsers: a caller that needs only +PromQL depends on `asap-frontend-promql` and never compiles DataFusion, and +vice-versa. `asap-lower` is the convenience facade for callers that want both. ``` crates/ - core/ # all 5 layers of shared infrastructure; no I/O - query_language/ # L1 — parsers - logical_plan/ # L2 — per-language algebra trees - intent_algebra/ # L3 IR — QueryExpr + AggIntent (intent only, ~25 variants superset) - sketch_algebra/ # L4 IR — SketchExpr (sketch-bound: kind + params committed) - lower/ # L1→L2→L3 passes - optimizer/ # L4 framework — rule engine + shared rule library + cost traits; produces SketchExpr - physical/ # L5 framework — PhysicalPlanner + stage allocator + topology + sketch catalogue - pipeline/ # L1→…→L5 driver, parameterized on deployment model - runtime/ # HTTP / OpAMP / replanner / store — service skeleton - deployment-model-asaplifecycle # DC-specific rules + 3-stage topology + OTel/backend emitters - deployment-model-asapquery # precompute-engine rules + 1-stage + StreamingConfig/InferenceConfig YAML emitters - deployment-model-asapfusion # DataFusion rewrites + 0-stage (in-process) + LogicalPlan emit - control-proto # OpAMP proto + internal proto (tonic/prost) - testing # shared fixtures -bin/ - asap-controller # long-running service, all deployment models - asap-query # one-shot CLI (deployment-model-asapquery only) - asap-lifecycle # OPTIONAL: standalone service with only deployment-model-asaplifecycle - asap-fusion-bench # OPTIONAL: benchmark harness over deployment-model-asapfusion + ir/ # L2 + L3 IR, converter, binder, schema — no query-language deps + sketch/ # L4 sketch-bound IR (asap-sketch) + plan/ # optimizer layer (asap-plan): CSE now; cost model / boundary / canonicalize are stubs + frontend-promql/ # PromQL L1→L2 (promql-parser) + frontend-sql/ # SQL L1→L2 (datafusion) + lower/ # asap-lower — facade re-exporting both front ends + e2e/ # cross-language integration tests + +# planned (per docs/design.md): L5 physical framework, runtime service, +# deployment-model-* crates, control-proto, and the bin/ entrypoints. ``` A new deployment model lands by adding one crate with `rules.rs` (pick L4 rules) + `topology.rs` + an emitter, plus one line in `bin/asap-controller/main.rs`. No changes to core or runtime. @@ -61,7 +78,7 @@ Three ways downstream can use ASAPController — mix as needed: | Mode | Use case | How | |---|---|---| -| **Rust library** | In-process use of the IR or a specific deployment model (e.g. asap-fusion benchmarks) | `Cargo.toml`: `asap-control-core = { git = "...", tag = "v0.1.0" }` or any individual `deployment-model-*` crate. Per-crate dep isolation keeps dep trees small (deployment-model-asapfusion pulls DataFusion; deployment-model-asapquery pulls PromQL/SQL parsers; neither pulls axum/OpAMP). | +| **Rust library** | In-process use of the IR or a specific deployment model (e.g. asap-fusion benchmarks) | `Cargo.toml`: `asap-ir = { git = "...", tag = "v0.1.0" }` or any individual `deployment-model-*` crate. Per-crate dep isolation keeps dep trees small (deployment-model-asapfusion pulls DataFusion; deployment-model-asapquery pulls PromQL/SQL parsers; neither pulls axum/OpAMP). | | **HTTP service sidecar** | Production control plane — e.g. ASAPQuery-backend POSTing QuerySpec on capability-miss | Run `asap-controller` binary, POST to `/api/v1/plan`. Same contract DC controller speaks today. | | **CLI / Docker image** | One-shot init container (e.g. docker-compose init job that writes `streaming_config.yaml`) | `asap-controller plan --workload ... --output-dir ...` or the dedicated `asap-query` binary | @@ -74,7 +91,7 @@ ASAPController is the **control plane only**. The data plane — OTel collectors Because core now owns the L4/L5 infrastructure (not just L1-3), deployment model crates are small and largely self-contained. A deployment model can live either: - **Inside ASAPController workspace** (lockstep release with core, one-PR cross-deployment-model changes) -- **In its own downstream repo** (independent release cadence, depends on published `asap-control-core`) +- **In its own downstream repo** (independent release cadence, depends on published `asap-ir`) Both produce functionally identical artifacts. The placement is a deployment/team-ownership decision, not an architectural fork. Default: - `deployment-model-asaplifecycle` + `deployment-model-asapquery` in ASAPController workspace (share a YAML emitter). @@ -82,7 +99,7 @@ Both produce functionally identical artifacts. The placement is a deployment/tea ## Building -The build is a normal `cargo build`. The only setup is GitHub access: `crates/lower` +The build is a normal `cargo build`. The only setup is GitHub access: `crates/frontend-promql` depends on our private PromQL parser fork (`ProjectASAP/promql-parser`, branch `asap`) as a git dependency, so Cargo has to be able to clone a private repo.