Design: separate DAG materialization from query-time execution placement

[zzylol]

Problem

Physical candidate generation currently couples two different decisions: whether to maintain a materialization and where to execute the computation when that materialization is absent.

The optional-key search primarily chooses between maintained counter/range-max materializations and external exact subqueries. enabled_materialization_keys expresses the materialization choice; finalize_residuals() externalizes residual raw subtrees, and the serving executor rejects local raw Scan nodes. This does not represent the full intended execution space.

An unmaterialized result should also be obtainable by executing a DAG at query time in ASAP, using summary operators, exact operators, or a legal composition of both. An external DB may execute another part of the DAG, with the results composed in ASAP.

Related: #708 and #730. Removing the arbitrary candidate count limits in #730 does not resolve this design limitation. Existing routing tests establish correctness within the currently supported choices, not completeness of the candidate space.

Proposed design

Separate the following decisions, subject to operator capabilities and semantic constraints:

Decision	Choices
Materialization	Maintain and reuse an intermediate result, or compute it at query time
Operator implementation	Summary operators, exact operators, or a semantically valid composition
Execution placement	ASAP, an external DB, or a partition across legal DAG boundaries

These decisions are related but not interchangeable: exact aggregates can be maintained, and summary merges can execute at query time.

Planner and backend responsibilities

• Planner supplies semantically legal DAGs and alternatives, including operator types, intermediate schemas, window/grouping constraints, and accuracy guarantees. Removing a materialization must not implicitly authorize a summary-to-exact semantic rewrite.
• Backend chooses materialization boundaries and executable implementations/placements for those DAGs, according to capabilities, input availability, and cost evidence.
• For an unmaterialized result, execution follows its dependencies until reaching available materialized state, an accessible data source, or a supported external subDAG boundary. A boundary may cover an entire subDAG rather than only a leaf.
• Runtime executes the bound DAG: reads maintained state, runs local summary/exact operators, evaluates external subDAGs, and composes their outputs.

Example

Maintained summary A --+
                      +--> ASAP query-time summary merge --> estimate --------+
Maintained summary B --+                                                       |
                                                                              |
Raw data --> ASAP input --> query-time exact operators ------------------------+--> ASAP composition --> result
                       \-> query-time summary build/operators --> estimate ---+
                                                                              |
External DB exact subDAG ------------------------------------------------------+

An external DB exact subDAG can feed its output directly into the downstream ASAP composition node. Additional ASAP query-time operators are needed only when the selected DAG places subsequent computation there; they are not a mandatory stage of the external execution path.

The merged intermediate need not be materialized. Exact operators need not all execute in the external DB. Raw data can also enter ASAP directly for query-time exact computation or query-time summary construction and subsequent summary operations/readout. This path does not require a previously maintained summary and is distinct from asking an external DB to execute the query's operators.

Raw input may come from a supported local source or be fetched from an external source; obtaining raw input and delegating computation are separate choices. Enumerate this path only when ASAP has a valid input-access mechanism and the required operators. Raw-to-summary execution must use an explicit supported summary-construction operator, preserving the Planner-authorized accuracy and window semantics. Do not assume that every summary operator accepts raw rows.

Representation and candidate generation

Reuse the existing semantic DAG, query nodes, and physical bindings where possible. The query plan already has summary merge/estimate and local computation nodes; extend concrete missing capabilities rather than introducing a parallel DAG framework or a generic provider hierarchy.

The target design removes the mask as the physical candidate representation and removes enabled_materialization_keys as a control of compilation/execution behavior. Each candidate is a complete DAG physical plan that explicitly describes how its results are obtained, rather than a mask from which the compiler infers execution choices.

The physical plan expresses the supported operations and their dependencies:

Physical operation	Dependencies
Read maintained summary state	Bound state identity and valid window
Build a summary in ASAP at query time	Accessible raw input and a supported summary-construction operator
Execute a summary operator in ASAP at query time	Upstream maintained or query-time summary outputs
Execute an exact operator in ASAP at query time	Upstream results or accessible input data
Execute an external DB subDAG	Lowered request, parameters, and any supported input dependencies

Preserve correspondence with the semantic DAG without requiring one physical node per semantic node. An external execution boundary may implement an entire semantic subDAG, and a materialized result may cover multiple upstream operations. A per-node local/remote flag alone is insufficient.

Materialization selection still exists, but a separate mask need not. Search may use temporary key sets internally; they must not become a second execution contract or duplicate materialization bindings already represented in the physical plan. Do not replace the mask with an expanding enum of combined execution modes.

Validation, costing, explain output, and runtime routing should consume the explicit physical plan. Runtime must not infer external execution from a missing summary binding. Reuse and extend existing DAG/binding structures for concrete capabilities; do not introduce a parallel DAG framework or speculative execution-engine plugin system.

Generate physical candidates from legal materialization and execution bindings. Enumerate supported local and external alternatives for unmaterialized regions instead of automatically externalizing them. Retain bounded search with explicit coverage reporting; do not claim an optimum outside the searched space.

Routing and correctness

• Validate each boundary's input/output types, schemas, labels/grouping, evaluation timestamps, window semantics, and accuracy guarantees.
• Treat materialized inputs as execution boundaries so absorbed dependencies are not recomputed.
• Preserve shared-subDAG reuse, dependency order, and binary operand roles.
• An external DB can execute only a subDAG for which a valid lowering and input/output contract exist. Account for dependencies on ASAP-produced inputs where supported.
• Mark unsupported local operators, unavailable input sources, or unsupported external boundaries as infeasible. Do not silently define recomputation as external execution.
• Keep runtime failure fallback distinct from the planned execution placement.

Cost comparison

Compare complete workload costs across candidates, including maintenance, retained state, raw-input retrieval/scanning and transfer, query-time summary construction and summary/exact computation, shared work, external execution, and boundary data transfer. A capability or missing quote must not be treated as free execution.

Suggested implementation sequence

1. Inventory current local summary/exact operators, input sources, external lowering support, and binding constraints. Identify which capabilities are already executable versus absent.
2. Define the minimal changes to existing DAG/binding structures and validation so a complete physical plan expresses materialization, operator implementation, and execution placement without mask-driven inference.
3. Implement one end-to-end case with competing local query-time and external subDAG candidates, both composable with maintained summaries.
4. Migrate candidate generation, compilation, and costing from mask-driven requests to explicit physical plans for supported cases. Remove enabled_materialization_keys as an execution control, preserving bounded-search reporting. Any transitional compatibility path must lower to the same explicit plan.
5. Expand operator and boundary coverage incrementally based on concrete workloads.

Acceptance criteria

• For the same semantically valid unmaterialized region, when capabilities and inputs permit, candidate generation produces both an ASAP query-time implementation and an external DB implementation.
• A query can combine maintained summaries, raw input processed in ASAP, query-time summary operations, query-time exact operations, and an external exact subDAG.
• For supported inputs/operators, candidates include both raw data -> ASAP -> query-time exact operators and raw data -> ASAP -> query-time summary construction/operators -> readout. Neither path requires pre-existing maintained summaries or delegation of those operators to the external DB.
• End-to-end tests exercise raw-input acquisition and local computation, including the query-time summary path's declared accuracy/window contract. Cost evidence accounts for raw-input access and query-time summary construction.
• Physical candidates explicitly describe materialization reads, local summary/exact execution, external subDAG boundaries, and their dependencies. Removing a materialization does not inherently select external execution.
• enabled_materialization_keys is removed as a compilation/execution control. Any key set retained for search is an internal implementation detail, not a duplicate source of execution truth.
• Validation, costing, explain output, and runtime consume the explicit physical plan; routing does not depend on inferring placement from absent bindings.
• The representation supports a physical boundary covering multiple semantic nodes while preserving semantic correspondence; it does not require a one-to-one node mapping.
• Regression/integration coverage verifies equivalent outputs within the declared accuracy contract, including labels and time/window semantics, across the supported placements.
• Tests verify shared-subDAG reuse, materialized boundaries, operand ordering, and rejection of incompatible boundaries or unavailable inputs.
• Cost evidence can change the selected placement without changing the query's semantic contract; explain output identifies materialization and execution choices separately.
• Existing whole-query native execution remains a supported candidate, and bounded search reports its limitations.

Design questions to resolve before implementation

• Which query-time summary/exact operators and raw-input access paths should the first implementation support?
• What are the minimal extensions to existing bindings for local versus external subDAG execution, including parameterized external inputs?
• Which semantic alternatives and timing constraints need explicit Planner support rather than backend-only placement changes?
• What bounded enumeration strategy gives useful materialization/placement coverage without an exponential search?

This issue proposes the design for discussion; it does not assume general local raw execution or arbitrary external subDAG placement is already implemented.

[zzylol]

@milindsrivastava1997