Skip to content

Do not silently select raw fallback when logical candidate costs are unavailable - #727

Open
zzylol wants to merge 1 commit into
mainfrom
fix/721-cost-unavailable
Open

zzylol wants to merge 1 commit into
mainfrom
fix/721-cost-unavailable

Conversation

@zzylol

@zzylol zzylol commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Closes #721.

What was happening

avg by (job) (data) resolved to a raw KeepPreAsap pass-through — not because anything priced it lower, but because nothing priced it at all. Reproduced from the selection trace before any change:

rank strategy kind cost status
0 SketchAlgorithmStrategy summary null / not_reported_by_cost_model selected
1 SemanticEquivalentRewriteStrategy rewrite null / not_reported_by_cost_model unselected

Rank 0's own rationale says it: "Avg has no summary realization and stays a logical pass-through." Rank 1 is the realizable sum / count rewrite. With no comparable cost, selection preserved candidate discovery order, and the deployment registers the sketch strategy first — so the raw arm won. Strategy registration order was acting as undeclared optimizer policy.

The fix

Per the issue's preferred resolution, selection now fails loudly instead of resolving that group from discovery order:

SelectionError::CostUnavailable {
    target_id, candidate_count, strategies,
    candidates: Vec<CostUnavailableCandidate>,  // id, strategy, replacement_kind, provenance
}

The message names the target and both strategies, and each candidate carries its explain identity, so a reader can see which cost inputs the model owes rather than only that ranking failed. The selection trace records the same resolution under unresolved_group (reason, policy, detail).

The guard is deliberately narrow

My first cut fired on every group with two or more materially different unpriced alternatives — the issue's literal wording — and broke 23 existing tests. A survey showed why:

query mixed-kind unpriced groups rank-0 is a pass-through
avg by (job) (data) 1 yes
sum(sum_over_time(...)) 1 no
7 others 0

sum(sum_over_time(...)) has the same unpriced mixed-kind shape, but the order-chosen candidate is already realizable — nothing degrades to raw, so nothing is silently lost. The distinguishing signal is exactly what the issue title names: the discovery-order winner keeps the subtree pre-ASAP while a realizable alternative sits behind it unranked.

So the guard requires all four of:

  1. more than one candidate (a single candidate offers no choice to get wrong);
  2. no candidate has a finite comparable cost;
  3. rank 0 is a SummaryExpr::KeepPreAsap;
  4. some later candidate is not.

Condition 3 matches on the post-ASAP IR, not on rationale prose, so rewording upstream cannot silently disable it.

Coverage change

Two process-e2e workloads dropped avg, which no longer plans by design — current_series_process (avg(a), avg by (job) (a)) and issue_701_702_process (avg_over_time(...), avg(...)). They plan their whole query set as one workload, so an unplannable root takes the entire test down; keeping avg there would have traded all their serving coverage for one assertion.

avg's behaviour is asserted at the selection layer instead, where the typed error is the observable outcome. The average_overflow fixture keeps avg under an Exact target — that candidate set does not produce this shape, and the test still passes.

This is the intended consequence of failing loudly: those workloads legitimately cannot plan until the cost model prices these roots. Complete-plan costing remains the long-term fix, and with finite comparable costs the guard never fires.

Acceptance criteria

  • Never silently treats unavailable cost as a tie — the group now errors instead of resolving on discovery order.
  • Trace identifies the missing inputs and the resolving policyunresolved_group plus per-candidate identity/strategy/kind/provenance.
  • Regression case for avg by (job) (data)unpriced_alternatives_fail_instead_of_resolving_on_discovery_order asserts the typed error, both strategy names, and that the group is genuinely mixed-kind (summary and rewrite), not two rankings of one shape.
  • Priced roots still selectpriced_roots_still_select guards the other direction, including that a resolved selection carries no unresolved_group.

Testing

  • cargo +1.98.0 fmt -- --check — clean.
  • cargo +1.98.0 clippy --workspace --all-targets -- -D warnings — clean.
  • cargo +1.98.0 test --workspace -- --test-threads=1 — exit 0, zero failures across 27 test binaries.

🤖 Generated with Claude Code

`avg by (job) (data)` resolved to a raw `KeepPreAsap` pass-through, not
because anything priced it lower, but because nothing priced it at all.
Two legal alternatives reach that root — `SketchAlgorithmStrategy`'s
`Avg` pass-through, which has no summary realization, and
`SemanticEquivalentRewriteStrategy`'s realizable `sum / count` rewrite —
and the cost model reports no comparable cost for either. Selection then
preserved candidate discovery order, and the deployment registers the
sketch strategy first, so the raw arm won. Strategy registration order
was acting as undeclared optimizer policy.

Selection now returns `SelectionError::CostUnavailable` for that group,
carrying the target identity, every candidate's identity, strategy,
replacement kind and provenance, so a reader can see which cost inputs
the model owes rather than only that ranking failed. The selection trace
records the same resolution under `unresolved_group`.

The guard is deliberately narrow. It fires only when the discovery-order
winner keeps the subtree pre-ASAP *and* a realizable alternative sits
behind it unranked — the silent raw fallback the report is about. A
group whose order-chosen candidate is already realizable still plans:
`sum(sum_over_time(...))`, for instance, has the same unpriced mixed-kind
shape but degrades to nothing. Detection matches on
`SummaryExpr::KeepPreAsap` rather than on rationale text, so rewording
upstream cannot silently disable it.

Two process e2e workloads dropped `avg`, which no longer plans by
design. Its behaviour is asserted at the selection layer instead, where
the typed error is the observable outcome. The `average_overflow`
fixture keeps `avg` under an Exact target, where the candidate set does
not produce this shape.

Complete-plan costing remains the long-term fix: with finite comparable
costs these roots rank on evidence and the guard never fires.

Closes #721.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Do not silently select raw fallback when logical candidate costs are unavailable

1 participant