Repository navigation
Commit 97b192b
* refactor(optimizer): derive Pass 1's summary candidates from the family-capability table
`summary_candidates` was a hand-written intent → family list that repeated
the family-capability table in the other direction (#953). It now takes
the intent's statistic and keeps, in `FAMILY_PREFERENCE` order, the
families whose table row answers it, less `UNIMPLEMENTED_FAMILIES` (Theta)
and less UnivMon for a distinct-tuple count. Candidate lists and their
order are unchanged.
- The intent → statistic mapping is written once, `sketch_statistic`;
Pass 1's `statistic` wraps it with its existing error.
- `local_guarantee` returns `None` for a sketch read for a statistic its
table row does not answer. Plain values and exact aggregates are not
checked: the table lists no readouts for them.
- Two estimator tests read CMS for a distinct count and Count Sketch for
a total, which the table does not answer; they now read a statistic
each family answers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build: pin asap_sketchlib to main 950bb5c
main after release 0.4.0 carries #245–#248, which the old pin took from a
branch, plus #249 (`memory_bytes` / `estimated_memory_bytes` across the
sketches; KMV builds without `experimental`) and #251 (the public capability
table). The executor drops the `experimental` feature: KMV was the only gated
type it used.
#251 makes `Kll::merge` panic when k or m differ. The KLL kernel checked
only k, so a state adopted with another m would reach the panic; it now
checks both and returns an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(types): read the sketch rows of the family-capability table from asap_sketchlib (#953)
The family-capability table kept its own copy of what each sketch family
answers and how it merges. A sketch family's row is now asap_sketchlib's
(`SketchFamily::capability`, #251), mapped from `SketchAlgorithm` and
`HydraKind` and read in the planner's vocabulary at query time: the seven
statistics `StatisticKind` names, `NoGuarantee` as `Approximate`. The
planner keeps the rows the library has none for: the exact accumulators
and Theta. `readouts` becomes a method; its readers follow.
Cells that change:
- Count-Min and UnivMon totals are `Exact`, not `AsIfBuiltOverUnion`.
Every consumer accepts both alike.
- HydraKll merges disjoint inputs, its quantile as if built over the
union: sketchlib's Hydra merge now requires the same k and m and returns
an error otherwise. Nothing builds a HydraKll yet: Pass 1 offers no
Hydra and the executor binds only HydraCms.
- UnivMon lists its readouts in sketchlib's order, so its entry in the
executor's supported summaries does too.
sketchlib also lists DDSketch's total, the heap sketches' item frequency
and HydraCms's total, which nothing here reads yet. `NOT_READ_YET` keeps
them out, so Pass 1's candidates, the IR's estimate validation and the
executor's supported summaries are unchanged; a test checks sketchlib
still lists each entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(executor): charge DDSketch and UnivMon states sketchlib's byte counts (#947)
A DDSketch state was reserved at the trait's flat 4096 bytes whatever it
held, and its updater counted the bins in use; a UnivMon state by a hand
formula (16 bytes per counter, 256 per heap slot). Both now read
asap_sketchlib's count (#249):
- DDSketch: `memory_bytes()`, the bins its stores allocate; the updater
charges the same.
- UnivMon: its own fields, `UnivMon::estimated_memory_bytes` for its shape
with keys that own no bytes, and the existing bound on the
variable-length keys its heaps retain. `memory_bytes()` walks every
heap, and a build charges memory after every row, so the charge stays
constant time.
KLL, KMV, Count-Min, Count Sketch (with or without a heap) and HLL already
charge what their sketch holds plus their own fields; Hydra has no
sketchlib count. They are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(plan-selection): price summary state bytes with sketchlib's estimates (#946, #947)
Plan selection priced Count-Min, Count Sketch (with or without a heap),
DDSketch, HLL, KMV and UnivMon states from hand-copied sketchlib
internals: 24 bytes per top-k heap slot (a slot holds 150–225 plus its
key), 8 per UnivMon counter, DDSketch's bucket chunks. Each is now
asap_sketchlib's `estimated_memory_bytes` (#249) for the type the
executor builds (`f64` counters, `Vector2D` storage), given what
`sketch_shape` assumes the state holds:
- Count-Min, Count Sketch, HLL: exact.
- KMV: the rows the state folds as its distinct items; exact only at
that many distinct inputs, an upper bound below.
- DDSketch: sketchlib's worst case over insertion orders and merges, for
the rows it folds over the assumed nine decades, with negative values
unless proven non-negative: 37,360 bytes at α = 0.01 with rows unknown,
where the old model gave 18,576.
- Count-Min / Count Sketch with heap: a full heap of keys of
`ASSUMED_HEAP_KEY_BYTES`, an assumption of three 16-byte labels in the
executor's tuple-key encoding (63 bytes). It applies to these heaps
only.
- UnivMon: every layer's heap full of keys of `ASSUMED_VALUE_KEY_BYTES`,
one 16-byte label value with the executor's type tag (17 bytes): Pass 1
feeds UnivMon one column, which a distinct count over a label makes
Utf8.
KLL keeps the planner's own formula, unchanged: sketchlib's estimate also
counts a CDF cache the executor never builds. Theta keeps its `k` hashes.
An α outside (0, 1), where sketchlib's DDSketch estimate panics, is
priced as an empty sketch.
A test builds each family the executor builds per group through its
factory, feeds it rows shaped as the pricing assumes, and compares the
executor's charge with the priced bytes plus each accumulator's own
named fields. The heap families' shapes are dominated by their heaps.
- Count-Min, Count Sketch, HLL and KMV: equal.
- KLL: equal up to k rows. Past k only a loose guard holds, at most
sketchlib's 2 (k - 1) items of compaction scratch over the price; the
formula can price below the charge there, a known gap until a sketchlib
estimate without the CDF cache can price KLL.
- DDSketch, the heap sketches and UnivMon: at least the charge, and at
most 1.5 times it once the state is full. UnivMon's price also bounds
the bytes its sketch holds, by at most 1.6 times: its estimate
includes digest-index growth under churn that the test's labels do not
reach.
HydraCms is left out: plan selection prices an independent Count-Min per
group, the executor one shared grid. Plan selection's tests gain the
executor as a dev-dependency for the test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: read asap_sketchlib's capability vocabulary directly; one executor support declaration
- SketchAlgorithm, StatisticKind and MergedError are asap_sketchlib's
(exhaustive, serde); the Planner's copies, the Readouts enum, NOT_READ_YET
and the conversion matches are gone. FamilyCapability's sketch rows are the
library's rows, unchanged; the exact accumulators keep Planner rows.
- Theta is deleted (no library sketch), with UNIMPLEMENTED_FAMILIES.
- The executor declares what it evaluates in one positive list,
EVALUATED_READOUTS, replacing EVALUATED_FAMILIES and UNEVALUATED_READOUTS;
a test checks every entry is in the table, so a sketchlib bump cannot widen
the declared support.
- The four hard-coded heap-family TopK checks read answers(TopK); the pcc.rs
statistic() wrapper is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: drop deleted Theta from KMV docs; name sketchlib #254 in the pin comment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zzylol <zeyingz@umd.edu>
Co-authored-by: zzylol <50204836+zzylol@users.noreply.github.com>
1 parent 67d35f4 commit 97b192b
40 files changed
Lines changed: 975 additions & 654 deletions
File tree
- crates
- executor
- src
- summary_kernels
- tests
- frontend-promql/tests
- integration-tests/tests
- planner_layering_common
- logical-asap-optimization
- src
- accuracy/estimators
- memo
- rules
- tests
- plan-selection
- src
- search/costing
- planner
- src
- tests
- types
- src/ir/schema
- tests
- docs
- design_docs/concepts
- develop_docs
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
325 | 325 | | |
326 | 326 | | |
327 | 327 | | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
340 | 352 | | |
341 | 353 | | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | 354 | | |
351 | 355 | | |
352 | 356 | | |
353 | 357 | | |
354 | 358 | | |
355 | 359 | | |
356 | | - | |
357 | | - | |
358 | | - | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
359 | 363 | | |
360 | 364 | | |
361 | 365 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
58 | 60 | | |
59 | | - | |
60 | | - | |
61 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
62 | 64 | | |
63 | 65 | | |
64 | 66 | | |
| |||
136 | 138 | | |
137 | 139 | | |
138 | 140 | | |
139 | | - | |
| 141 | + | |
140 | 142 | | |
141 | | - | |
| 143 | + | |
142 | 144 | | |
143 | 145 | | |
144 | 146 | | |
145 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
146 | 153 | | |
147 | 154 | | |
148 | 155 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
45 | 51 | | |
46 | 52 | | |
47 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
266 | | - | |
267 | | - | |
268 | | - | |
269 | | - | |
270 | | - | |
| 266 | + | |
271 | 267 | | |
272 | 268 | | |
273 | 269 | | |
| |||
716 | 712 | | |
717 | 713 | | |
718 | 714 | | |
719 | | - | |
720 | | - | |
721 | | - | |
722 | | - | |
| 715 | + | |
| 716 | + | |
723 | 717 | | |
724 | 718 | | |
725 | 719 | | |
| |||
872 | 866 | | |
873 | 867 | | |
874 | 868 | | |
875 | | - | |
876 | | - | |
877 | | - | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
878 | 872 | | |
879 | 873 | | |
880 | 874 | | |
| |||
884 | 878 | | |
885 | 879 | | |
886 | 880 | | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
887 | 885 | | |
888 | 886 | | |
889 | 887 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
169 | 172 | | |
170 | | - | |
| 173 | + | |
171 | 174 | | |
172 | | - | |
| 175 | + | |
173 | 176 | | |
174 | 177 | | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
186 | 187 | | |
187 | 188 | | |
188 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
| 6 | + | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
12 | | - | |
| 10 | + | |
| 11 | + | |
13 | 12 | | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
23 | 16 | | |
| 17 | + | |
24 | 18 | | |
25 | 19 | | |
26 | 20 | | |
| |||
34 | 28 | | |
35 | 29 | | |
36 | 30 | | |
37 | | - | |
38 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
39 | 34 | | |
40 | 35 | | |
41 | 36 | | |
42 | | - | |
| 37 | + | |
43 | 38 | | |
44 | 39 | | |
45 | 40 | | |
46 | 41 | | |
47 | 42 | | |
48 | 43 | | |
49 | 44 | | |
50 | | - | |
51 | 45 | | |
52 | 46 | | |
53 | 47 | | |
| |||
95 | 89 | | |
96 | 90 | | |
97 | 91 | | |
98 | | - | |
99 | 92 | | |
100 | 93 | | |
101 | 94 | | |
| |||
105 | 98 | | |
106 | 99 | | |
107 | 100 | | |
| 101 | + | |
108 | 102 | | |
109 | 103 | | |
110 | 104 | | |
111 | 105 | | |
112 | 106 | | |
113 | | - | |
| 107 | + | |
114 | 108 | | |
115 | 109 | | |
116 | 110 | | |
| |||
133 | 127 | | |
134 | 128 | | |
135 | 129 | | |
| 130 | + | |
136 | 131 | | |
137 | 132 | | |
138 | 133 | | |
| |||
230 | 225 | | |
231 | 226 | | |
232 | 227 | | |
233 | | - | |
234 | | - | |
235 | | - | |
| 228 | + | |
| 229 | + | |
236 | 230 | | |
237 | 231 | | |
238 | 232 | | |
| |||
252 | 246 | | |
253 | 247 | | |
254 | 248 | | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
260 | 253 | | |
261 | 254 | | |
262 | 255 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
311 | 311 | | |
312 | 312 | | |
313 | 313 | | |
314 | | - | |
| 314 | + | |
315 | 315 | | |
316 | 316 | | |
317 | 317 | | |
| |||
0 commit comments