Skip to content

Add column-group statistics - #118

Open
chopradish wants to merge 5 commits into
ac/stat-selection-observabilityfrom
ac/multi-col-hll
Open

chopradish wants to merge 5 commits into
ac/stat-selection-observabilityfrom
ac/multi-col-hll

Conversation

@chopradish

Copy link
Copy Markdown
Contributor

Problem

Single-column distinct-value estimates cannot capture correlations between grouping keys or composite join keys. As a result, cardinality estimates can be substantially wrong for correlated multi-column data.

Summary of changes

Collect query-local HyperLogLog statistics for bounded logical-schema column prefixes and store them as validated column-group statistics. Extend cardinality estimation to use compatible joint NDVs for grouping and direct composite inner joins, with single-column fallbacks for uncovered keys.

Design

Statistics are keyed by logical schema ordinals and looked up as sets, independent of predicate or grouping-key order. Groups are used only when their population and null-handling metadata are compatible with the input; classic HLL remains an NDV synopsis and is not used for set-intersection estimation.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@codecov-commenter

codecov-commenter commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.41867% with 195 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (ac/stat-selection-observability@22c445c). Learn more about missing BASE report.

Files with missing lines Patch % Lines
optd/core/src/analysis.rs 65.7% 163 Missing ⚠️
optd/core/src/catalog.rs 65.2% 25 Missing ⚠️
optd/connectors/datafusion/src/statistics.rs 91.4% 6 Missing ⚠️
...td/connectors/datafusion/src/runtime_statistics.rs 98.4% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
optd/core/src/lib.rs 86.8% <ø> (ø)
optd/core/src/optimize/join_ordering.rs 96.4% <100.0%> (ø)
optd/core/src/statistics.rs 76.9% <100.0%> (ø)
...td/connectors/datafusion/src/runtime_statistics.rs 99.6% <98.4%> (ø)
optd/connectors/datafusion/src/statistics.rs 91.5% <91.4%> (ø)
optd/core/src/catalog.rs 81.9% <65.2%> (ø)
optd/core/src/analysis.rs 91.3% <65.7%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants