refactor: deprecate the bundled statistics providers by asolimando · Pull Request #25969 · apache/datafusion

asolimando · 2026-10-02T11:11:16Z

Which issue does this PR close?

Closes Consolidate built-in statistics estimation in one place #25571.

Rationale for this change

StatisticsRegistry::default_with_builtin_providers() registers providers that are not the default estimation and that duplicate or replace what the operators already do in statistics_from_inputs. They can make estimates worse (TPC-H Q14: 14.73 billion rows instead of 73,650, see #25570), and fixes in the operators do not reach their users. Estimation improvements belong in the operators; StatisticsRegistry stays as the extension point for user-defined providers.

What changes are included in this PR?

default_with_builtin_providers() and all seven bundled providers are deprecated. Their code is unchanged.
statistics_registry.slt and the join_reorder example use user-defined providers instead of bundled ones.
dfbench statistics reports the operators' own estimates instead of the bundled providers' estimates.

Follow-ups: moving the Filter distinct count survival model into FilterExec (#26052); multi-key join estimation is tracked in #21583.

What is the testing strategy for this PR?

statistics_registry.slt passes unchanged; the join_reorder example still flips the build side.

Are there any user-facing changes?

Deprecations only, see the 56.0.0 upgrade guide.

Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

…mates Rename `StatisticsRegistry::default_with_builtin_providers` to `with_experimental_providers` and deprecate the old name. The experimental set holds `FilterStatisticsProvider` and `JoinStatisticsProvider`. Both start from the operator's `statistics_from_inputs` and replace only the values their technique estimates. `JoinStatisticsProvider` only replaces the row count of multi-key inner equi-joins with the product of per-key distinct counts, and no longer falls back to the Cartesian product. `FilterStatisticsProvider` applies the survival model to the input distinct count, capped at the operator's. Deprecate the providers that duplicate operator estimates: Projection, Passthrough, Aggregate, Limit and Union. Closes apache#25571.

github-actions · 2026-10-02T11:19:52Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-ffi v55.1.0 (current)
       Built [  59.318s] (current)
     Parsing datafusion-ffi v55.1.0 (current)
      Parsed [   0.059s] (current)
    Building datafusion-ffi v55.1.0 (baseline)
       Built [  53.784s] (baseline)
     Parsing datafusion-ffi v55.1.0 (baseline)
      Parsed [   0.061s] (baseline)
    Checking datafusion-ffi v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.247s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 115.028s] datafusion-ffi
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  36.387s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.187s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  36.760s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.178s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.635s] 223 checks: 221 pass, 2 fail, 0 warn, 31 skip

--- failure type_marked_deprecated: #[deprecated] added on type ---

Description:
A type is now #[deprecated]. Downstream crates will get a compiler warning when using this type.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_marked_deprecated.ron

Failed in:
  Struct ProjectionStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:662
  Struct JoinStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:863
  Struct PassthroughStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:709
  Struct LimitStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:993
  Struct FilterStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:598
  Struct AggregateStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:765
  Struct UnionStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:1049

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_physical_plan::operator_statistics::StatisticsRegistry::default_with_builtin_providers in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:405

     Summary semver requires new minor version: 0 major and 2 minor checks failed
    Finished [  75.462s] datafusion-physical-plan
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  93.498s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  94.198s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.016s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.106s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 190.388s] datafusion-sqllogictest

codecov-commenter · 2026-10-02T11:36:01Z

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.68%. Comparing base (c0e872f) to head (45db23b).

Files with missing lines	Patch %	Lines
datafusion/sqllogictest/src/test_context.rs	90.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25969      +/-   ##
==========================================
- Coverage   82.69%   82.68%   -0.02%     
==========================================
  Files        1147     1147              
  Lines      447348   447364      +16     
  Branches   447348   447364      +16     
==========================================
- Hits       369936   369890      -46     
- Misses      54999    55058      +59     
- Partials    22413    22416       +3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gabotechs · 2026-10-05T06:05:20Z

+`StatisticsRegistry::default_with_builtin_providers()` is deprecated in favor of
+`StatisticsRegistry::with_experimental_providers()`. Most of the providers it


For these providers, rather than just moving them to "experimental", it'd be nice to clarify their future and direction.

For example, if they do are experimental, what's that scope of the experiment, is someone investing time in experimenting with them so that at some point they are eligible to replace their equivalent ExecutionPlan::partition_statistics() implementation?

To be honest I don't really see much benefit in having two different concurrent implementations of the same statistics estimation code committed to main in order to keep one implementation as an "experiment". Typically experiments are something confined to a developer's PR until it proves an improvement via benchmarks, and once the improvement is qualified it replaces the previous code, rather than co-living as a mutually exclusive code path in main.

Thanks for your feeback, @gabotechs.

I was considering a two-phased approach, as the Join and Filter providers had improvements we want to keep, but I agree it's better to file issues for later than to keep duplicate code:

Multi-key join estimation is already tracked in Use exponential decay for multi-column join selectivity estimation #21583 (I added a comment)

For NDV estimation after filters, I have filed Estimate distinct counts after a filter with the survival formula in FilterExec #26052

NDV-based improvements are not covered by the existing benchmarks, as they don't provide NDV yet, but with the work of @Rich-T-kid (#25576) we might get there eventually and be able to merge those, once we can show the numbers get better.

I have updated the PR to deprecate the bundled (formerly known as "built-in") providers entirely. Please take another look when you have time, thanks!

I was considering a two-phased approach, as the Join and Filter providers had improvements we want to keep

👍 My impression is that we could fold those into the built-in statistics of FilterExec and HashJoinExec, WDYT? (for another PR of course)

I realized there is some overlap with #25570, but I guess that will land soon and we should wait a bit for this one

I'd not worry too much, whatever lands first is fine I think

gabotechs · 2026-10-05T13:29:05Z

+    #[deprecated(
+        since = "56.0.0",
+        note = "the bundled providers are deprecated; the statistics walk uses each operator's `statistics_from_inputs` when no provider matches"
+    )]
+    #[expect(deprecated)]


👍 looks good. Not sure which tickets are already open for this, but what comes to mind is:

Fold the improvements brought by these providers into the built-in partition_statistics method.

Remove the now redundant providers

The two follow-up issues are listed here, which follow your proposal 1. to fold improvements in the built-in estimation.

For 2. I have filed #26055 following https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines so we won't forget.

gabotechs · 2026-10-05T13:33:14Z

+            // Replaces the join estimate with the Cartesian product
+            let join_provider = ClosureStatisticsProvider::with_matches(
+                |plan| plan.downcast_ref::<HashJoinExec>().is_some(),


I see this is still using a ClosureStatisticsProvider. I see this is mainly just for testing that the registry still works and actually listens to what the ClosureStatisticsProvider has to say.

If it's just that, all good, but let me know if missed something.

Yes, I trimmed the example down to exactly that, just testing the registry works, you got it right

gabotechs · 2026-10-05T13:33:31Z

Thanks @asolimando!

…viders

github-actions Bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate labels Oct 2, 2026

github-actions Bot added the auto detected api change Auto detected API change label Oct 2, 2026

gabotechs reviewed Oct 5, 2026

View reviewed changes

refactor: deprecate all bundled statistics providers

7a6782c

asolimando mentioned this pull request Oct 5, 2026

Estimate distinct counts after a filter with the survival formula in FilterExec #26052

Open

asolimando changed the title ~~refactor: deprecate statistics providers that duplicate operator estimates~~ refactor: deprecate the bundled statistics providers Oct 5, 2026

asolimando added 4 commits October 5, 2026 14:45

refactor: keep the deprecated Filter and Join providers unchanged

5c9b6b0

docs: simplify the join_reorder statistics example to catalog statistics

2b3c10d

docs: drop issue link from the statistics providers upgrade note

b74c6b5

test: address review comments on the statistics registry test

7811713

gabotechs approved these changes Oct 5, 2026

View reviewed changes

asolimando mentioned this pull request Oct 5, 2026

Remove the deprecated bundled statistics providers #26055

Open

gabotechs added this pull request to the merge queue Oct 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026

gabotechs added this pull request to the merge queue Oct 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 5, 2026

Merge branch 'main' into asolimando/25571-experimental-statistics-pro…

45db23b

…viders

gabotechs enabled auto-merge October 6, 2026 05:48

gabotechs added this pull request to the merge queue Oct 6, 2026

Merged via the queue into apache:main with commit bb24345 Oct 6, 2026
43 checks passed

github-actions Bot added the v56.0.0 label Oct 6, 2026

asolimando deleted the asolimando/25571-experimental-statistics-providers branch October 6, 2026 07:39

		`StatisticsRegistry::default_with_builtin_providers()` is deprecated in favor of
		`StatisticsRegistry::with_experimental_providers()`. Most of the providers it

Conversation

asolimando commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabotechs commented Oct 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

asolimando commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

codecov-commenter commented Oct 2, 2026 •

edited

Loading