Skip to content

fix: report Parquet Bloom filter pruning metrics accurately - #25822

Open
rgbuilds wants to merge 7 commits into
apache:mainfrom
rgbuilds:issue-18355-bloom-pruning-metrics
Open

rgbuilds wants to merge 7 commits into
apache:mainfrom
rgbuilds:issue-18355-bloom-pruning-metrics

Conversation

@rgbuilds

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

row_groups_pruned_bloom_filter currently reports retained row groups as Bloom filter matches even when Bloom pruning was not evaluated—for example, when Bloom filters are disabled or unavailable, no relevant Bloom statistics were loaded, or predicate evaluation failed.

This makes EXPLAIN ANALYZE suggest that Bloom filters evaluated and matched row groups when they did not participate in the pruning decision. It also displays an idle Bloom pruning metric when all its counters are zero.

A retained row group is not necessarily a Bloom filter match. The metric should describe actual Bloom pruning outcomes.

What changes are included in this PR?

  • Record a Bloom filter match only when the Bloom pruning predicate is successfully evaluated and determines that the row group may match.
  • Leave Bloom pruning counters unchanged when Bloom pruning is unavailable or cannot be evaluated.
  • Continue retaining row groups conservatively when Bloom predicate evaluation fails, while recording the failure in predicate_evaluation_errors.
  • Omit row_groups_pruned_bloom_filter from physical-plan displays when all of its pruning counters are zero.
  • Preserve the registered metric and direct metric lookup; only its accounting and displayed output change.
  • Update affected Rust assertions and SQL logic test snapshots.

The row-group access decisions and query results are unchanged.

What is the testing strategy for this PR?

Added bloom_filter_pruning_error_retains_row_group_without_match, which verifies that a Bloom predicate evaluation error:

  • retains the row group;
  • increments predicate_evaluation_errors;
  • does not increment either the Bloom matched or pruned counter.

The display tests verify that:

  • an idle Bloom pruning metric is omitted from aggregated and full Indent, Graphviz, and PostgreSQL JSON output;
  • unrelated zero-valued metrics remain visible;
  • genuine nonzero Bloom pruning results remain visible.

The following existing test coverage was also run:

  • Parquet Bloom-filter and row-group-filter unit tests;
  • parquet_integration row-group-pruning tests;
  • core_integration explain-analyze tests;
  • physical-plan display tests;
  • affected SQL logic tests for dynamic filtering, dynamic row-group pruning, explain-analyze, limit pruning, and Parquet filter pushdown.

Formatting checks also pass.

Are there any user-facing changes?

Yes. Parquet Bloom filter pruning metrics in EXPLAIN ANALYZE and other physical-plan displays more accurately represent actual Bloom filter evaluation. An entirely idle row_groups_pruned_bloom_filter metric is no longer displayed.

There are no public API changes and no changes to query results or row-group access decisions.

@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Sep 28, 2026
@github-actions github-actions Bot added the auto detected api change Auto detected API change label Oct 1, 2026
@codecov-commenter

codecov-commenter commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.67552% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.73%. Comparing base (6f3f7b7) to head (278c7db).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/display.rs 88.50% 33 Missing ⚠️
datafusion/datasource-parquet/src/opener/mod.rs 88.88% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25822    +/-   ##
========================================
  Coverage   82.72%   82.73%            
========================================
  Files        1147     1147            
  Lines      448179   448437   +258     
  Branches   448179   448437   +258     
========================================
+ Hits       370751   371004   +253     
- Misses      54898    54904     +6     
+ Partials    22530    22529     -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rgbuilds
rgbuilds force-pushed the issue-18355-bloom-pruning-metrics branch from a99b6c7 to b876d88 Compare October 1, 2026 03:10
@rgbuilds

rgbuilds commented Oct 1, 2026

Copy link
Copy Markdown
Author

Rebased onto current main and updated the accounting for the fully matched row-group path introduced by #25854.

Extended the existing scan test to verify Bloom matched/pruned counters for statistics-disabled, mixed, and entirely fully matched cases. Restoring the previous increment makes the mixed-case assertion fail.

Focused Bloom, row-group, display, and affected SQL logic tests pass locally, along with formatting checks.

@rgbuilds

rgbuilds commented Oct 1, 2026

Copy link
Copy Markdown
Author

@xudong963 Thanks for approving the earlier workflow runs. I’ve since updated the Bloom accounting for the fully matched path introduced by #25854 and added regression tests. Fork CI is green on 22cf2a2; the upstream workflows are awaiting approval again. Could you please approve the latest runs when convenient?

@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Oct 1, 2026

@xudong963 xudong963 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THanks for the fix, solid fix

@rgbuilds

rgbuilds commented Oct 2, 2026

Copy link
Copy Markdown
Author

@xudong963 Thanks for reviewing and approving! Could you please merge this when convenient?

@xudong963

Copy link
Copy Markdown
Member

@xudong963 Thanks for reviewing and approving! Could you please merge this when convenient?

Plan to leave it for two days to see if others wanna have a look

@rgbuilds

rgbuilds commented Oct 3, 2026

Copy link
Copy Markdown
Author

@xudong963 Thanks for reviewing and approving! Could you please merge this when convenient?

Plan to leave it for two days to see if others wanna have a look

Makes sense.

@xudong963
xudong963 added this pull request to the merge queue Oct 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 6, 2026
@rgbuilds
rgbuilds force-pushed the issue-18355-bloom-pruning-metrics branch from 22cf2a2 to 278c7db Compare October 6, 2026 15:06
@rgbuilds

rgbuilds commented Oct 6, 2026

Copy link
Copy Markdown
Author

@xudong963 I rebased onto current main and fixed the merge-queue failure in dynamic_row_group_pruning.slt. The relevant tests and extended suite pass locally. The new upstream workflow runs are awaiting approval again — could you please approve them when convenient? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve parquet row group pruning metrics display

3 participants