Skip to content

test: cover CrossJoinExec row metrics across batched inputs - #25530

Merged
kosiew merged 5 commits into
apache:mainfrom
rgbuilds:cursor/cross-join-metrics-tests-6895
Oct 9, 2026
Merged

kosiew merged 5 commits into
apache:mainfrom
rgbuilds:cursor/cross-join-metrics-tests-6895

Conversation

@rgbuilds

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #25333 (CrossJoinExec test coverage only).

Rationale for this change

Join metrics can be incorrect without affecting query results. Existing CrossJoinExec tests did not assert build_input_rows or input_rows across different input batch layouts.

What changes are included in this PR?

Adds Rust test coverage that:

  • Verifies build_input_rows, input_rows, and output_rows for the existing single-batch case.
  • Runs the same 3 × 2 cross join with input batch sizes of 2 and 1.
  • Confirms the row metrics remain 3, 2, and 6 regardless of input batching.

This is a test-only change.

What is the testing strategy for this PR?

cargo fmt --check
cargo test -p datafusion-physical-plan --lib joins::cross_join
cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings

All checks pass locally.

Are there any user-facing changes?

No.

Add a Rust test that asserts build_input_rows, input_rows, and
output_rows stay stable when CrossJoinExec inputs are split into
multiple batches.

Related to apache#25333.
@rgbuilds
rgbuilds force-pushed the cursor/cross-join-metrics-tests-6895 branch from e12556c to 7c49814 Compare September 20, 2026 06:24
@rgbuilds

Copy link
Copy Markdown
Contributor Author

@CuteChuanChuan I picked up the CrossJoinExec portion of #25333 in this PR. Since you outlined the join-metrics test approach, could you take a look at whether these batch-size cases and metric assertions cover what you had in mind? Thanks!

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  36.364s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.132s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  26.260s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.134s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.878s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_physical_plan::TopKDynamicFilters, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/91b395bed2ab60801fc19787831bbb6e751d5677/datafusion/physical-plan/src/topk/mod.rs:187

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  65.560s] datafusion-physical-plan

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Oct 7, 2026

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rgbuilds,

Thanks for adding this regression coverage. The new tests verify the cross-join row metrics across different input batch layouts and address the issue requirements. I have one optional suggestion to make the multi-batch setup even more explicit.

assert_eq!(num_rows, 6);
assert_join_metrics!(metrics, 6);
assert_eq!(metric_count(&metrics, "build_input_rows"), 3);
assert_eq!(metric_count(&metrics, "input_rows"), 2);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: could we also assert build_input_batches and input_batches, expecting (2, 1) for batch size 2 and (3, 2) for batch size 1? That would ensure future test setup changes do not accidentally remove the multi-batch coverage while keeping the same row totals.

@codecov-commenter

codecov-commenter commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.27586% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.73%. Comparing base (97c7593) to head (1ff8898).
⚠️ Report is 22 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/joins/cross_join.rs 98.27% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #25530   +/-   ##
=======================================
  Coverage   82.73%   82.73%           
=======================================
  Files        1147     1147           
  Lines      449509   449568   +59     
  Branches   449509   449568   +59     
=======================================
+ Hits       371893   371948   +55     
- Misses      54944    54946    +2     
- Partials    22672    22674    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rgbuilds

rgbuilds commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@rgbuilds,

Thanks for adding this regression coverage. The new tests verify the cross-join row metrics across different input batch layouts and address the issue requirements. I have one optional suggestion to make the multi-batch setup even more explicit.

Thanks @kosiew for the review and suggestion! I’ve added assertions for both batch metrics to ensure the test continues to cover the intended multi-batch scenarios.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rgbuilds,

Thanks for the follow-up. The additional assertions verify that build and probe row counts remain correct across different input batch layouts, while the batch counters reflect the actual number of batches. The changes are limited to tests and address the regression coverage gap.

LGTM!

@kosiew

kosiew commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🚀
@rgbuilds
Thank you for your contribution.

@kosiew
kosiew added this pull request to the merge queue Oct 9, 2026
Merged via the queue into apache:main with commit 816e002 Oct 9, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change physical-plan Changes to the physical-plan crate v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants