chore: Hide internal public utility APIs by 2010YOUY01 · Pull Request #25963 · apache/datafusion

2010YOUY01 · 2026-10-02T10:07:44Z

Which issue does this PR close?

Closes #.

Rationale for this change

See the previous PR for rationale

docs: Document the convention for internal public APIs #25495

This PR adds #[doc(hidden)] and comments for public APIs that is used as internal utility.

The major categories are

already includes #doc(hidden)
Already explained in the comment that this public API is internal-only
Optimizer utilities, with strict preconditions to use it correctly.

What changes are included in this PR?

comments only

What is the testing strategy for this PR?

Are there any user-facing changes?

github-actions · 2026-10-02T10:12:04Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-expr v55.1.0 (current)
       Built [  28.504s] (current)
     Parsing datafusion-expr v55.1.0 (current)
      Parsed [   0.087s] (current)
    Building datafusion-expr v55.1.0 (baseline)
       Built [  28.459s] (baseline)
     Parsing datafusion-expr v55.1.0 (baseline)
      Parsed [   0.090s] (baseline)
    Checking datafusion-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   2.168s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  60.975s] datafusion-expr
    Building datafusion-physical-expr v55.1.0 (current)
       Built [  26.908s] (current)
     Parsing datafusion-physical-expr v55.1.0 (current)
      Parsed [   0.051s] (current)
    Building datafusion-physical-expr v55.1.0 (baseline)
       Built [  26.890s] (baseline)
     Parsing datafusion-physical-expr v55.1.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.499s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  55.735s] datafusion-physical-expr
    Building datafusion-physical-expr-common v55.1.0 (current)
       Built [  22.530s] (current)
     Parsing datafusion-physical-expr-common v55.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-physical-expr-common v55.1.0 (baseline)
       Built [  22.167s] (baseline)
     Parsing datafusion-physical-expr-common v55.1.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-physical-expr-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.332s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  46.102s] datafusion-physical-expr-common
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  35.289s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.189s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  35.051s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.186s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.995s] 223 checks: 220 pass, 3 fail, 0 warn, 31 skip

--- failure enum_now_doc_hidden: pub enum is now #[doc(hidden)] ---

Description:
A pub enum is now #[doc(hidden)], removing it from the crate's public API.
        ref: https://doc.rust-lang.org/rustdoc/write-documentation/the-doc-attribute.html#hidden
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_now_doc_hidden.ron

Failed in:
  enum Map in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/mod.rs:89

--- failure inherent_method_now_doc_hidden: inherent method #[doc(hidden)] added ---

Description:
A method or associated fn is now #[doc(hidden)], removing it from the crate's public API.
        ref: https://doc.rust-lang.org/rustdoc/write-documentation/the-doc-attribute.html#hidden
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/inherent_method_now_doc_hidden.ron

Failed in:
  AggregateExec::try_optimize_distinct_soft_limit in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/aggregates/mod.rs:1145
  AggregateExec::with_new_aggr_exprs in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/aggregates/mod.rs:1170
  CrossJoinExec::swap_inputs in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/joins/cross_join.rs:190
  NestedLoopJoinExec::swap_inputs in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/joins/nested_loop_join.rs:481
  SortMergeJoinExec::swap_inputs in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/joins/sort_merge_join/exec.rs:352
  HashJoinExec::swap_inputs in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/joins/hash_join/exec.rs:1499
  SymmetricHashJoinExec::check_if_order_information_available in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/663c27b65ea0f15009428dc1731d61b05806ac9c/datafusion/physical-plan/src/joins/symmetric_hash_join.rs:339

--- failure trait_now_doc_hidden: pub trait is now #[doc(hidden)] ---

Description:
A pub trait is now #[doc(hidden)], removing it from the crate's public API.
        ref: https://doc.rust-lang.org/rustdoc/write-documentation/the-doc-attribute.html#hidden
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_now_doc_hidden.ron

Failed in:
  trait JoinHashMapType in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/join_hash_map.rs:110

     Summary semver requires new major version: 3 major and 0 minor checks failed
    Finished [  73.730s] datafusion-physical-plan

2010YOUY01 · 2026-10-02T10:13:04Z

+    /// This is not a public API and is for internal use only; see [API policy] for details.
+    ///
+    /// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
+    #[doc(hidden)]


This API seem like reasonable to be public, however now it requires cooperation from several other optimizer rules, to make it safe. The comments right above can demonstrate it is tricky to use.

Note downstreams can still have access to it, this marker is only a hint for (a) might change often (b) tricky to use it correctly, as explained in the link

This one in particular I would like to maybe add a comment about "might change" or "is tricky to use" if posisble to make it easier to understand the internal marking

codecov-commenter · 2026-10-02T10:33:56Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.77%. Comparing base (e6dfef5) to head (fbbd6e1).
⚠️ Report is 112 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25963      +/-   ##
==========================================
+ Coverage   82.59%   82.77%   +0.18%     
==========================================
  Files        1145     1148       +3     
  Lines      443934   451075    +7141     
  Branches   443934   451075    +7141     
==========================================
+ Hits       366669   373385    +6716     
+ Misses      55009    54937      -72     
- Partials    22256    22753     +497

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb

Thank you @2010YOUY01

alamb · 2026-10-08T19:51:34Z

    /// - the `predicate` expression is not aliased
    /// - the `predicate` expression contains no window function calls
+    ///
+    /// # Public Only for Internal Use:


in fact the comment above says "for internal use in Datafusion only"

alamb · 2026-10-08T19:53:16Z

+    /// This is not a public API and is for internal use only; see [API policy] for details.
+    ///
+    /// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
+    #[doc(hidden)]


This one in particular I would like to maybe add a comment about "might change" or "is tricky to use" if posisble to make it easier to understand the internal marking

2010YOUY01 · 2026-10-10T02:29:23Z

@alamb Thank you for the review! Feedbacks addressed.

chore: Hide internal public utility APIs

be2d525

github-actions Bot added documentation Improvements or additions to documentation logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate labels Oct 2, 2026

github-actions Bot added the auto detected api change Auto detected API change label Oct 2, 2026

2010YOUY01 commented Oct 2, 2026

View reviewed changes

revert questionable ones

9ca4c88

alamb approved these changes Oct 8, 2026

View reviewed changes

2010YOUY01 added 2 commits October 10, 2026 08:18

review: polish wording

175e0ee

review: add clarification on other join type's swap_inputs()

fbbd6e1

2010YOUY01 added this pull request to the merge queue Oct 10, 2026

Merged via the queue into apache:main with commit 4e6e564 Oct 10, 2026
43 checks passed

2010YOUY01 deleted the hide-internal-util branch October 10, 2026 02:42

github-actions Bot added the v56.0.0 label Oct 10, 2026

Navigation Menu

Repository navigation

chore: Hide internal public utility APIs - #25963

chore: Hide internal public utility APIs#25963
2010YOUY01 merged 4 commits into
apache:mainfrom
2010YOUY01:hide-internal-util

2010YOUY01 commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

2010YOUY01 Oct 2, 2026 •

edited

Loading

Uh oh!

alamb Oct 8, 2026

Uh oh!

codecov-commenter commented Oct 2, 2026 •

edited

Loading

Uh oh!

alamb left a comment

Uh oh!

alamb Oct 8, 2026

Uh oh!

alamb Oct 8, 2026

Uh oh!

2010YOUY01 commented Oct 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

2010YOUY01 commented Oct 2, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

2010YOUY01 Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Oct 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Oct 2, 2026 •

edited

Loading

2010YOUY01 Oct 2, 2026 •

edited

Loading

codecov-commenter commented Oct 2, 2026 •

edited

Loading