refactor: separate compact IN-list pruning threshold from the default cap

[goutamadwant]

Which issue does this PR close?

• Closes Revisit the 20-value lower bound for compact IN-list pruning #24710.

Rationale for this change

The compact representation threshold currently reuses MAX_IN_LIST_SIZE, coupling it to the default maximum eligible list length. These are separate decisions. Small-list performance depends on the statistics batch size and distribution, so this change names the existing boundary independently of the configurable cap without claiming that boundary is universally optimal.

What changes are included in this PR?

• Introduce private MIN_COMPACT_IN_LIST_SIZE = 21, preserving the existing strictly-greater-than-20 boundary.
• Keep the public default cap and zero/over-cap behavior unchanged.
• Add boundary coverage for both IN and NOT IN.
• Extend the string benchmark with one-container cases and small-list homogeneous NOT IN cases. Make NULL-result assertions follow the selected representation.

This retains the ordered-type compact evaluator from #25012. It does not change the evaluator's algorithm or tune the threshold.

What is the testing strategy for this PR?

Boundary tests cover list sizes 1, 2, 4, 8, 20, and 21 with disabled, below-size, exact-size, and larger caps for both IN and NOT IN. The benchmark checks representation choice and pruning results as well as measuring construction/evaluation.

The reported GKE slowdown compared de602a59e0 with d170cbc279, not this refreshed head. The follow-up comparison uses head 65f827dae6 and its base 925d7f8ffd, the same benchmark harness on both, Rust 1.98.1, and separate build targets. It covers 12 large-IN/NULL and NOT-IN control cases, with 50 samples per case and two runs per revision in alternating order after local builds/tests finish.

The two-run average differences range from 1.3% faster to 0.6% slower locally. The three highlighted 256-container IN cases differ by approximately -0.23%, -0.17%, and -0.11%. Both revisions pass all 292 cases of the identical benchmark smoke harness. This does not reproduce the reported 20–44% slowdown on the current patch, but a fresh GKE run of these refs is still needed to compare on that hardware. No evaluator change was made in response to the older report.

Are there any user-facing changes?

No. The default cap, representation boundary, pruning behavior, and public APIs remain unchanged. No performance improvement is claimed.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.15385% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.38%. Comparing base (925d7f8) to head (65f827d).

Files with missing lines	Patch %	Lines
datafusion/pruning/src/pruning_predicate.rs	96.15%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25044      +/-   ##
==========================================
- Coverage   82.38%   82.38%   -0.01%     
==========================================
  Files        1138     1138              
  Lines      434309   434333      +24     
  Branches   434309   434333      +24     
==========================================
+ Hits       357803   357822      +19     
- Misses      54875    54878       +3     
- Partials    21631    21633       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

The benchmark does not show a general regression. Most results are flat. The main thing worth investigating is the ~20–44% slowdown for large IN / IN ... NULL evaluation, especially under 256_containers; meanwhile, NOT IN actually gets noticeably faster in several comparable cases.

Benchmark                                              HEAD       bench-25044    Change
256_containers/in_list/1024                            14.4 µs    20.2 µs        ~40% slower
256_containers/in_list/256                             12.6 µs    18.0 µs        ~42% slower
256_containers/in_list_with_null/1025                  14.0 µs    20.2 µs        ~44% slower
evaluate/in_list/256                                  193.3 µs   231.0 µs        ~20% slower
evaluate/in_list_with_null/257                        192.5 µs   230.7 µs        ~20% slower
evaluate/not_in_list/256                              157.5 µs   142.8 µs        ~9% faster
uniform_singleton/not_in_list/21                       88.8 µs    79.0 µs        ~11% faster
uniform_singleton/not_in_list/1024                    130.3 µs   118.6 µs        ~9% faster

[kosiew]

@goutamadwant
I don't see any difference in the pushed update.
Is this ready for review?

[goutamadwant]

@goutamadwant I don't see any difference in the pushed update. Is this ready for review?

You're right @kosiew there was no new code push. I updated the description with the benchmark investigation; sorry if that wasn't clear. The code is ready for review. I couldn't reproduce the reported large-IN slowdown when comparing the current head and base locally, but a fresh run of those refs on the original GKE runner would help settle the performance concern before merge. let me know!

[kosiew]

benchmark run 2
filtered for slower cases only:

Benchmark                                              HEAD       bench-25044    Change
256_containers/in_list/1024                            13.6 µs    20.4 µs        ~50% slower
256_containers/in_list/256                             12.7 µs    18.3 µs        ~44% slower
256_containers/in_list_with_null/1025                  13.6 µs    20.5 µs        ~51% slower
256_containers/in_list_with_null/257                   12.8 µs    18.3 µs        ~43% slower
evaluate/in_list/256                                  202.5 µs   228.6 µs        ~13% slower
evaluate/in_list_with_null/257                        205.5 µs   228.8 µs        ~11% slower

...faster cases only:

Benchmark                                              HEAD       bench-25044    Change

evaluate/not_in_list/21                                90.9 µs    81.7 µs        ~10% faster
uniform_singleton/not_in_list/256                     109.7 µs    99.6 µs         ~9% faster
uniform_singleton/not_in_list/1024                    126.3 µs   115.7 µs         ~8% faster
long_common_prefix/not_in_list/21                      60.0 µs    45.3 µs        ~25% faster
long_literal_prefix/not_in_list/21                     60.3 µs    47.4 µs        ~21% faster