Skip to content

fix: preserve lambda bindings in nested-loop join projection pushdown - #26399

Merged
xudong963 merged 2 commits into
apache:mainfrom
xudong963:xudong963/fix-nlj-lambda-projection
Oct 19, 2026
Merged

xudong963 merged 2 commits into
apache:mainfrom
xudong963:xudong963/fix-nlj-lambda-projection

Conversation

@xudong963

@xudong963 xudong963 commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

A nested-loop join whose lambda captures a column from the other input can fail during physical optimization with unable to unwrap lambda, even though the query executes correctly with physical optimization disabled.

These queries should work with the default optimizer while retaining projection pushdown for expressions that can be evaluated on one join input.

What changes are included in this PR?

  • Keep each lambda and its body in the enclosing high-order function's scope when extracting join-filter expressions. Ordinary sibling expressions, other function arguments, and complete high-order expressions that depend on only one input remain eligible for pushdown.
  • Remap captured columns and rebase lambda parameter indices when moving an expression to an input projection, extending the intermediate filter schema, and pruning unused filter columns. Nested lambdas retain their relative parameter offsets and bindings.
  • Add nested_loop_join_lambda.slt with result and physical-plan regressions.

This change preserves lambda bodies within their enclosing function; it does not introduce hoisting of individual computations from inside a lambda body.

What is the testing strategy for this PR?

The new SQL test covers the original reproducer with either join-input order, mixed predicates, nested lambdas, parameter shadowing, empty/NULL arrays, and extraction of complete high-order functions onto either input. Different input and filter schema widths exercise parameter rebasing. Three EXPLAIN checks verify that ordinary expressions and complete high-order functions are still pushed down.

Are there any user-facing changes?

Nested-loop joins with lambda predicates now plan and execute with the default physical optimizer. There are no public API or configuration changes.

Keep lambda bodies within their high-order functions while retaining single-input expression pushdown. Rebase lambda parameters as expressions move between input projections and join filter schemas, with SQL result and plan regressions.

Generated-by: OpenAI Codex
@xudong963 xudong963 added the bug Something isn't working label Oct 16, 2026
@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Oct 16, 2026
@codecov-commenter

codecov-commenter commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.95652% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.89%. Comparing base (fe612e2) to head (1349bd7).

Files with missing lines Patch % Lines
...sion/physical-optimizer/src/projection_pushdown.rs 86.95% 6 Missing and 3 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #26399   +/-   ##
=======================================
  Coverage   82.89%   82.89%           
=======================================
  Files        1154     1154           
  Lines      458022   458058   +36     
  Branches   458022   458058   +36     
=======================================
+ Hits       379683   379720   +37     
+ Misses      55303    55298    -5     
- Partials    23036    23040    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lonless9

Copy link
Copy Markdown
Contributor

I think this issue is related. #24372

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xudong963 , LGTM!

@xudong963
xudong963 added this pull request to the merge queue Oct 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 19, 2026
@xudong963
xudong963 added this pull request to the merge queue Oct 19, 2026
Merged via the queue into apache:main with commit 6cf7cb5 Oct 19, 2026
42 checks passed
@xudong963
xudong963 deleted the xudong963/fix-nlj-lambda-projection branch October 19, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ProjectionPushdown fails on a nested-loop join whose lambda captures a column from the other input

4 participants