Skip to content

fix: infer LIMIT and OFFSET parameter types - #25364

Open
1fanwang wants to merge 7 commits into
apache:mainfrom
1fanwang:1fannnw/limit-parameter-types-2e62cbd8
Open

1fanwang wants to merge 7 commits into
apache:mainfrom
1fanwang:1fannnw/limit-parameter-types-2e62cbd8

Conversation

@1fanwang

@1fanwang 1fanwang commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Clients preparing queries need usable parameter types. Bare LIMIT/OFFSET parameters should default to Int64, but an explicit CAST($1 AS BIGINT) must leave its input unknown because the cast also accepts strings. Reporting Int64 incorrectly tells clients they must supply an integer.

Previously, optimized plans lost the bare parameter's default. Looking through every Int64 cast restored it but incorrectly typed explicit cast inputs.

What changes are included in this PR?

The analyzer records bare LIMIT/OFFSET parameter fields before coercion, preserving a type inferred elsewhere. Lookup does not unwrap user-written casts. This keeps explicit cast inputs unknown and allows Int32 binding followed by Int64 LIMIT coercion.

Coverage uses the existing parameter harnesses, optimizer integration, prepare.slt, and named/metadata DataFrame execution tests.

What is the testing strategy for this PR?

Testing Done

macOS arm64, Rust 1.98.1, with the repository's test-data submodules initialized.

Before the explicit-cast fix, the BIGINT cases failed on the production code from b66d80b. The regression then lived in core integration:

cargo test --profile ci -p datafusion --test core_integration test_limit_offset_parameters_leave_cast_inputs_unresolved -- --nocapture
explicit cast SQL: SELECT $1 AS value LIMIT CAST($1 AS BIGINT)
  left: {"$1": Some(Int64)}
 right: {"$1": None}

explicit cast SQL: SELECT $1 AS value FROM (VALUES (1), (2)) AS t(v) OFFSET CAST($1 AS BIGINT)
  left: {"$1": Some(Int64)}
 right: {"$1": None}

After the fix, the real CLI accepts string inputs in LIMIT, OFFSET and nested explicit casts, and paging returns the requested row:

cargo run --profile ci -p datafusion-cli -- --quiet -c "PREPARE page AS SELECT value FROM (VALUES (10), (20), (30)) AS t(value) ORDER BY value LIMIT \$1 OFFSET \$2; EXECUTE page(1, 1); DEALLOCATE page; PREPARE cast_page AS SELECT \$1 AS value LIMIT CAST(\$1 AS BIGINT); EXECUTE cast_page('1'); DEALLOCATE cast_page; PREPARE cast_offset AS SELECT \$1 AS value FROM (VALUES (1), (2)) AS t(v) OFFSET CAST(\$1 AS BIGINT); EXECUTE cast_offset('1'); DEALLOCATE cast_offset; PREPARE nested_cast AS SELECT \$1 AS value FROM (SELECT 1 LIMIT CAST(\$1 AS BIGINT)) AS t; EXECUTE nested_cast('1'); DEALLOCATE nested_cast;"
+-------+
| value |
+-------+
| 20    |
+-------+
+-------+
| value |
+-------+
| 1     |
+-------+
+-------+
| value |
+-------+
| 1     |
+-------+
+-------+
| value |
+-------+
| 1     |
+-------+

Are there any user-facing changes?

Bare LIMIT/OFFSET parameters default to Int64 before and after optimization. Explicit casts leave their input parameter types unresolved.

@github-actions github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate labels Sep 16, 2026
@codecov-commenter

codecov-commenter commented Sep 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.72%. Comparing base (416002a) to head (68665c5).
⚠️ Report is 59 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/optimizer/src/analyzer/type_coercion.rs 88.88% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25364      +/-   ##
==========================================
+ Coverage   82.59%   82.72%   +0.12%     
==========================================
  Files        1145     1147       +2     
  Lines      444096   448217    +4121     
  Branches   444096   448217    +4121     
==========================================
+ Hits       366806   370781    +3975     
+ Misses      55021    54898     -123     
- Partials    22269    22538     +269     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@1fanwang
1fanwang force-pushed the 1fannnw/limit-parameter-types-2e62cbd8 branch from 7666e87 to f12c31d Compare September 24, 2026 06:26
Preserve known parameter fields before applying row-count defaults and keep incomplete PREPARE signatures deferred.

Fixes apache#15978

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang force-pushed the 1fannnw/limit-parameter-types-2e62cbd8 branch from f12c31d to 3104b5b Compare September 24, 2026 06:58

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1fanwang,

Thanks for working on this. The change looks good overall. I left one non-blocking suggestion to add nested-query coverage for LIMIT/OFFSET parameter inference.

Comment thread datafusion/core/tests/sql/select.rs Outdated

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @1fanwang , here are some suggestions

Comment thread datafusion/expr/src/logical_plan/plan.rs
Comment thread datafusion/core/tests/sql/select.rs Outdated
coerce_limit_expr cast-wrapped untyped LIMIT/OFFSET placeholders,
hiding them from get_parameter_fields's row-count-default pass, which
only matches a direct Limit-node placeholder, so optimized
get_parameter_types() returned None instead of Some(Int64). Leave bare
untyped placeholders unwrapped.

Add a prepare.slt case mixing a typed filter with an untyped LIMIT.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Oct 3, 2026

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @1fanwang , 1 suggestion

Comment thread datafusion/optimizer/src/analyzer/type_coercion.rs Outdated
Signed-off-by: 1fanwang <1fannnw@gmail.com>

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only request left: could we minimize the tests? Most of the 371 lines fit existing harnesses:

  • metadata, type-conflict and field-metadata cases → ParameterTest / ParameterTestWithMetadata in datafusion/sql/tests/cases/params.rs (it checks types, binding and the Prepare fields)
  • EXECUTE result cases → prepare.slt
  • the analyzer→bind→optimize guard → datafusion/optimizer/tests/optimizer_integration.rs

Only the named-parameter (DataFrame API) case needs to stay in core_integration. While moving them, please drop the leftover println! calls

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1fanwang,

Thanks for working on this. I found one issue with explicit BIGINT casts that needs to be addressed before this can merge.

Comment thread datafusion/expr/src/logical_plan/plan.rs Outdated
Record bare row-count parameter fields before analyzer coercion instead of inferring their inputs through arbitrary Int64 casts. Preserve types inferred elsewhere and string binding through explicit BIGINT casts.

Signed-off-by: 1fanwang <1fannnw@gmail.com>

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1fanwang,

Thanks for the follow-up. The explicit BIGINT issue is fixed, and I don't see any remaining merge-blocking problems. I have one non-blocking cleanup suggestion below.

Comment thread datafusion/core/tests/sql/select.rs Outdated
}

#[tokio::test]
async fn test_limit_parameter_binding_with_inferred_type() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is still quite a bit of parameter-test setup duplicated here. Could you move the planning and binding assertions into the existing parameter harnesses where they fit, while keeping the analyzer/optimizer and execution-specific cases in coverage that actually exercises those phases?

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sql SQL Planner sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Placeholder datatype not inferred after LIMIT clause

4 participants