Skip to content

feat(parquet): prune explicitly ordered INT96 timestamps - #25987

Open
Seanium wants to merge 1 commit into
apache:mainfrom
Seanium:int96-timestamp-pruning
Open

Seanium wants to merge 1 commit into
apache:mainfrom
Seanium:int96-timestamp-pruning

Conversation

@Seanium

@Seanium Seanium commented Oct 3, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Part of #25484. Depends on the Arrow StatisticsConverter INT96 support in apache/arrow-rs#11348 and a compatible parquet release; this draft is not ready to merge until that dependency is available.

Rationale for this change

Files carrying explicit INT96 timestamp ordering can safely support timestamp pruning, but DataFusion currently rejects their bounds.

What changes are included in this PR?

  • Trust INT96 bounds only when the footer explicitly contains INT96_TIMESTAMP_ORDER; missing, legacy, and unknown orders remain untrusted.
  • Use the reader's configured INT96 unit and timezone for file statistics. A table-schema cast alone does not authorize direct decoding at that resolution.
  • Preserve conservative fallback and null counts when bounds are unavailable or incompatible.

What is the testing strategy for this PR?

statistics_order_tests writes and reads real INT96 files, then checks file, row-group, page, and runtime pruning across all timestamp units and timezones. It also verifies legacy footer behavior and overflowing bounds. int96_format_statistics_use_configured_resolution covers both production statistics-inference paths and distinguishes configured decoding from a table-schema cast.

The integration has been validated locally with the Arrow prerequisite applied to parquet 60.0.0, using released Arrow 60 crates. The dependency override is not included in this PR.

Are there any user-facing changes?

Newly written files with explicit INT96 timestamp ordering can be pruned. Existing Spark/Hive files without this metadata do not gain pruning retroactively. Public APIs remain compatible.

@github-actions github-actions Bot added core Core DataFusion crate datasource Changes to the datasource crate labels Oct 3, 2026
@Seanium
Seanium marked this pull request as ready for review October 3, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant