Skip to content

fix: Validate execution.time_zone at SET time - #25788

Merged
kumarUjjawal merged 5 commits into
apache:mainfrom
KassaSana:validate-time-zone-config
Oct 3, 2026
Merged

kumarUjjawal merged 5 commits into
apache:mainfrom
KassaSana:validate-time-zone-config

Conversation

@KassaSana

@KassaSana KassaSana commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

datafusion.execution.time_zone is stored as a plain string, so an invalid value such as SET TIME ZONE = 'Asia/Taipei2' (or '08:00', '+08:00:00') is accepted and shown by SHOW TIME ZONE. The error only appears later, on the first ::TIMESTAMPTZ cast. Worse, current_date() and current_time() parse the value with .ok(), so they silently ignore a bad time zone and return results as if none was set.

PostgreSQL and Spark (spark.sql.session.timeZone is checked with isValidTimezone in SQLConf) both reject an invalid time zone when it is set.

This picks up #23224 by @Probablism, which was closed as stale, and answers its open review questions:

  • Normalization (the timestamps.slt question): fix: validate time zone config before applying changes #23224 stored a parsed Tz, so +08 came back as +08:00 and arrow_typeof(now()) changed. Here the value is checked with Tz::from_str but stored exactly as written, so the only change users see is that invalid values are rejected. timestamps.slt is unchanged, and set_variable.slt now checks that SHOW TIME ZONE returns +08.
  • AEST in tests: AEST was never a valid Arrow time zone. The tests that used it only passed the string through without parsing it, so they now use Australia/Sydney.

What changes are included in this PR?

  • Add ConfigTimeZone in datafusion/common/src/config.rs. It can only be built through FromStr, which checks the value with Arrow's Tz parser and keeps the original string. It provides as_str() and Display.
  • Change ExecutionOptions::time_zone from Option<String> to Option<ConfigTimeZone>. Its ConfigField impl parses before assigning, following DFParquetStatistics, so a rejected value leaves the previous value (or None) in place, and RESET restores None.
  • Update the readers (SQL planner, now, current_date, current_time, to_timestamp*, from_unixtime, Spark cast / date_trunc) to use as_str().
  • Remove to_timestamp_invalid_execution_timezone_behavior and to_timestamp_formats_invalid_execution_timezone_behavior, along with their now-unused helper. They put an invalid string straight into the config to test the error at invoke time, and that state can no longer be constructed. The unit test and slt cases below cover the rejection instead.

What is the testing strategy for this PR?

  • set_variable.slt: the invalid values +08:00:00, 08:00, 08 and Asia/Taipei2 now fail on SET (through both SET TIME ZONE and SET datafusion.execution.time_zone), and SHOW TIME ZONE confirms the previous value is kept. It also checks that +08 is shown as written.
  • test_execution_time_zone_validation in datafusion-common: valid values round-trip exactly; invalid values error and keep the previous value; an invalid update of an unset value leaves it unset; nested keys are rejected.
  • Existing time zone suites pass unchanged (timestamps.slt, timestamps_timezone.slt, current_*_timezone.slt, to_timestamp_timezone.slt, from_unixtime_timezone.slt, Spark cast/date_trunc).

Commands run (all passing):

  • cargo test -p datafusion-common --lib config::tests (with and without --features parquet)
  • cargo test --profile=ci --test sqllogictests (all 523 files)
  • cargo test -p datafusion-functions --lib datetime
  • cargo test -p datafusion-spark --lib function::
  • cargo test -p datafusion-sql --lib
  • cargo test -p datafusion --test user_defined_integration config_options
  • cargo test -p datafusion-ffi --features integration-tests --test ffi_udf
  • cargo test -p datafusion --test parquet_integration
  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

Yes. SET TIME ZONE / SET datafusion.execution.time_zone now fails immediately on an invalid time zone instead of failing on a later query, or being silently ignored by current_date() / current_time().

This changes the public field type ExecutionOptions::time_zone from Option<String> to Option<ConfigTimeZone>. Code that reads it can use .as_ref().map(ConfigTimeZone::as_str), and code that sets it can use Some("UTC".parse()?). Could a maintainer please add the api change label? I'm happy to add a note to the 56.0.0 upgrade guide if that's wanted.

cc @Jefffrey (you reviewed #23224), @kumarUjjawal @alamb (reviewed the other #17498 slices)

`datafusion.execution.time_zone` was stored as a plain string, so invalid
values such as `Asia/Taipei2` or `08:00` were accepted by `SET` and only
failed on a later `::TIMESTAMPTZ` cast, while `current_date()` and
`current_time()` silently ignored them.

Add `ConfigTimeZone`, which checks the value with Arrow's `Tz` parser but
keeps the string as written, so `+08` is not normalized to `+08:00` and
`arrow_typeof(now())` is unchanged. A rejected value leaves the previous
setting in place.

Part of apache#17498. Builds on apache#23224.
Copilot AI lite review requested due to automatic review settings September 27, 2026 03:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added sql SQL Planner core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate functions Changes to functions implementation ffi Changes to the ffi crate spark labels Sep 27, 2026

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @KassaSana

Looks good! Please update this changes in the upgrade guide for 56.0.0

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Sep 27, 2026
@codecov-commenter

codecov-commenter commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.14286% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.61%. Comparing base (6c91773) to head (3b17f87).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/common/src/config.rs 96.15% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25788      +/-   ##
==========================================
- Coverage   82.62%   82.61%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      445238   445190      -48     
  Branches   445238   445190      -48     
==========================================
- Hits       367872   367815      -57     
- Misses      55044    55050       +6     
- Partials    22322    22325       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 27, 2026
@KassaSana

Copy link
Copy Markdown
Contributor Author

Thanks for the review @kumarUjjawal! Added the 56.0.0 upgrade guide entry in 0e4254d.

@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Sep 28, 2026
Close the code fence left open in the time_zone migration example so the
following section is not swallowed into the code block.
@kumarUjjawal

Copy link
Copy Markdown
Contributor

@KassaSana can you resolve the conflicts

…nfig

# Conflicts:
#	datafusion/functions/src/datetime/from_unixtime.rs
@github-actions github-actions Bot added the physical-expr Changes to the physical-expr crates label Oct 2, 2026
@KassaSana

Copy link
Copy Markdown
Contributor Author

Thanks @kumarUjjawal, merged main and fixed the conflict in from_unixtime.rs. I also had to tweak a test that came in with #25668 (scalar_function.rs) since it was still setting time_zone as a plain String. Should be good now.

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @KassaSana

Looks good 👍

@kumarUjjawal
kumarUjjawal added this pull request to the merge queue Oct 3, 2026
Merged via the queue into apache:main with commit b550f59 Oct 3, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation ffi Changes to the ffi crate functions Changes to functions implementation physical-expr Changes to the physical-expr crates spark sql SQL Planner sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants