Repository navigation
fix: Validate execution.time_zone at SET time - #25788
Conversation
`datafusion.execution.time_zone` was stored as a plain string, so invalid values such as `Asia/Taipei2` or `08:00` were accepted by `SET` and only failed on a later `::TIMESTAMPTZ` cast, while `current_date()` and `current_time()` silently ignored them. Add `ConfigTimeZone`, which checks the value with Arrow's `Tz` parser but keeps the string as written, so `+08` is not normalized to `+08:00` and `arrow_typeof(now())` is unchanged. A rejected value leaves the previous setting in place. Part of apache#17498. Builds on apache#23224.
kumarUjjawal
left a comment
There was a problem hiding this comment.
Thank you @KassaSana
Looks good! Please update this changes in the upgrade guide for 56.0.0
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25788 +/- ##
==========================================
- Coverage 82.62% 82.61% -0.01%
==========================================
Files 1147 1147
Lines 445238 445190 -48
Branches 445238 445190 -48
==========================================
- Hits 367872 367815 -57
- Misses 55044 55050 +6
- Partials 22322 22325 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for the review @kumarUjjawal! Added the 56.0.0 upgrade guide entry in 0e4254d. |
Close the code fence left open in the time_zone migration example so the following section is not swallowed into the code block.
|
@KassaSana can you resolve the conflicts |
…nfig # Conflicts: # datafusion/functions/src/datetime/from_unixtime.rs
|
Thanks @kumarUjjawal, merged main and fixed the conflict in |
kumarUjjawal
left a comment
There was a problem hiding this comment.
Thank you @KassaSana
Looks good 👍
Which issue does this PR close?
datafusion.execution.time_zoneonly.Rationale for this change
datafusion.execution.time_zoneis stored as a plain string, so an invalid value such asSET TIME ZONE = 'Asia/Taipei2'(or'08:00','+08:00:00') is accepted and shown bySHOW TIME ZONE. The error only appears later, on the first::TIMESTAMPTZcast. Worse,current_date()andcurrent_time()parse the value with.ok(), so they silently ignore a bad time zone and return results as if none was set.PostgreSQL and Spark (
spark.sql.session.timeZoneis checked withisValidTimezoneinSQLConf) both reject an invalid time zone when it is set.This picks up #23224 by @Probablism, which was closed as stale, and answers its open review questions:
timestamps.sltquestion): fix: validate time zone config before applying changes #23224 stored a parsedTz, so+08came back as+08:00andarrow_typeof(now())changed. Here the value is checked withTz::from_strbut stored exactly as written, so the only change users see is that invalid values are rejected.timestamps.sltis unchanged, andset_variable.sltnow checks thatSHOW TIME ZONEreturns+08.AESTin tests:AESTwas never a valid Arrow time zone. The tests that used it only passed the string through without parsing it, so they now useAustralia/Sydney.What changes are included in this PR?
ConfigTimeZoneindatafusion/common/src/config.rs. It can only be built throughFromStr, which checks the value with Arrow'sTzparser and keeps the original string. It providesas_str()andDisplay.ExecutionOptions::time_zonefromOption<String>toOption<ConfigTimeZone>. ItsConfigFieldimpl parses before assigning, followingDFParquetStatistics, so a rejected value leaves the previous value (orNone) in place, andRESETrestoresNone.now,current_date,current_time,to_timestamp*,from_unixtime, Sparkcast/date_trunc) to useas_str().to_timestamp_invalid_execution_timezone_behaviorandto_timestamp_formats_invalid_execution_timezone_behavior, along with their now-unused helper. They put an invalid string straight into the config to test the error at invoke time, and that state can no longer be constructed. The unit test and slt cases below cover the rejection instead.What is the testing strategy for this PR?
set_variable.slt: the invalid values+08:00:00,08:00,08andAsia/Taipei2now fail onSET(through bothSET TIME ZONEandSET datafusion.execution.time_zone), andSHOW TIME ZONEconfirms the previous value is kept. It also checks that+08is shown as written.test_execution_time_zone_validationindatafusion-common: valid values round-trip exactly; invalid values error and keep the previous value; an invalid update of an unset value leaves it unset; nested keys are rejected.timestamps.slt,timestamps_timezone.slt,current_*_timezone.slt,to_timestamp_timezone.slt,from_unixtime_timezone.slt, Spark cast/date_trunc).Commands run (all passing):
cargo test -p datafusion-common --lib config::tests(with and without--features parquet)cargo test --profile=ci --test sqllogictests(all 523 files)cargo test -p datafusion-functions --lib datetimecargo test -p datafusion-spark --lib function::cargo test -p datafusion-sql --libcargo test -p datafusion --test user_defined_integration config_optionscargo test -p datafusion-ffi --features integration-tests --test ffi_udfcargo test -p datafusion --test parquet_integrationcargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningsAre there any user-facing changes?
Yes.
SET TIME ZONE/SET datafusion.execution.time_zonenow fails immediately on an invalid time zone instead of failing on a later query, or being silently ignored bycurrent_date()/current_time().This changes the public field type
ExecutionOptions::time_zonefromOption<String>toOption<ConfigTimeZone>. Code that reads it can use.as_ref().map(ConfigTimeZone::as_str), and code that sets it can useSome("UTC".parse()?). Could a maintainer please add theapi changelabel? I'm happy to add a note to the 56.0.0 upgrade guide if that's wanted.cc @Jefffrey (you reviewed #23224), @kumarUjjawal @alamb (reviewed the other #17498 slices)