fix(array): convert a date scalar into a timestamp instead of re-labelling it

[lukekim]

What was wrong

Scalar::cast sent every cast whose target is an extension type through that target's storage type and then re-labelled the result:

if let Some(ext_dtype) = target_dtype.as_extension_opt() {
    let cast_storage_scalar_value = self.cast(ext_dtype.storage_dtype())?.into_value();
    return Scalar::try_new(target_dtype.clone(), cast_storage_scalar_value);
}

That is sound for a source that carries no interpretation of its own. It is not sound for a source that is itself an extension type: vortex.date and vortex.timestamp both count from the epoch but in different units, so the number is not transferable between them.

The path that reaches it is file-statistics pruning. A scan falsifies cast(col as timestamp) > lit into cast(max(col) as timestamp) <= lit, binds max(col) to the file's statistic and casts that scalar to decide whether to read the file at all — vortex-file/src/pruning.rs::can_prune_file_stats. Extension's CastKernel converts the array, so the rows are handled; the statistic is not.

Both storage layouts went wrong, in opposite ways:

source	storage	result before
vortex.date[days]	i32	storage cast refused → the scan fails: No CastReduce to cast constant array from vortex.date[days](i32?) to vortex.timestamp[ns](i64?)
vortex.date[ms]	i64, same as timestamp[ns]	storage cast succeeds and returns the millisecond count as a nanosecond one — an instant 10^6 too small

The second is the serious one. 2024-03-01 came back as 1970-01-01T00:28:29.2512Z, the falsifier concluded the file could hold no matching row, and the file was pruned. The query returned zero rows with no error.

What this does

Lets an extension source convert itself. Scalar::cast no longer re-labels through the target's storage type when the source is an extension type; ExtScalar::cast performs the vortex.date → vortex.timestamp rescale, and refuses any other pair of extension types rather than re-labelling one as the other. vortex.timestamp[ms] → vortex.timestamp[ns] now errors, matching what the array kernel already does (cast_different_ext_dtype), instead of silently returning a wrong instant.

The rescale is the one Extension's CastKernel already applied to arrays, moved into extension::datetime::cast so both callers share it. That sharing is the point: a file is pruned on the scalar conversion and its surviving rows are filtered on the array one, so the two disagreeing is what drops rows.

Evidence

Before, on a four-row Vortex file with event_date DATE (date[days]), through spiceai/spiceai's DataFusion adapter:

SELECT count(*) FROM events WHERE CAST(event_date AS TIMESTAMP) > TIMESTAMP '2024-01-15 12:00:00';

ERROR: External error: Failed to read Vortex file: date_cast/…vortex:
  No CastReduce to cast constant array from vortex.date[days](i32?) to vortex.timestamp[ns](i64?)

The same file with a date[ms] column and the row 2024-03-01, which matches:

+---------------------+
| event_date          |
+---------------------+
+---------------------+     <- empty; the file was pruned

After, both queries are right — count(*) = 1, and:

+---------------------+
| event_date          |
+---------------------+
| 2024-03-01T00:00:00 |
+---------------------+

And the scalar cast directly:

before:  date[ms] -> ts[ns]   = Ok("1970-01-01T00:28:29.2512Z")
         date[days] -> ts[ns] = Err(cannot cast extension dtype with id vortex.date and storage type i32? to i64?)
after:   date[ms] -> ts[ns]   = Ok("2024-03-01T00:00:00Z")
         date[days] -> ts[ns] = Ok("2024-03-01T00:00:00Z")

Tests

vortex-array/src/scalar/typed_view/extension/tests.rs:

• test_ext_scalar_cast_date_to_timestamp — both storage layouts land on the right instant.
• test_ext_scalar_cast_date_to_timestamp_matches_the_array_kernel — every date/timestamp unit pair, asserting the scalar cast and the array kernel agree value for value, and that the scalar cast refuses whatever the kernel refuses. This is the property the pruning path depends on.
• test_ext_scalar_cast_between_timestamp_units_is_refused — the re-label that used to succeed.

All three fail on this branch's parent and pass on it. cargo test -p vortex-array is green (3001 passed); vortex-file, vortex-scan and vortex-arrow are unchanged, and vortex-layout's 7 layouts::table::tests failures reproduce on the parent commit (they need an installed vortex-io runtime).

SPICE_PATCHES.md gets a row, and row 7 — the array-side half of this cast — gets the Verify command it was missing.

Reported as spiceai/spiceai#13624.

[Copilot]

🟡 Changes recommended

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes date-scalar-to-timestamp conversion so file-statistics pruning matches row-level array casting.

Changes:

• Routes extension scalar casts through extension-aware conversion.
• Shares checked date-to-timestamp scaling between scalar and array paths.
• Adds regression tests and patch tracking.

Checks: Static review only; tests were not run.

File summaries

File	Description
vortex-array/src/scalar/cast.rs	Dispatches extension sources correctly.
vortex-array/src/scalar/typed_view/extension/mod.rs	Implements scalar temporal conversion.
vortex-array/src/scalar/typed_view/extension/tests.rs	Adds scalar and array agreement tests.
vortex-array/src/extension/datetime/mod.rs	Exposes shared conversion helpers.
vortex-array/src/extension/datetime/cast.rs	Centralizes checked temporal scaling.
vortex-array/src/arrays/extension/compute/cast.rs	Reuses shared conversion logic.
SPICE_PATCHES.md	Records verification commands.

Review details

Suppressed comments (2)

vortex-array/src/scalar/typed_view/extension/tests.rs:384

• Move this test's imports into the module-level import block at tests.rs:4-15, as done in arrays/extension/compute/cast.rs:167-186. Function-scoped imports are inconsistent with the repository's established import placement.

    use crate::IntoArray;
    use crate::arrays::ExtensionArray;
    use crate::arrays::PrimitiveArray;
    use crate::builtins::ArrayBuiltins;
    use crate::executor::VortexSessionExecute;

vortex-array/src/scalar/typed_view/extension/tests.rs:462

• Move these imports into the module-level import block at tests.rs:4-15, consistent with arrays/extension/compute/cast.rs:167-186; imports in this test module should not be scoped to individual test functions.

    use crate::extension::datetime::TimeUnit;
    use crate::extension::datetime::Timestamp;

• Files reviewed: 7/7 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lukekim]

One consequence worth flagging, since it is a deliberate choice rather than an oversight.

Refusing every extension pair other than date → timestamp also refuses a timezone-label-only difference — vortex.timestamp[us,"+00:00"] → vortex.timestamp[us,"UTC"]. Re-labelling was correct there: same unit, same instant, only the tz string differs, and Spice stores tz strings verbatim (Iceberg writes +00:00, an IANA name stays an IANA name).

Two reasons to leave it refused rather than widen the conversion:

• The array kernel already refuses it. CastReduce for Extension requires eq_ignore_nullability on the whole ExtDType, and a tz-string difference fails that, so the pair falls through to CastKernel, which handles only Date → Timestamp. Making the scalar succeed where the array fails puts the two back out of step, which is the property this PR exists to establish.
• A general timestamp → timestamp conversion is not just a rescale. Arrow's Timestamp(unit, None) is a naive local time and Timestamp(unit, Some(tz)) is a UTC instant, so converting between those two is a timezone interpretation, not arithmetic. Getting that right is a larger change than this one and wants its own review.

The cost is bounded and is not correctness. In spiceai/spiceai's crates/vortex/src/persistent/format.rs::scalar_stat_to_df, a refused cast is swallowed into Precision::Absent, so a column whose file dtype differs from the table's declared dtype loses its min/max and therefore some pruning — it does not return a wrong bound. On the same path today a unit mismatch (timestamp[ms] file column under a Timestamp(ns) schema) produces a bound 10^6 too small and DataFusion prunes on it, which is the same defect as the one this PR fixes; Absent is the better answer.

[lukekim]

Pushed 53b1edd0e, which fixes a real gap review found in the first commit.

The conversion checked only that the rescaled value fits i64, and the i64 range is wider than the instants a timestamp can hold. So the two paths still disagreed — at the far end of the range rather than the near one. Casting each value on its own through both paths, before the fix:

days=  2147483647 ->  s: array=ok    scalar=PANIC
days=  2147483647 -> ms: array=ok    scalar=PANIC
days=     3000000 ->  s: array=ok    scalar=ERR(Invalid timestamp scalar: adding span overflowed
                                     timestamp: parameter 'Unix timestamp seconds' is not in the
                                     required range of -377705023201..=253402207200)
days=     2932896 ->  s: array=ok    scalar=ERR(same)
days=      106751 -> ns: array=ok    scalar=ok

i32::MAX days is an ordinary Date32 value, so this is not a synthetic input: a scan would fail casting a file's statistic while the rows it gates converted fine — the same shape of defect as the one this PR fixes.

The PANIC is a second, pre-existing defect. Timestamp::unpack_native built its Jiff span with Span::new().seconds(v), which panics outside Jiff's span range — wider than its timestamp range — so a storage value between the two limits aborted before reaching the checked_add meant to reject it. Reachable from any read of such a value, not only from a cast.

Two changes:

• The conversion is now a DateToTimestamp, built once per cast and carrying the target's representable range (jiff::Timestamp::MIN/MAX in the target unit) alongside the scale, so neither caller can apply the scale without the check. Refusing in the shared code is what keeps the paths in step — widening the scalar side to accept what the array accepts would instead have produced statistics no scalar can hold.
• unpack_native builds its span through the checked TimeUnit::to_jiff_span.

The conformance test now casts one value per array rather than a batch: an array cast fails as a whole, so a batch containing one unconvertible value says nothing about the ones beside it. It covers i32::MAX days, 10000-01-01 and the last representable day, for both date[days] and date[ms] sources across all four target units.

Both guards were checked by mutation. Removing the range check fails the conformance test and the three a_date_beyond_the_last_representable_instant_is_refused cases; restoring the unchecked span constructors fails four of the five an_out_of_range_storage_value_is_an_error_not_a_panic cases with the Jiff panic. cargo test -p vortex-array is green at 3011 passed, and vortex-file, vortex-scan and vortex-arrow are unchanged.

[Copilot]

🟡 Changes recommended

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 3
• Review effort level: Balanced

[lukekim]

For anyone reading the red checks: Rust tests (coverage) is the runner running out of disk, on both commits, and neither failure is this change.

On 0065a343 the test step passed in full and the job died generating the report:

Summary [ 273.901s] 7089 tests run: 7089 passed (2 slow), 557 skipped
##[warning]You are running out of disk space. … Free space left: 57 MB
grcov . --binary-path target/debug/ -s . -t lcov --llvm …
[ERROR] A panic occurred at src/output.rs:267: called `Result::unwrap()` on an `Err` value:
        Os { code: 28, kind: StorageFull, message: "No space left on device" }

On 53b1edd0e the squeeze arrived earlier and a slow test hit its 150s cap:

Summary [ 333.931s] 7099 tests run: 7098 passed (2 slow), 1 timed out, 557 skipped
TIMEOUT [ 150.013s] vortex-sqllogictest::sqllogictests slt::duckdb::duckdb/projection_expression_pushdown.slt
##[warning]You are running out of disk space. … Free space left: 38 MB

projection_expression_pushdown.slt contains no date, timestamp or cast — grep -inE 'date|timestamp|cast' over its 415 lines returns nothing — so it does not reach the code this PR touches. It was already climbing through the SLOW thresholds at 30s/60s/90s/120s before the cap, alongside vortex-fsst tests::fsst_compress_offsets_overflow_i32 at 115s, which is the shape of a starved runner rather than a behaviour change.

License Check and Audit Check (advisories) is RUSTSEC-2026-0258 (h2 unbounded empty DATA frames). This PR touches no manifest and adds no dependency, so it cannot have introduced it.

I'll re-run the coverage job once the in-progress workflow finishes and this becomes possible.

[Copilot]

🟡 Changes recommended

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

vortex-array/src/extension/datetime/cast.rs:172

• This boundary date is mislabeled: 2,932,896 days after 1970-01-01 is 9999-12-31, not 10000-01-01. It is rejected because Jiff's timestamp ceiling falls before that day's midnight, so the current explanation gives the wrong reason for the boundary.

This issue also appears on line 187 of the same file.

        // Jiff stops at the end of year 9999; 2_932_896 days is 10000-01-01.

vortex-array/src/scalar/typed_view/extension/tests.rs:392

• The named boundary is off by one day: the rejected day count in this test is 9999-12-31, not 10000-01-01. Please describe the actual first date whose midnight exceeds Jiff's timestamp range.

This issue also appears on line 407 of the same file.

    // Ordinary dates, then the boundaries: the largest `Date32` value, the first day no
    // timestamp can represent (10000-01-01) and the last one that can. The boundaries are

vortex-array/src/extension/datetime/cast.rs:187

• last_day is 2,932,895 days after the epoch, which is 9999-12-30, not 9999-12-31. The latter day's midnight is already beyond Jiff's timestamp range, so the comment currently contradicts the boundary asserted above.

        // 9999-12-31 fits in seconds through microseconds; nanoseconds run out in 2262.

vortex-array/src/scalar/typed_view/extension/tests.rs:408

• These Date64 cases do not test the boundary described above: they are 10000-01-01 and 9999-12-31T23:59:59, so both are already outside Jiff's timestamp range. Use the same adjacent date boundary as the Date32 cases—9999-12-31T00:00 (rejected) and 9999-12-30T00:00 (accepted)—to verify scalar/array agreement on both sides for millisecond storage.

                253_402_300_800_000,
                253_402_300_799_000,

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

vortex-array/src/extension/datetime/cast.rs:172

• 2_932_896 days after 1970-01-01 is 9999-12-31, not 10000-01-01. This value is rejected because Jiff's timestamp range ends earlier on 9999-12-30, so the current explanation misidentifies both the date and the boundary being tested.

This issue also appears on line 187 of the same file.

        // Jiff stops at the end of year 9999; 2_932_896 days is 10000-01-01.

vortex-array/src/scalar/typed_view/extension/tests.rs:394

• This description does not match both value sets: the day pair straddles Jiff's whole-day boundary, but the millisecond pair (253_402_300_799_000 and 253_402_300_800_000) is entirely beyond Jiff's maximum. Describe these as accepted/rejected and out-of-range cases rather than claiming each source unit contains the first unrepresentable day and the last representable one.

    // Ordinary dates, then the boundaries: the largest `Date32` value, the first day no
    // timestamp can represent (10000-01-01) and the last one that can. The boundaries are
    // where a range check applied to one path and not the other shows up — the array holds
    // the converted value and the scalar's validation rejects it.

vortex-array/src/extension/datetime/cast.rs:187

• last_day is 2_932_895, which denotes 9999-12-30, not 9999-12-31. The latter is already outside Jiff's timestamp range, as the preceding test asserts.

        // 9999-12-31 fits in seconds through microseconds; nanoseconds run out in 2262.

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[lukekim]

Differential check of the validation change, since replacing checked_add on a span with a range comparison could in principle have relaxed more than intended.

Both validations reproduced against jiff 0.2.31 and run over the same 816,068 values per the four stored units — every boundary of Timestamp::storage_range and i64 plus ±2 around each, ~200k xorshift values, and a dense sweep:

probed 816068 values across 4 units

  S:  0 disagreement(s)
  Ms: 0 disagreement(s)
  Us: 0 disagreement(s)
  Ns: 4 disagreement(s)

first disagreements: [(Ns, -9223372036854775808, old=false, new=true), ...]

All four nanosecond disagreements are the same value — i64::MIN, which appears four times in the probe set (as i64::MIN, as the range floor, and through two boundary deltas). So the change accepts exactly one value it did not before, and it is the one it was meant to: 1677-09-21T00:12:43.145224192Z.

Seconds, milliseconds and microseconds are bit-for-bit unchanged, which is the part worth knowing — Timestamp::MAX.as_millisecond() is 253402207200999 while Timestamp::from_millisecond stops at 253402207200000, so those units genuinely needed the wider bound the span path already gave them, and they still have it.

Harness: Span::new().try_*(v).map_or(false, |s| Timestamp::UNIX_EPOCH.checked_add(s).is_ok()) for the old path, (min..=max).contains(&v) over storage_range(unit) for the new one.