phillipleblanc · Sep 10, 2026 · Sep 9, 2026 · Sep 9, 2026 · Sep 9, 2026 · Sep 10, 2026
@@ -300,10 +300,15 @@ The project half of the report comes from Spice Cloud, so the command needs an a
   "datasets_total": 0,
   "datasets_unhealthy": [],
   "runtime_error": null,
+  "health_not_reported": null,
   "link": { "connection": {}, "service": {}, "deployment": {} }
 }
 ```
 
+For Cloud Connect and BYOC runtimes, `instances` is empty and `runtime_error` is `null`.
+`health_not_reported` explains why health is unavailable; the table output shows the same reason.
+This does not affect the exit code.
+
 ### `logs`
 
 ```shell

@@ -7,7 +7,20 @@ pagination_next: null
 
 Run Spice - starts the Spice runtime, installing if necessary.
 
-`spice run` is a wrapper around the [`spiced`](./spiced) runtime binary. It installs `spiced` on first use, applies developer-friendly defaults, and forwards arguments after `--` to the runtime. To invoke the runtime directly (for containers, systemd units, or CI), see the [`spiced` reference](./spiced).
+`spice run` is a wrapper around the [`spiced`](./spiced) runtime binary. It applies developer-friendly defaults and forwards arguments after `--` to the runtime. To invoke the runtime directly (for containers, systemd units, or CI), see the [`spiced` reference](./spiced).
+
+### Runtime selection
+
+`spice run` and [`spice version`](./version) look for `spiced` in this order:
+
+1. `$SPICED_PATH`.
+2. Beside the running `spice` binary.
+3. `$HOME/.spice/bin/spiced`, managed by [`spice install`](./install).
+4. Under `sudo`, the invoking user's `~/.spice/bin/spiced`.
+
+`PATH` is not searched. An invalid `$SPICED_PATH` causes an error; otherwise, `spice run` installs
+the runtime if none is found. For a runtime outside the managed install, the CLI reports
+its path and source. [`spice version`](./version) also reports these values.
 
 ### Usage
 

@@ -41,4 +41,16 @@ Learn more about upgrading the Spice CLI and runtime using `spice upgrade` [here
 
 CLI version:     v1.1.0
 Runtime version: v1.1.0+models
- ```
+ ```
+
+For a runtime outside the managed install, the output also includes its path and source:
+
+```shell
+> spice version
+
+CLI version:     v1.1.0
+Runtime version: v1.1.0+models
+Runtime path:    /usr/local/bin/spiced (beside the spice CLI)
+```
+
+JSON output includes `runtime_path` and `runtime_source`, following the same [runtime selection rules](./run#runtime-selection).
@@ -60,6 +60,7 @@ The following parameters are supported:
 | `adbc_driver_options`     | Optional. Semicolon-delimited driver-specific database options (e.g., `key1=value1;key2=value2`).                    |
 | `connection_pool_size`    | Optional. Maximum number of connections in the connection pool. Default: `5`.                                         |
 | `connection_pool_min_idle` | Optional. Minimum number of idle connections to keep open in the pool. Default: `1`.                                 |
+| `query_federation`        | Optional. Controls whether queries are federated to the ADBC source. Values: `enabled`, `disabled`. Default: `enabled`.                |
 
 ## Examples
 

@@ -223,6 +223,9 @@ Write operations require `s3:PutObject` permission on the target S3 bucket in ad
 
 :::
 
+Tables with unsupported formats are skipped. A warning summarizes skipped tables per database;
+debug logs provide each reason. Warnings are rate-limited unless the set of skipped tables changes.
+
 ## Cookbook
 
 There is a [cookbook recipe](https://github.com/spiceai/cookbook/tree/trunk/catalogs/glue) to configure an AWS Glue Data Connector in Spice.
@@ -117,6 +117,7 @@ Generic acceleration metrics are available with the `dataset_acceleration_` pref
 | `cayenne_compaction_duration_ms` | Histogram | ms | Wall-clock time of Cayenne background compaction passes. The histogram's count doubles as the compaction-pass counter. |
 | `cayenne_compaction_memory_pool_bytes` | Gauge | By | Size of the dedicated compaction memory pool carved from the query memory limit (see `cayenne_compaction_memory_fraction`). |
 | `cayenne_compaction_memory_exhausted_total` | Counter | passes | Compaction passes that hit `ResourcesExhausted` on the dedicated compaction memory pool. |
+| `cayenne_delete_main_visibility_downgrade_total` | Counter | deletes | Delete passes that deferred row cleanup to protect concurrent replacement rows. Label: `table`. A sustained increase indicates deferred cleanup under ingest load. |
 
 ### Memory Reconciliation Metrics
 

@@ -638,7 +638,7 @@ datasets:
 
 ## Data Type Support
 
-Cayenne (via Vortex) supports most Arrow data types with the following considerations:
+Cayenne supports most Arrow data types. The [accelerator data type table](../../reference/datatypes/accelerators) lists storage representations and engine comparisons.
 
 ### Fully Supported Types
 
@@ -647,18 +647,25 @@ Cayenne (via Vortex) supports most Arrow data types with the following considera
 - Boolean
 - Utf8 and LargeUtf8 strings
 - Binary and LargeBinary
-- Timestamps (normalized to Microsecond precision)
+- Timestamps, preserving the source unit and timezone
 - Date32 and Date64
 - Lists and FixedSizeLists
 - Maps
 - Structs
 
 ### Automatically Converted Types
 
-| Original Type               | Converted To             | Notes                                         |
-| --------------------------- | ------------------------ | --------------------------------------------- |
-| `Float16`                   | `Float32`                | Automatic conversion for Vortex compatibility |
-| `Timestamp(Nanosecond/...)` | `Timestamp(Microsecond)` | Precision normalized                          |
+| Original Type | Converted To | Notes                                         |
+| ------------- | ------------ | --------------------------------------------- |
+| `Float16`     | `Float32`    | Automatic conversion for Vortex compatibility |
+
+:::note Tables created before v2.2.0
+
+These tables continue to normalize timestamps to microseconds. Preserving the source unit requires
+recreating the table with `mode: file_create` in an empty directory. The
+[`on_schema_change`](../../reference/spicepod/datasets#on_schema_change) setting does not migrate existing timestamps.
+
+:::
 
 ### Unsupported Types
 
@@ -737,6 +744,8 @@ A file-mode Cayenne dataset whose resolved metastore directory falls inside its
 
 Paths are compared after `.`/`..` are collapsed and symlinks are resolved, so neither hides an overlap, and a sibling that merely shares a name prefix (`…/meta` next to `…/metadata`) is not affected. Datasets whose data lives on object storage (for example an S3 Express `cayenne_file_path`) are exempt — the metastore is always local, so it cannot sit inside an object-store data path.
 
+Before recreating or deleting a data directory, Spice checks for `cayenne.db` and its SQLite sidecars, including those belonging to other datasets. Deletion is refused if the directory contains a metastore, links directly to one, or has unreadable entries. Metastore files must reside outside data directories.
+
 ### CPU
 
 Query performance scales with available CPU cores. Vortex's columnar format supports parallel decompression and scanning across multiple threads. Allocate sufficient CPU for:
@@ -779,14 +788,23 @@ Remove `on_conflict` to keep writes on the accelerator, or choose a different [`
 **Durable write-back requirements:**
 
 - The federated source must be **PostgreSQL** (see the warning above).
-- The dataset must declare a **single-column [`acceleration.primary_key`](../../reference/spicepod/datasets#accelerationprimary_key)**. Delivery keys each committed row on that column, and a composite key cannot be expressed as the key filter it delivers with, so a multi-column key is rejected at registration with an error naming the columns rather than registering and then never delivering.
-- The accelerator must be the **sole writer** of the rows it delivers. The upsert is an unconditional `ON CONFLICT ... DO UPDATE` with no compare-and-set against the source, so a second writer mutating the same source row directly can be overwritten by a later delivery.
+- A **single-column [`acceleration.primary_key`](../../reference/spicepod/datasets#accelerationprimary_key)** is required.
+- [`acceleration.mode`](../../reference/spicepod/datasets#accelerationmode) must be **`file`**.
+- No acceleration retention may be configured, to preserve undelivered rows.
+- The accelerator must be the **sole writer** of these source rows; delivery can overwrite changes made directly at the source.
+
+**Supported writes:**
+
+- `INSERT` and `UPDATE` must run inside a [transaction](#transactions).
+- `DELETE` and `MERGE` are not supported.
+
+Delivery is asynchronous. Failed deliveries and temporarily unreadable rows remain pending for retry; an unreadable row does not trigger a source deletion. A persistent [`dataset_acceleration_write_back_pending_keys`](../../features/observability) backlog indicates a delivery problem.
 
-Delivery is asynchronous and does not block accelerator commits: a delivery failure leaves the pending set to grow and retries on the next pass, and only a delivery that has been accepted by the source clears its marker.
+Before a source deletion, writes must stop and pending keys must reach zero **while write-back is enabled**. Write-back can then be disabled and the source rows deleted, with CDC refreshing the accelerator. Disabling write-back resets the gauge without delivering pending rows.
 
 **Requirements and v1 limitations:**
 
-- Write targets must be **accelerator-only, non-partitioned Cayenne datasets**. Other dataset modes route writes to the federated source — where the gate cannot govern them — and are rejected.
+- Write targets must be **non-partitioned Cayenne datasets** configured as accelerator-only or for durable write-back.
 - Only **`INSERT` and `UPDATE`** writes are supported inside a transaction. `DELETE` and `MERGE` are rejected.
 - At most **one write per table** per transaction. Multiple tables may be written in the same transaction and are committed atomically together.
 - Reading a Cayenne table that is not a registered participant (for example, a partitioned table) fails the transaction closed.
@@ -800,10 +818,11 @@ Consider the following limitations when using Spice Cayenne acceleration:
 - **Memory Mode Constraints**: `mode: memory` (fully in-RAM, ephemeral) is supported alongside `mode: file`, but it does not persist any data (the dataset reloads from its source on restart), does not support partitioned tables (`partition_by`), and enforces a hard per-table RAM bound instead of spilling to disk — a breach returns an error rather than growing without limit. Use `mode: file` when persistence across restarts is required.
 - **S3 Express Only**: Standard S3 buckets are not supported for remote storage. Only S3 Express One Zone directory buckets are supported.
 - **Unsupported Data Types**: `Interval`, `Duration`, `FixedSizeBinary`, `Union`, and `RunEndEncoded` types require `unsupported_type_action` configuration.
-- **No Traditional Indexes**: Spice Cayenne does not support explicit index creation via the `indexes` configuration. Vortex's segment statistics and fast random access encodings provide equivalent or better performance for most point lookup workloads.
+- **Indexes**: `indexes` is ignored with a warning, including `unique` indexes. Deduplication requires `primary_key` with [`on_conflict`](../../reference/spicepod/datasets#accelerationon_conflict).
+- **SQL Retention**: [`retention_sql`](../../reference/spicepod/datasets#accelerationretention_sql) runs during maintenance after writes, full refreshes, and CDC checkpoints, independently of periodic retention settings. It is ignored in `mode: memory`, with a warning; matching rows remain queryable.
+- **Time-Based Retention**: [`retention_period`](../../reference/spicepod/datasets#accelerationretention_period) hides expired rows in either mode. Scheduled deletion requires both [`retention_check_enabled: true`](../../reference/spicepod/datasets#accelerationretention_check_enabled) and [`retention_check_interval`](../../reference/spicepod/datasets#accelerationretention_check_interval), which has no default. Without both, a warning is logged and storage is reclaimed only when compaction rewrites affected files.
 - **No MVCC**: Multi-version concurrency control is not yet implemented. Snapshots and time-travel queries are planned for future releases.
-- **Transaction Constraints**: [Transactions](#transactions) support gated `INSERT`/`UPDATE` writes on accelerator-only, non-partitioned Cayenne tables only (no `DELETE`/`MERGE`, one write per table). See [Transactions](#transactions) for the full list.
-- **No `refresh_append_overlap`**: A dataset accelerated by Spice Cayenne that sets [`acceleration.refresh_append_overlap`](../../reference/spicepod/datasets#accelerationrefresh_append_overlap) fails to load, with `Cayenne data accelerator does not yet support refresh_append_overlap. Please remove this configuration`. [`refresh_mode: append`](../../features/data-acceleration/data-refresh) itself is supported — only the overlap window is not, so late-arriving rows behind the high-water mark are missed rather than re-read. The check runs during file-mode initialization, so a `mode: memory` dataset is not rejected.
+- **Transaction Constraints**: [Transactions](#transactions) support `INSERT`/`UPDATE` on non-partitioned, accelerator-only or durable write-back datasets, with one write per table. `DELETE` and `MERGE` are not supported.
 
 ## Example Spicepod
 

@@ -152,6 +152,7 @@ Turso supports query federation, where queries can span multiple data sources. T
 - **Remote databases not supported**: Only local Turso databases (file-based or in-memory) are supported as accelerators. Remote Turso databases using `turso_url` and `turso_auth_token` are not supported in this accelerator context. Remote Turso support will be available when Turso is implemented as a data connector.
 - **Arrow Interval types**: Not supported, as SQLite/libSQL doesn't have a native interval type.
 - **Complex List types**: Only Arrow `List` types of primitive data types are supported; lists with structs are not supported.
+- **Older list encodings**: List values stored without an encoding-version marker read as `NULL` with a warning. Refreshing the acceleration or using a fresh file rewrites them in the supported encoding.
 - **Dictionary and Map types**: Not supported.
 - **Hot-reload federation**: Updating a dataset with Turso acceleration while the Spice Runtime is running (hot-reload) may disable query federation until the runtime is restarted.
 - **ROLLUP and GROUPING**: Advanced grouping features are not supported.

@@ -275,6 +275,8 @@ The ADBC connector maintains a pool of database connections for concurrent query
 
 Both values must be positive integers, and `connection_pool_min_idle` must not exceed `connection_pool_size` — a larger value causes connection pool initialization to fail.
 
+Query cancellation or deadline expiry cancels the driver statement and releases the pooled connection. For BigQuery, this also stops the remote job.
+
 ### Query Pushdown
 
 The ADBC connector pushes SQL operations down to the source database when possible, reducing the amount of data transferred:
@@ -284,7 +286,9 @@ The ADBC connector pushes SQL operations down to the source database when possib
 - **Limit pushdown**: `LIMIT` clauses are applied at the source.
 - **Aggregation pushdown**: `GROUP BY`, `SUM`, `COUNT`, `AVG`, and other aggregations are executed on the source.
 - **Sort pushdown**: `ORDER BY` clauses are applied at the source.
-- **Join pushdown**: Joins between datasets from the same ADBC URI are executed on the remote database.
+- **Join pushdown**: Joins between datasets that reach the same remote engine are executed on the remote database.
+
+Join pushdown requires matching driver, URI, credentials, driver options, and explicit `adbc_catalog`/`adbc_schema` settings. BigQuery tables in different datasets within one project can share a job: dataset qualifiers in `from:` do not need to match, but explicit `adbc_schema` settings do.
 
 No special configuration is required. Pushdown happens automatically when the source database supports the operation.
 

@@ -207,7 +207,7 @@ datasets:
 
 :::warning[Limitations]
 
-- Spice uses DuckDB 1.5.3, which supports DuckLake 1.0. Older DuckLake catalogs require a metadata migration before use — set `ducklake_automatic_migration: true` to perform it on attach (this rewrites catalog metadata and cannot be undone). See [DuckLake migration guide](https://ducklake.select/docs/stable/duckdb/guides/troubleshooting#connecting-to-an-older-ducklake).
+- Spice embeds DuckDB **1.4.4** and supports the catalog schema required by its `ducklake` extension. For older catalogs, `ducklake_automatic_migration: true` performs an irreversible [metadata migration](https://ducklake.select/docs/stable/duckdb/guides/troubleshooting#connecting-to-an-older-ducklake) on attach.
 - The DuckLake DuckDB extension is downloaded at runtime on first use, requiring network connectivity.
 - The `ducklake_connection_string` parameter is required — unlike the catalog connector, it cannot be omitted.
 - Each dataset creates its own DuckDB connection pool. For querying many tables from the same catalog, consider using the [DuckLake Catalog Connector](../catalogs/ducklake) instead, which shares a single connection pool.

@@ -154,7 +154,7 @@ The table below shows the PostgreSQL data types supported, along with the type m
 | `money`         | `Int64`                          |
 | `float4`        | `Float32`                        |
 | `float8`        | `Float64`                        |
-| `numeric`       | `Decimal128`                     |
+| `numeric`       | `Decimal128` <sup>[1](#numeric-precision-and-scale)</sup> |
 | `text`          | `Utf8`                           |
 | `varchar`       | `Utf8`                           |
 | `bpchar`        | `Utf8`                           |
@@ -187,6 +187,14 @@ The Postgres federated queries may result in unexpected result types due to the
 
 :::
 
+### `numeric` precision and scale
+
+`numeric(p,s)` maps to `Decimal128(p,s)`, including negative scales. An omitted scale defaults to
+zero: `numeric(12)` becomes `Decimal128(12,0)`.
+
+Unconstrained `numeric` maps to `Decimal128(38,20)`. Reads fail for values requiring more than
+18 integral or 20 fractional digits. An explicit column precision and scale controls this mapping.
+
 ## Write Support
 
 The PostgreSQL connector supports writing data to PostgreSQL tables using SQL [`INSERT INTO`](../../reference/sql/dml#insert), `UPDATE`, and `DELETE FROM` statements.