You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs(partitioning): Cayenne refuses stored data written under a different partition_by - #2349
spiceai/spiceai#15019 makes a file-mode Cayenne dataset fail to load when its stored acceleration data was written with a different partitioning than its current partition_by: data written without partition_by when the dataset now sets one, data written with partition_by when the dataset no longer sets one, or data written with different partition expressions. Before, adding or removing partition_by opened an empty table that reported Ready. The partitioning page said only that changing existing partition expressions is rejected, quoted an older generic error, and told readers to delete "the partitioned acceleration directory" or set accelerator_dir, a key that does not exist on trunk. For Cayenne, deleting only the data directory is not enough, because the metastore keeps the dataset's file list and refresh checkpoint.
Pages changed
website/docs/features/data-acceleration/partitioning.md: rewrites "Changing partition_by after refresh" (the page the new error links to). It covers the three refused cases, gives the error's form, and lists the two fixes: restore or remove partition_by, or stop Spice, delete both the data directory and the Cayenne metastore directory, and restart. It also notes that other datasets sharing the metastore reload from their sources.
website/docs/components/data-accelerators/cayenne/index.md: adds a "No Re-partitioning" entry under Limitations that links to that section.
Versioned docs: no changes. The merge commit f23a6aa6 is in no release tag (git tag --contains returns nothing), so this is vNext only.
Verification
Unverified, code inspection only: the stored-partitioning check compares partition column names, not expressions — .find(|partition| partition.partition_columns != configured) at crates/accelerators/accelerator-cayenne/src/lib.rs:4541, with names from Expr::Column(col) => col.name.clone(), _ => p.name.clone(), at crates/cayenne/src/partition_creator.rs:53-54. So an expression change that keeps its name (for example expr0) is not detected. Not run.
Review round 1: added the backup requirement before deleting a shared metastore, because accelerator-only (write_mode: acceleration, sink) and pending write-back rows exist only in Cayenne.
Reproduced: the error says Cayenne does not re-partition: Cayenne does not re-partition existing acceleration data. \ (crates/accelerators/accelerator-cayenne/src/lib.rs:120)
Reproduced: the error tells the user to delete both directories: To apply the new 'partition_by' instead, stop Spice, delete the acceleration data directory '{data_dir}' and the Cayenne metastore directory '{metadata_dir}', then start Spice to reload the dataset from the source. \ (crates/accelerators/accelerator-cayenne/src/lib.rs:122)
Reproduced: the stored-partitioning wording: "without 'partition_by'".to_string() (crates/accelerators/accelerator-cayenne/src/lib.rs:4488) and format!("partitioned by [{}]", stored.join(", ")) (crates/accelerators/accelerator-cayenne/src/lib.rs:4490)
Reproduced: the configured wording: "no 'partition_by'".to_string() (crates/accelerators/accelerator-cayenne/src/lib.rs:4496) and format!("'partition_by: [{}]'", configured.join(", ")) (crates/accelerators/accelerator-cayenne/src/lib.rs:4498)
Reproduced: the keep-the-data fixes: "remove 'partition_by'" (crates/accelerators/accelerator-cayenne/src/lib.rs:4504) and "restore the 'partition_by' it was written with" (crates/accelerators/accelerator-cayenne/src/lib.rs:4506)
Reproduced: accelerator_dir does not exist on trunk (git grep -n accelerator_dir origin/trunk returns nothing).
Observed, not isolated: the add, remove, and change cases are each refused, and data reopens under its own partitioning, per the source PR's tests in crates/accelerators/accelerator-cayenne/tests/stored_partitioning.rs: https://github.com/spiceai/spiceai/pull/15019/files
Unverified, code inspection only: the runtime logs one error and does not retry, because PartitionByChanged is listed in rejects_configuration (crates/accelerators/accelerator-cayenne/src/lib.rs:4203).
Unverified, code inspection only: deleting only the data directory is not enough, because the metastore keeps the file list and refresh checkpoint (taken from the source PR description, not run).
cd website && npm run build at 9e69702: [SUCCESS] Generated static files in "build". exit=0 (no broken links or anchors)
The backup must be taken after Spice stops. Copying the live SQLite metastore and data directories while catalog commits or compactions are in progress may produce an inconsistent recovery point. State the ordering explicitly, and include every data directory cataloged by the shared metastore.
Report or identify the active Cayenne metastore path
The error path is not always the active metastore path. PartitionByChanged derives <metadata_dir> from the failing dataset's configuration, but Cayenne uses the process-wide catalog opened by the first dataset; this page documents at components/data-accelerators/cayenne/index.md:942 that later datasets can resolve another directory. In that case, deleting the named directory leaves the real catalog intact and the load fails again. Either make the runtime report the active catalog path or document how to identify and delete that path here.
Addressed both previously-missed items in bd6cfd4: the steps now stop Spice with SIGTERM first, then back up cayenne.db with its -wal/-shm files and every data directory the metastore catalogs, and only then delete. Step 3 says the error names the metastore resolved for the failing dataset, explains that the process uses the catalog of whichever Cayenne dataset initializes first (linking Metastore location), and recommends one explicit cayenne_metadata_dir for all Cayenne datasets so the named directory is the active one. Having the runtime report the active catalog path is a runtime change for spiceai/spiceai.
The deletion target is still ambiguous when the resolved path in the error differs from the process's active catalog. The runtime constructs the error with the failing dataset's resolved metadata_dir, while the table was opened through the first dataset's shared catalog, so deleting only the named directory can leave the stale file list and checkpoint active. Require either confirming that the named path is the active catalog or normalizing every dataset to one explicit path before deletion and restart.
Addressed the previously-missed active-catalog item in c76bd5f: a new step 3, before any deletion, requires every Cayenne dataset to resolve the same metastore directory, setting one explicit cayenne_metadata_dir on all of them when the resolved paths can differ, and step 4 deletes that directory.
Deleting the shared catalog invalidates every data directory it references, not only the failing dataset's directory. The metastore documentation at website/docs/components/data-accelerators/cayenne/index.md:946 states that opening an empty catalog leaves the prior Vortex files on disk. With this step, other datasets reload while their old files remain orphaned, potentially consuming roughly another full copy of their storage. After the backup/reloadability check, delete every data directory cataloged by each metastore being removed, or explicitly include an orphan-file cleanup step.
Addressed the previously-missed orphaned-files item in f3075b9: step 4 now deletes every data directory the removed metastore catalogs, and says a directory left behind keeps its files on disk while the dataset reloads into new ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
spiceai/spiceai#15019 makes a file-mode Cayenne dataset fail to load when its stored acceleration data was written with a different partitioning than its current
partition_by: data written withoutpartition_bywhen the dataset now sets one, data written withpartition_bywhen the dataset no longer sets one, or data written with different partition expressions. Before, adding or removingpartition_byopened an empty table that reportedReady. The partitioning page said only that changing existing partition expressions is rejected, quoted an older generic error, and told readers to delete "the partitioned acceleration directory" or setaccelerator_dir, a key that does not exist on trunk. For Cayenne, deleting only the data directory is not enough, because the metastore keeps the dataset's file list and refresh checkpoint.Pages changed
website/docs/features/data-acceleration/partitioning.md: rewrites "Changingpartition_byafter refresh" (the page the new error links to). It covers the three refused cases, gives the error's form, and lists the two fixes: restore or removepartition_by, or stop Spice, delete both the data directory and the Cayenne metastore directory, and restart. It also notes that other datasets sharing the metastore reload from their sources.website/docs/components/data-accelerators/cayenne/index.md: adds a "No Re-partitioning" entry under Limitations that links to that section.Versioned docs: no changes. The merge commit
f23a6aa6is in no release tag (git tag --containsreturns nothing), so this is vNext only.Verification
.find(|partition| partition.partition_columns != configured)at crates/accelerators/accelerator-cayenne/src/lib.rs:4541, with names fromExpr::Column(col) => col.name.clone(), _ => p.name.clone(),at crates/cayenne/src/partition_creator.rs:53-54. So an expression change that keeps its name (for exampleexpr0) is not detected. Not run.write_mode: acceleration,sink) and pending write-back rows exist only in Cayenne.Trunk SHA:
f23a6aa686ddff43ae5a73effa36087a3b8425f9.Cayenne does not re-partition existing acceleration data. \(crates/accelerators/accelerator-cayenne/src/lib.rs:120)To apply the new 'partition_by' instead, stop Spice, delete the acceleration data directory '{data_dir}' and the Cayenne metastore directory '{metadata_dir}', then start Spice to reload the dataset from the source. \(crates/accelerators/accelerator-cayenne/src/lib.rs:122)"without 'partition_by'".to_string()(crates/accelerators/accelerator-cayenne/src/lib.rs:4488) andformat!("partitioned by [{}]", stored.join(", "))(crates/accelerators/accelerator-cayenne/src/lib.rs:4490)"no 'partition_by'".to_string()(crates/accelerators/accelerator-cayenne/src/lib.rs:4496) andformat!("'partition_by: [{}]'", configured.join(", "))(crates/accelerators/accelerator-cayenne/src/lib.rs:4498)"remove 'partition_by'"(crates/accelerators/accelerator-cayenne/src/lib.rs:4504) and"restore the 'partition_by' it was written with"(crates/accelerators/accelerator-cayenne/src/lib.rs:4506)accelerator_dirdoes not exist on trunk (git grep -n accelerator_dir origin/trunkreturns nothing).crates/accelerators/accelerator-cayenne/tests/stored_partitioning.rs: https://github.com/spiceai/spiceai/pull/15019/filesPartitionByChangedis listed inrejects_configuration(crates/accelerators/accelerator-cayenne/src/lib.rs:4203).cd website && npm run buildat 9e69702:[SUCCESS] Generated static files in "build". exit=0(no broken links or anchors)