From 062dc357776afdcbc545b07b490635f6f09a068c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 10 Jul 2026 14:09:57 -0400 Subject: [PATCH 01/12] Add spec (1st draft) --- SPEC.md | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 SPEC.md diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..e148410 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,85 @@ +# Zarr SQLiteStore Specification Version 1.0 + +## Context + +SQLiteStore is a single-file storage backend for [Zarr (v3)](https://zarr.dev/) +that stores array metadata and chunk data in an SQLite database. + +Key advantages of SQLiteStore over alternative single-file formats (e.g., Zip): + + - Full support for key deletion, overwriting, and partial value writes. + - Full ACID guarantees provided by SQLite. + - High availability of SQLite implementations across programming languages + and environments. + +SQLiteStore implements the [*Abstract store +interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) +as defined by the *Zarr v3 core specification*. SQLiteStore as defined by the +current document is a store implementation which is **Readable**, **Writeable** +and **Listable**. However, implementations of SQLiteStore may choose to support +only some of these capabilities (for example, an SQLiteStore implementation may +support only Readable and Listable). + +## File Format + +Zarr SQLiteStore files MUST conform to the on-disk SQLite file format used by +SQLite releases 3.0.0 and later. This file format is described by the document +[*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore +files are therefore regular SQLite files, but are structured with a specific +schema, defined by this document. + +The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. + +## Database schema + +The database contains two tables, as described below. + +The schema may be created by the following SQL statements: + +```sql +CREATE TABLE IF NOT EXISTS sqlitestore_metadata(k TEXT PRIMARY KEY, v TEXT); +CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); +``` + +### Table `sqlitestore_metadata` + +| Column | Type | Description | +|--------|------|-------------| +| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | +| v | `TEXT` | Binary value. The chunk or metadata content. | + +The `sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore +implementation. + +The `sqlitestore_metadata` table MUST include the following records. + +| Key | Description of value | +|-----|----------------------| +| `sqlitestore_version` | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | +| `compatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| `incompatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | + +Compatible flags are used to declare that the store uses optional features but +may still be read or written to by an implementation which does not implement +these features. + +Incompatible flags are used to declare that the store uses optional features +that must be supported by the implementation in order to read or write the +store. + +### Table `zarr` + +| Column | Type | Description | +|--------|------|-------------| +| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | +| v | `BLOB` | Binary value. The chunk or metadata content. | + +The `zarr` table acts as a key-value store for the Zarr data. + +Keys are stored exactly as provided by the Zarr client, without modification. +Per the *Abstract store interface* specification, keys are Unicode strings. Keys +may be stored in any of the Unicode encodings supported by SQLite (database +encoding). + +Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by +the Zarr client, without modification. From 3d39c73ad7c595399edbf59203114ef64e6562b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 11 Jul 2026 12:23:56 -0300 Subject: [PATCH 02/12] Spec clarifications and add created-by metadata - Clarifications - Add description of which metadata records are required - Add created-by record to metadata table (not required) - Add link to the spec in README --- README.md | 13 +++++++++++++ SPEC.md | 40 ++++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 34e2c07..c4402cb 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,13 @@ Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python. +SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore): + + - Support for key deletion, overwriting, and partial value writes. + - Full ACID guarantees provided by SQLite. + - High availability of SQLite implementations across programming languages + and environments. + Example usage: ```python @@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store: `SQLiteStore` otherwise behaves identically to other stores used with zarr, see the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html) for more information. + +## Specification + +The store format is described in the document [SPEC.md](/SPEC.md). This document +should allow the implementation of SQLiteStore for other programming languages +or Zarr libraries. \ No newline at end of file diff --git a/SPEC.md b/SPEC.md index e148410..be8863d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -7,26 +7,28 @@ that stores array metadata and chunk data in an SQLite database. Key advantages of SQLiteStore over alternative single-file formats (e.g., Zip): - - Full support for key deletion, overwriting, and partial value writes. + - Support for key deletion, overwriting, and partial value writes. - Full ACID guarantees provided by SQLite. - High availability of SQLite implementations across programming languages and environments. +## Store Capabilities + SQLiteStore implements the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) -as defined by the *Zarr v3 core specification*. SQLiteStore as defined by the -current document is a store implementation which is **Readable**, **Writeable** -and **Listable**. However, implementations of SQLiteStore may choose to support -only some of these capabilities (for example, an SQLiteStore implementation may -support only Readable and Listable). +as defined by the *Zarr v3 core specification*. SQLiteStore provides the store +capabilities **Readable**, **Writeable** and **Listable**. However, +specific implementations of SQLiteStore may choose to support only some of these +capabilities (for example, an SQLiteStore implementation may support only +Readable and Listable). ## File Format Zarr SQLiteStore files MUST conform to the on-disk SQLite file format used by SQLite releases 3.0.0 and later. This file format is described by the document [*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore -files are therefore regular SQLite files, but are structured with a specific -schema, defined by this document. +files are therefore regular SQLite files which use a specific database schema, +defined by this document. The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. @@ -43,6 +45,8 @@ CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); ### Table `sqlitestore_metadata` +Schema: + | Column | Type | Description | |--------|------|-------------| | k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | @@ -51,13 +55,14 @@ CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); The `sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore implementation. -The `sqlitestore_metadata` table MUST include the following records. +The `sqlitestore_metadata` table MUST include records which are marked REQUIRED in the table below. -| Key | Description of value | -|-----|----------------------| -| `sqlitestore_version` | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | -| `compatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | -| `incompatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| Key | Required | Description of value | +|-----|----------|----------------------| +| `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | +| `compatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| `incompatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| `created_by` | No | Arbitrary string describing the software that wrore the file. | Compatible flags are used to declare that the store uses optional features but may still be read or written to by an implementation which does not implement @@ -67,8 +72,15 @@ Incompatible flags are used to declare that the store uses optional features that must be supported by the implementation in order to read or write the store. +The record `created_by` MUST NOT be relied on by readers to determine the method +by which the file is to be interpreted. The file format MUST be fully described +by the metadata records `sqlitestore_version`, `compatible_flags` and +`incompatible_flags`. + ### Table `zarr` +Schema: + | Column | Type | Description | |--------|------|-------------| | k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | From ce6433dcaa946f05a17dedbae38c6483b202e7b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 17 Jul 2026 21:03:03 -0400 Subject: [PATCH 03/12] Various additions and modifications to the proposed spec - Replace the Context section with a Scope section with more neutral terminology. - Add a section for notes about design decisions - Add document conventions with a reference to RFC-2119 - Add a definitions section - Add application_id and require sqlite 3.7.17 (released 2013-05-20) - Specify explicitly how readers must behave in various places - Add a NOT NULL constraint to the schema - Add an optional created_time metadata record - Specify explicitly how readers should handle versions and flags - Add a recommendation on journaling mode - Add an informative section on limitations. --- SPEC.md | 186 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 147 insertions(+), 39 deletions(-) diff --git a/SPEC.md b/SPEC.md index be8863d..5983e0b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,78 +1,163 @@ -# Zarr SQLiteStore Specification Version 1.0 +# Zarr SQLiteStore specification version 1.0 -## Context +## Scope -SQLiteStore is a single-file storage backend for [Zarr (v3)](https://zarr.dev/) -that stores array metadata and chunk data in an SQLite database. +This document describes **SQLiteStore**, a [Zarr (v3)](https://zarr.dev/) store +that allows the storage of a complete Zarr hierarchy inside a single file by +using the SQLite database engine. -Key advantages of SQLiteStore over alternative single-file formats (e.g., Zip): +## Notes about the design decisions - - Support for key deletion, overwriting, and partial value writes. - - Full ACID guarantees provided by SQLite. - - High availability of SQLite implementations across programming languages - and environments. +SQLiteStore is designed primarily for sharing moderately sized datasets by storing an entire dataset in a single file. + +Although SQLite supports databases several terabytes in size, no file size limit is defined by this specification. However, SQLiteStore is intended for datasets up to approximately 10 GB. + +SQLite was chosen to accomplish this goal for the following reasons: + +- SQLite has been in widespread use for a long time and is considered to be + highly robust software. +- Unlike many archive formats, SQLite natively supports Zarr store operations such as key deletion, overwriting values, and modifying stored values. +- SQLite libraries are available for nearly all programming languages and + computers. + +SQLiteStore was not designed to compete in read or write speed with other +stores, such as *FileSystemStore*. + +## Document conventions + +The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, +“SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be +interpreted as described in [IETF RFC +2119](https://datatracker.ietf.org/doc/html/rfc2119). + +## Definitions + +The terms *store*, *hierarchy*, *chunk* and *array* are defined by the [Zarr core specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#concepts-and-terminology). + +A SQLiteStore file is a SQLite database conforming to this specification. + +An *implementation* refers to library or software that is designed to read and +write SQLiteStore files by implementing the operations defined by the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface). + +A *reader* is a implementation which is performing read operations on a SQLiteStore +file. + +A *writer* is a implementation which is creating, modifying or adding data to a +SQLiteStore file. + +A *client* refers to software which uses an *implementation*, for example +by calling the functions defined by the implementation. Examples of clients +may include [zarr-python](https://zarr.readthedocs.io/en/stable/) and [zarrs]( +https://zarrs.dev/). ## Store Capabilities SQLiteStore implements the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) -as defined by the *Zarr v3 core specification*. SQLiteStore provides the store -capabilities **Readable**, **Writeable** and **Listable**. However, -specific implementations of SQLiteStore may choose to support only some of these -capabilities (for example, an SQLiteStore implementation may support only -Readable and Listable). +as defined by the *Zarr v3 core specification*. SQLiteStore supports the +capabilities **Readable**, **Writable** and **Listable**. However, specific +implementations of SQLiteStore may choose to support only some of these +capabilities (for example, a read-only SQLiteStore implementation may support +only Readable and Listable). -## File Format +## Canonical URI -Zarr SQLiteStore files MUST conform to the on-disk SQLite file format used by +## File format + +Zarr SQLiteStore files must conform to the on-disk SQLite file format used by SQLite releases 3.0.0 and later. This file format is described by the document [*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore files are therefore regular SQLite files which use a specific database schema, defined by this document. -The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. +The file extension `.zarrdb` is recommended for Zarr SQLiteStore files. + +### Minimal SQLite version + +Files conforming to this specification require features introduced in SQLite +3.7.17. + +### Application ID + +The SQLite file format defines a 4-byte integer at offset 68 which can be used +to identify the application-specific file type. SQLiteStore files must have this +value set to the hexadecimal integer `0x10b50760`. This requirement may achieve +this by executing the following SQL pragma statement: + +```sql +PRAGMA application_id = 0x10b50760 +``` ## Database schema -The database contains two tables, as described below. +SQLiteStore files must contain two tables, as described below. The value columns +*v* must not contain NULL values and must therefore be specified with a +`NOT NULL` constraint in the table schemas. The schema may be created by the following SQL statements: ```sql -CREATE TABLE IF NOT EXISTS sqlitestore_metadata(k TEXT PRIMARY KEY, v TEXT); -CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); +CREATE TABLE IF NOT EXISTS sqlitestore_metadata( + k TEXT PRIMARY KEY, + v TEXT NOT NULL +); +CREATE TABLE IF NOT EXISTS zarr( + k TEXT PRIMARY KEY, + v BLOB NOT NULL +); ``` +Readers must ignore all other tables present within the file. + ### Table `sqlitestore_metadata` Schema: | Column | Type | Description | |--------|------|-------------| -| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | -| v | `TEXT` | Binary value. The chunk or metadata content. | +| k | `TEXT` | Metadata key | +| v | `TEXT` | Metadata value string | -The `sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore -implementation. +The `sqlitestore_metadata` table stores metadata that is useful for the +SQLiteStore implementation. -The `sqlitestore_metadata` table MUST include records which are marked REQUIRED in the table below. +The `sqlitestore_metadata` table must include records which are marked +*required* in the table below. Writers must not insert in the +`sqlitestore_metadata` table any record with a key that is not described here. +Readers must ignore any record with a key that is not described here. | Key | Required | Description of value | |-----|----------|----------------------| | `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | -| `compatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | -| `incompatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | -| `created_by` | No | Arbitrary string describing the software that wrore the file. | +| `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | +| `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | +| `created_by` | No | Arbitrary string describing the software that wrote the file. | +| `created_time` | No | Timestamp indicating when the file was most recently modified, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339). | + +Readers shall reject files whose major version is unsupported, as specified by +the `sqlitestore_version` metadata record. Readers should ignore minor versions +greater than those they implement unless incompatible_flags contains unknown +values. + +Compatible flags may be used in future versions of SQLiteStore to declare that +the store uses optional features but may still be read or written to by an +implementation which does not implement these features. + +Incompatible flags may be used in future versions of SQLiteStore to declare that +the store uses optional features that must be supported by the implementation in +order to read or write the store. -Compatible flags are used to declare that the store uses optional features but -may still be read or written to by an implementation which does not implement -these features. +Readers shall: -Incompatible flags are used to declare that the store uses optional features -that must be supported by the implementation in order to read or write the -store. +1. ignore all compatible flags it does not recognize; -The record `created_by` MUST NOT be relied on by readers to determine the method +2. reject the file if any incompatible flag is not recognized. + +In SQLiteStore version 1.0, no flags are defined. Writers must therefore set the +values of "compatible_flags" and "incompatible_flags" to the empty string +(`""`). + +The record `created_by` must not be relied upon by readers to determine the method by which the file is to be interpreted. The file format MUST be fully described by the metadata records `sqlitestore_version`, `compatible_flags` and `incompatible_flags`. @@ -83,15 +168,38 @@ Schema: | Column | Type | Description | |--------|------|-------------| -| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | +| k | `TEXT` | Primary key. The Zarr key (full path as a Unicode string). | | v | `BLOB` | Binary value. The chunk or metadata content. | The `zarr` table acts as a key-value store for the Zarr data. Keys are stored exactly as provided by the Zarr client, without modification. -Per the *Abstract store interface* specification, keys are Unicode strings. Keys -may be stored in any of the Unicode encodings supported by SQLite (database -encoding). +Per the *Abstract store interface* specification, keys are Unicode strings. +Implementations shall preserve the logical Unicode string regardless of the +underlying SQLite database encoding. No Unicode normalization shall be +performed. Keys must be compared using SQLite's native TEXT comparison, for +example when running `SELECT` queries for key lookup. Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by the Zarr client, without modification. + +## Database journaling mode + +The default journaling mode of SQLite is `DELETE`, and this mode is recommended +for SQLiteStore files. + +Implementations may use journal mode `WAL` (write-ahead log) for performance or +concurrency reasons, especially in write-heavy use cases. When using `WAL` +journal mode, implementations should attempt to restore the journal mode to +`DELETE` or perform a WAL checkpoint is executed before the databased is closed. + +The purpose of this best-effort requirement is to ensure all data is moved by +SQLite from the WAL files (files suffixed with `-wal` and `-shm` automatically +created by SQLite) to the main database file, so that the SQLiteStore database +consists of a single self-contained file that can be safely shared. + +## Limitations + +The maximum size of a value which can be stored (for example, an array chunk) is +limited to 2147483645 bytes (3 bytes less than 2 GiB) by current SQLite +implementations. From c089d3d13455a6c8ffae96a974e42be72146ebb8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 18 Jul 2026 21:22:16 -0400 Subject: [PATCH 04/12] Edits for clarity --- SPEC.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/SPEC.md b/SPEC.md index 5983e0b..069e65b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -39,13 +39,13 @@ A SQLiteStore file is a SQLite database conforming to this specification. An *implementation* refers to library or software that is designed to read and write SQLiteStore files by implementing the operations defined by the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface). -A *reader* is a implementation which is performing read operations on a SQLiteStore +A *reader* is an implementation that is reading a SQLiteStore file. -A *writer* is a implementation which is creating, modifying or adding data to a +A *writer* is an implementation that is creating or modifying a SQLiteStore file. -A *client* refers to software which uses an *implementation*, for example +A *client* refers to software that uses an *implementation*, for example by calling the functions defined by the implementation. Examples of clients may include [zarr-python](https://zarr.readthedocs.io/en/stable/) and [zarrs]( https://zarrs.dev/). @@ -81,8 +81,8 @@ Files conforming to this specification require features introduced in SQLite The SQLite file format defines a 4-byte integer at offset 68 which can be used to identify the application-specific file type. SQLiteStore files must have this -value set to the hexadecimal integer `0x10b50760`. This requirement may achieve -this by executing the following SQL pragma statement: +value set to the hexadecimal integer `0x10b50760`. Implementations may satisfy +this requirement by executing the following SQL pragma statement: ```sql PRAGMA application_id = 0x10b50760 @@ -94,7 +94,8 @@ SQLiteStore files must contain two tables, as described below. The value columns *v* must not contain NULL values and must therefore be specified with a `NOT NULL` constraint in the table schemas. -The schema may be created by the following SQL statements: +Implementations may satisfy this requirement by executing the following SQL +statements: ```sql CREATE TABLE IF NOT EXISTS sqlitestore_metadata( @@ -128,39 +129,39 @@ Readers must ignore any record with a key that is not described here. | Key | Required | Description of value | |-----|----------|----------------------| -| `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | +| `sqlitestore_version` | Yes | SQLiteStore file format version, formatted as `MAJOR.MINOR`. For SQLiteStore version 1.0, this value must be the string "1.0". | | `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `created_by` | No | Arbitrary string describing the software that wrote the file. | | `created_time` | No | Timestamp indicating when the file was most recently modified, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339). | -Readers shall reject files whose major version is unsupported, as specified by +Readers must reject files whose major version is unsupported, as specified by the `sqlitestore_version` metadata record. Readers should ignore minor versions greater than those they implement unless incompatible_flags contains unknown values. Compatible flags may be used in future versions of SQLiteStore to declare that the store uses optional features but may still be read or written to by an -implementation which does not implement these features. +implementation that does not implement these features. Incompatible flags may be used in future versions of SQLiteStore to declare that the store uses optional features that must be supported by the implementation in order to read or write the store. -Readers shall: +Readers must: -1. ignore all compatible flags it does not recognize; +1. Ignore all compatible flags it does not recognize; -2. reject the file if any incompatible flag is not recognized. +2. Reject the file if any incompatible flag is not recognized. In SQLiteStore version 1.0, no flags are defined. Writers must therefore set the values of "compatible_flags" and "incompatible_flags" to the empty string (`""`). -The record `created_by` must not be relied upon by readers to determine the method -by which the file is to be interpreted. The file format MUST be fully described -by the metadata records `sqlitestore_version`, `compatible_flags` and -`incompatible_flags`. +The `created_by` record is informational. Readers must not use it to determine +how the file is to be interpreted. The file format must be fully determined by +the metadata records sqlitestore_version, compatible_flags, and +incompatible_flags. ### Table `zarr` @@ -191,15 +192,15 @@ for SQLiteStore files. Implementations may use journal mode `WAL` (write-ahead log) for performance or concurrency reasons, especially in write-heavy use cases. When using `WAL` journal mode, implementations should attempt to restore the journal mode to -`DELETE` or perform a WAL checkpoint is executed before the databased is closed. +`DELETE` or perform a WAL checkpoint before the database is closed. The purpose of this best-effort requirement is to ensure all data is moved by SQLite from the WAL files (files suffixed with `-wal` and `-shm` automatically -created by SQLite) to the main database file, so that the SQLiteStore database +created by SQLite) to the main database file, so that the SQLiteStore file consists of a single self-contained file that can be safely shared. ## Limitations -The maximum size of a value which can be stored (for example, an array chunk) is +The maximum size of a value that can be stored (for example, an array chunk) is limited to 2147483645 bytes (3 bytes less than 2 GiB) by current SQLite implementations. From 93885d6053d8a457f845b1cba7e13baf7b0d04f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 18 Jul 2026 21:23:52 -0400 Subject: [PATCH 05/12] Add requirement of valid keys --- SPEC.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/SPEC.md b/SPEC.md index 069e65b..72634e1 100644 --- a/SPEC.md +++ b/SPEC.md @@ -181,6 +181,15 @@ underlying SQLite database encoding. No Unicode normalization shall be performed. Keys must be compared using SQLite's native TEXT comparison, for example when running `SELECT` queries for key lookup. +Writers must insert only valid keys in the zarr table. Valid keys are +those which respect the following rules: + +1. The final character of the key must not be a `/` character. +2. The first character of the key must not be a `/` character. +3. A key must not contain the substring `//`. + +The empty string is a valid key. + Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by the Zarr client, without modification. From 704b60050bde8f83f669632350566af9fa9df2d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Mon, 20 Jul 2026 10:28:27 -0400 Subject: [PATCH 06/12] Edit key comparison requirement --- SPEC.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 72634e1..ef5e686 100644 --- a/SPEC.md +++ b/SPEC.md @@ -176,10 +176,10 @@ The `zarr` table acts as a key-value store for the Zarr data. Keys are stored exactly as provided by the Zarr client, without modification. Per the *Abstract store interface* specification, keys are Unicode strings. -Implementations shall preserve the logical Unicode string regardless of the +Implementations must preserve the logical Unicode string regardless of the underlying SQLite database encoding. No Unicode normalization shall be -performed. Keys must be compared using SQLite's native TEXT comparison, for -example when running `SELECT` queries for key lookup. +performed. Implementations must preserve the key comparison semantics defined by +the Zarr core specification. Writers must insert only valid keys in the zarr table. Valid keys are those which respect the following rules: From dcbaf90a924ac0c7508b3125cde4937fa926af80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 25 Jul 2026 09:16:29 -0400 Subject: [PATCH 07/12] Remove Canonical URI section (empty) Zarr core's spec is ambiguous on URI and changes are being considered. Leave URI unspecified for now. --- SPEC.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index ef5e686..baf485b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -60,8 +60,6 @@ implementations of SQLiteStore may choose to support only some of these capabilities (for example, a read-only SQLiteStore implementation may support only Readable and Listable). -## Canonical URI - ## File format Zarr SQLiteStore files must conform to the on-disk SQLite file format used by From 15fcc6b895d90f06655313b08ab7e93d100e5c81 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 25 Jul 2026 09:19:06 -0400 Subject: [PATCH 08/12] Add NOT NULL constraint on k PRIMARY KEY columns --- SPEC.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/SPEC.md b/SPEC.md index baf485b..0753268 100644 --- a/SPEC.md +++ b/SPEC.md @@ -88,20 +88,20 @@ PRAGMA application_id = 0x10b50760 ## Database schema -SQLiteStore files must contain two tables, as described below. The value columns -*v* must not contain NULL values and must therefore be specified with a -`NOT NULL` constraint in the table schemas. +SQLiteStore files must contain two tables, as described below. Records must not +contain NULL values and the table schemas should therefore be specified with a +`NOT NULL` constraint on each column. Implementations may satisfy this requirement by executing the following SQL statements: ```sql CREATE TABLE IF NOT EXISTS sqlitestore_metadata( - k TEXT PRIMARY KEY, + k TEXT PRIMARY KEY NOT NULL, v TEXT NOT NULL ); CREATE TABLE IF NOT EXISTS zarr( - k TEXT PRIMARY KEY, + k TEXT PRIMARY KEY NOT NULL, v BLOB NOT NULL ); ``` From a2cbcf380b3181e22e387c2cf617dae10af4eda0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 31 Jul 2026 10:07:33 -0400 Subject: [PATCH 09/12] Add constraint on timestamp format Co-authored-by: Chris Barnes --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 0753268..eb59dcd 100644 --- a/SPEC.md +++ b/SPEC.md @@ -131,7 +131,7 @@ Readers must ignore any record with a key that is not described here. | `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `created_by` | No | Arbitrary string describing the software that wrote the file. | -| `created_time` | No | Timestamp indicating when the file was most recently modified, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339). | +| `created_time` | No | Timestamp indicating when the file was most recently modified in UTC, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339), using an upper-case T to separate the date and time and an upper-case Z to represent the timezone e.g. `1999-12-31T23:59:59.999Z`. | Readers must reject files whose major version is unsupported, as specified by the `sqlitestore_version` metadata record. Readers should ignore minor versions From 52b7ba0fb205fbeffbd6d80351f5ae22dd14d153 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 31 Jul 2026 16:23:20 -0400 Subject: [PATCH 10/12] Edits reflecting recent comment - Rename `sqlitestore_metadata` table to `zarr_sqlitestore_metadata` - rename `create_time` record to `modified_at` - Specify that readers should not rely on the accuracy of the `modified_at` timestamp - Suggest use of COLLATE BINARY to ensure correct (case sensitive) key comparison (this is the default) --- SPEC.md | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/SPEC.md b/SPEC.md index eb59dcd..c97bb87 100644 --- a/SPEC.md +++ b/SPEC.md @@ -8,17 +8,22 @@ using the SQLite database engine. ## Notes about the design decisions -SQLiteStore is designed primarily for sharing moderately sized datasets by storing an entire dataset in a single file. - -Although SQLite supports databases several terabytes in size, no file size limit is defined by this specification. However, SQLiteStore is intended for datasets up to approximately 10 GB. +SQLiteStore is designed primarily for sharing moderately sized datasets by +storing an entire dataset in a single file. SQLite was chosen to accomplish this goal for the following reasons: - SQLite has been in widespread use for a long time and is considered to be highly robust software. -- Unlike many archive formats, SQLite natively supports Zarr store operations such as key deletion, overwriting values, and modifying stored values. +- Unlike many archive formats, SQLite natively supports Zarr store operations + such as key deletion, overwriting values, and modifying stored values. - SQLite libraries are available for nearly all programming languages and computers. + + +SQLite supports databases several terabytes in size and this specification +does not define any file size limit. However, the design of SQLiteStore is +intended for datasets up to approximately 10 GB. SQLiteStore was not designed to compete in read or write speed with other stores, such as *FileSystemStore*. @@ -96,7 +101,7 @@ Implementations may satisfy this requirement by executing the following SQL statements: ```sql -CREATE TABLE IF NOT EXISTS sqlitestore_metadata( +CREATE TABLE IF NOT EXISTS zarr_sqlitestore_metadata( k TEXT PRIMARY KEY NOT NULL, v TEXT NOT NULL ); @@ -108,7 +113,7 @@ CREATE TABLE IF NOT EXISTS zarr( Readers must ignore all other tables present within the file. -### Table `sqlitestore_metadata` +### Table `zarr_sqlitestore_metadata` Schema: @@ -117,12 +122,12 @@ Schema: | k | `TEXT` | Metadata key | | v | `TEXT` | Metadata value string | -The `sqlitestore_metadata` table stores metadata that is useful for the +The `zarr_sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore implementation. -The `sqlitestore_metadata` table must include records which are marked +The `zarr_sqlitestore_metadata` table must include records which are marked *required* in the table below. Writers must not insert in the -`sqlitestore_metadata` table any record with a key that is not described here. +`zarr_sqlitestore_metadata` table any record with a key that is not described here. Readers must ignore any record with a key that is not described here. | Key | Required | Description of value | @@ -131,7 +136,7 @@ Readers must ignore any record with a key that is not described here. | `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `created_by` | No | Arbitrary string describing the software that wrote the file. | -| `created_time` | No | Timestamp indicating when the file was most recently modified in UTC, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339), using an upper-case T to separate the date and time and an upper-case Z to represent the timezone e.g. `1999-12-31T23:59:59.999Z`. | +| `modified_at` | No | Timestamp indicating when the file was most recently modified in UTC, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339), using an upper-case T to separate the date and time and an upper-case Z to represent the timezone e.g. `1999-12-31T23:59:59.999Z`. | Readers must reject files whose major version is unsupported, as specified by the `sqlitestore_version` metadata record. Readers should ignore minor versions @@ -161,6 +166,9 @@ how the file is to be interpreted. The file format must be fully determined by the metadata records sqlitestore_version, compatible_flags, and incompatible_flags. +Writers are not required to create or update the created_at field. Consequently, +readers should not rely on its presence or accuracy. + ### Table `zarr` Schema: @@ -176,8 +184,7 @@ Keys are stored exactly as provided by the Zarr client, without modification. Per the *Abstract store interface* specification, keys are Unicode strings. Implementations must preserve the logical Unicode string regardless of the underlying SQLite database encoding. No Unicode normalization shall be -performed. Implementations must preserve the key comparison semantics defined by -the Zarr core specification. +performed. Writers must insert only valid keys in the zarr table. Valid keys are those which respect the following rules: @@ -188,6 +195,10 @@ those which respect the following rules: The empty string is a valid key. +Implementations must preserve the key comparison semantics defined by the Zarr +core specification. In SQLite, this can be achieved by using the "BINARY" +collating function for all key comparisons. + Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by the Zarr client, without modification. From 4558814a9e4e942b0254f4d7ba7d40f948eaf207 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 31 Jul 2026 16:24:37 -0400 Subject: [PATCH 11/12] Typo created_at -> modified_at Clarified that writers are not required to update the modified_at field. --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index c97bb87..2d82e45 100644 --- a/SPEC.md +++ b/SPEC.md @@ -166,7 +166,7 @@ how the file is to be interpreted. The file format must be fully determined by the metadata records sqlitestore_version, compatible_flags, and incompatible_flags. -Writers are not required to create or update the created_at field. Consequently, +Writers are not required to create or update the modified_at field. Consequently, readers should not rely on its presence or accuracy. ### Table `zarr` From 4d8f9cb59d60116dc775d7f30b2162b755e669bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Wed, 5 Aug 2026 10:39:25 -0400 Subject: [PATCH 12/12] Add appendix to spec with example SQL statements --- SPEC.md | 329 ++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 309 insertions(+), 20 deletions(-) diff --git a/SPEC.md b/SPEC.md index 2d82e45..f066fb6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -33,7 +33,12 @@ stores, such as *FileSystemStore*. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [IETF RFC -2119](https://datatracker.ietf.org/doc/html/rfc2119). +2119](https://datatracker.ietf.org/doc/html/rfc2119). However, for readability, +these words do not appear in all uppercase letters in this specification. + +All of the text of this specification is normative except sections explicitly +marked as non-normative, examples, and notes. Examples in this specification are +introduced with the words “for example”. ## Definitions @@ -55,7 +60,7 @@ by calling the functions defined by the implementation. Examples of clients may include [zarr-python](https://zarr.readthedocs.io/en/stable/) and [zarrs]( https://zarrs.dev/). -## Store Capabilities +## Store capabilities SQLiteStore implements the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) @@ -93,23 +98,10 @@ PRAGMA application_id = 0x10b50760 ## Database schema -SQLiteStore files must contain two tables, as described below. Records must not -contain NULL values and the table schemas should therefore be specified with a -`NOT NULL` constraint on each column. - -Implementations may satisfy this requirement by executing the following SQL -statements: - -```sql -CREATE TABLE IF NOT EXISTS zarr_sqlitestore_metadata( - k TEXT PRIMARY KEY NOT NULL, - v TEXT NOT NULL -); -CREATE TABLE IF NOT EXISTS zarr( - k TEXT PRIMARY KEY NOT NULL, - v BLOB NOT NULL -); -``` +SQLiteStore files must contain two exactly tables, `zarr_sqlitestore_metadata` and +`zarr`. The schema for each table are described in the +subsections below. Records must not contain NULL values and the table schemas +should therefore be specified with a `NOT NULL` constraint on each column. Readers must ignore all other tables present within the file. @@ -136,7 +128,7 @@ Readers must ignore any record with a key that is not described here. | `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `created_by` | No | Arbitrary string describing the software that wrote the file. | -| `modified_at` | No | Timestamp indicating when the file was most recently modified in UTC, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339), using an upper-case T to separate the date and time and an upper-case Z to represent the timezone e.g. `1999-12-31T23:59:59.999Z`. | +| `modified_at` | No | Timestamp indicating when the file was most recently modified in UTC, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339), using an upper-case T to separate the date and time and an upper-case Z to represent the timezone e.g. `1999-12-31T23:59:59.999Z`. Note that this format is natively understood as a "time-value" by SQLite. | Readers must reject files whose major version is unsupported, as specified by the `sqlitestore_version` metadata record. Readers should ignore minor versions @@ -222,3 +214,300 @@ consists of a single self-contained file that can be safely shared. The maximum size of a value that can be stored (for example, an array chunk) is limited to 2147483645 bytes (3 bytes less than 2 GiB) by current SQLite implementations. + +## Appendix: Example SQL statements (non-normative) + +This appendix provides example SQL statements that implement the operations +defined by the *Zarr core specification abstract store interface*, as well +as other operations which are specific to the metadata elements described in the +current document. These examples are informative only and are intended to +illustrate one possible implementation. + +Implementations may use different SQL statements or SQLite APIs provided they +exhibit equivalent externally observable behaviour. + +In all examples below: + +- `:key`, `:value`, `:prefix`, and similar identifiers denote bound SQL + parameters. +- The `zarr` table is assumed to have the schema defined in this specification. +- Implementations should use parameter binding rather than constructing SQL + statements by interpolating values into SQL strings. + +Several operations in this appendix require selecting all keys that begin with a +given prefix. Throughout this appendix, such operations assume the following +parameter definitions: + +- `:prefix` is the prefix to search for. `:prefix` must respect the following rules: + * `:prefix` must not be the empty string (`''`) + * `:prefix` must end with the slash character (`/`) + * `:prefix` must not start with a slash character (`/`) + * `:prefix` must not contain the substring (`//`) +- `:upper` is derived from `:prefix` by replacing the trailing slash character + `/` with the character zero (`0`, U+0030). If `:prefix` is the empty string, + Then `:upper` is omitted and no upper bound is required. + +Because Zarr keys use `/` exclusively as a path separator, no valid key is equal +to a non-empty `:prefix`. Furthermore, under the required `BINARY` collating +function, every key beginning with `:prefix` compares lexicographically greater +than `:prefix` and less than `:upper`. + +Thus, for non-empty prefixes, a prefix search may be implemented as: + +```sql +WHERE k > :prefix + AND k < :upper; +``` + +For an empty prefix, every key matches and the `WHERE` clause should therefore +be omitted. + +### Read operations + +#### `get` + +Retrieve the value associated with a key. + +```sql +SELECT v +FROM zarr +WHERE k = :key; +``` + +If no row is returned, the key does not exist. + +--- + +#### `get_partial_values` + +Retrieve one or more byte ranges from stored values. + +Implementations may use the SQLite incremental BLOB API (`sqlite3_blob_open()` +and related functions) for efficient partial reads of large values. +Alternatively, implementations may use SQL functions such as `substr()`. + +For example, a partial read using SQL may be performed as: + +```sql +SELECT substr(v, :offset + 1, :length) +FROM zarr +WHERE k = :key; +``` + +where `offset` is zero-based and `length` is the number of bytes to retrieve. + +The Zarr API permits multiple key/range pairs to be requested simultaneously. +Implementations may satisfy these requests individually. + +### Write operations + +#### `set` + +Insert or replace a key/value pair. + +```sql +INSERT INTO zarr(k, v) +VALUES (:key, :value) +ON CONFLICT(k) +DO UPDATE SET v = excluded.v; +``` + +--- + +#### `set_partial_values` + +Modify one or more byte ranges within an existing value. + +Implementations may use the SQLite incremental BLOB API to update only the +requested byte ranges without replacing the entire value. + +If incremental BLOB access is not used, an implementation may instead read the +existing value, modify the requested byte ranges in memory, and write the +updated value using the `set` operation. + +### Delete operations + +#### `erase` + +Delete a single key. + +```sql +DELETE FROM zarr +WHERE k = :key; +``` + +--- + +#### `erase_values` + +Delete multiple keys. + +```sql +DELETE FROM zarr +WHERE k IN (...); +``` + +Implementations should use a parameterized statement with one bound parameter +for each key. + +--- + +#### `erase_prefix` + +Delete every key having the specified prefix. + +```sql +DELETE FROM zarr +WHERE k > :prefix + AND k < :upper; +``` + +### Listing operations + +#### `list` + +Return every key in the store. + +```sql +SELECT k +FROM zarr; +``` + +--- + +#### `list_prefix` + +Return every key having a given prefix. + +```sql +SELECT k +FROM zarr +WHERE k > :prefix + AND k < :upper; +``` + +--- + +#### `list_dir` + +The `list_dir` operation returns: + +- keys immediately beneath the supplied prefix; and +- immediate child prefixes. + +```sql +WITH matches AS ( + SELECT + k, + substr(k, length(:prefix) + 1) AS rest + FROM zarr + WHERE k > :prefix + AND k < :upper +) +SELECT + 'key' AS type, + k AS path +FROM matches +WHERE instr(rest, '/') = 0 + +UNION + +SELECT DISTINCT + 'prefix' AS type, + :prefix || substr(rest, 1, instr(rest, '/')) AS path +FROM matches +WHERE instr(rest, '/') > 0 + +ORDER BY path; +``` + +This query returns rows containing two TEXT columns: `type` and `path`. The +`type` column indicates whether the value in `path` represents a stored key +(`'key'`) or a child prefix (`'prefix'`). Additionally, all returned prefixes +end in a trailing slash (`/`) character. + +Alternatively, implementations may choose to implement `list_dir` by first +obtaining all keys returned by `list_prefix`. The returned keys can then be +processed by the implementation to separate immediate child keys from immediate +child prefixes according to the Zarr storage API semantics. + +### SQLiteStore specific operations + +#### Create database schema + +The required database schema may be created using the following SQL statements: + +```sql +CREATE TABLE IF NOT EXISTS zarr_sqlitestore_metadata ( + k TEXT PRIMARY KEY NOT NULL, + v TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS zarr ( + k TEXT PRIMARY KEY NOT NULL, + v BLOB NOT NULL +); +``` + +--- + +#### Metadata + +The metadata records defined by this specification may be inserted using the +following SQL statements. + +```sql +INSERT INTO zarr_sqlitestore_metadata(k, v) +VALUES + ('sqlitestore_version', '1.0'), + ('created_by', :created_by), + ('compatible_flags', ''), + ('incompatible_flags', ''); +``` + +Where `:created_by` may be an arbitrary string that identifies the writer (not +required). + +The `modified_at` timestamp record may be created or updated with the current +UTC time using: + +```sql +INSERT INTO zarr_sqlitestore_metadata(k, v) +VALUES ('modified_at', strftime('%Y-%m-%dT%H:%M:%fZ', 'now', 'utc')) +ON CONFLICT(k) +DO UPDATE SET v = excluded.v; +``` + +--- + +#### Other timestamp operations + +Because `modified_at` is stored in a format that is natively understood by +SQLite as a "time-value", SQLite's date and time functions can be used to +convert it into other representations. For example: + +```sql +SELECT strftime('%m/%d/%Y %H:%M', v) +FROM zarr_sqlitestore_metadata +WHERE k = 'modified_at'; +``` + +This query returns the timestamp formatted as `MM/DD/YYYY HH:MM`. + +SQLite's `unixepoch()` function may be used to compute the elapsed time, in +seconds, since the `modified_at` timestamp: + +```sql +SELECT unixepoch('now') - unixepoch(v) +FROM zarr_sqlitestore_metadata +WHERE k = 'modified_at'; +``` + +Or the `julianday()` function to obtain the elapsed time as fractional number of +days: + +```sql +SELECT julianday('now') - julianday(v) +FROM zarr_sqlitestore_metadata +WHERE k = 'modified_at'; +```