Skip to content

RFC: Add SQLiteStore specification #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
auxym wants to merge 12 commits into
base: main
Choose a base branch
from
Open

RFC: Add SQLiteStore specification #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
062dc35
Add spec (1st draft)
auxym Jul 10, 2026
3d39c73
Spec clarifications and add created-by metadata
auxym Jul 11, 2026
ce6433d
Various additions and modifications to the proposed spec
auxym Jul 18, 2026
c089d3d
Edits for clarity
auxym Jul 19, 2026
93885d6
Add requirement of valid keys
auxym Jul 19, 2026
704b600
Edit key comparison requirement
auxym Jul 20, 2026
dcbaf90
Remove Canonical URI section (empty)
auxym Jul 25, 2026
15fcc6b
Add NOT NULL constraint on k PRIMARY KEY columns
auxym Jul 25, 2026
a2cbcf3
Add constraint on timestamp format
auxym Jul 31, 2026
52b7ba0
Edits reflecting recent comment
auxym Jul 31, 2026
4558814
Typo created_at -> modified_at
auxym Jul 31, 2026
4d8f9cb
Add appendix to spec with example SQL statements
auxym Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python.

SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore):

- Support for key deletion, overwriting, and partial value writes.
- Full ACID guarantees provided by SQLite.
- High availability of SQLite implementations across programming languages
and environments.

@jbms jbms Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.

@newville newville Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, but "a key disadvantage" compared to what?

@jbms jbms Jul 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to zip, for example.

@auxym auxym Jul 13, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.

For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).

Example usage:

```python
Expand All @@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store:
`SQLiteStore` otherwise behaves identically to other stores used with zarr, see
the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html)
for more information.

## Specification

The store format is described in the document [SPEC.md](/SPEC.md). This document
should allow the implementation of SQLiteStore for other programming languages
or Zarr libraries.
Loading