Convert to a pnpm monorepo and add an S3 upload strategy

Restructure the repository into a pnpm workspace while keeping the core
sqlite-js-stream package at the root unchanged (same API, name, and source).
Upload strategies now live as addon packages under packages/.

- Add pnpm-workspace.yaml + pnpm-lock.yaml; drop package-lock.json.
- Core package.json gains an 'exports' map ('.' -> ./src/index.ts) so addons can
  import the ReplicaClient contract, plus workspace scripts (test:all /
  typecheck:all). Export the Logger type so addons implement ReplicaClient.
- Root tsconfig now type-checks src/, test/, and packages/.
- Add packages/backend-s3: a sqlite-js-stream-backend-s3 addon implementing an
  S3ReplicaClient (AWS SDK v3). It stores objects as
  {prefix}/ltx/{level}/{min}-{max}.ltx, preserves timestamps as ltx-timestamp
  metadata, supports range reads, batched deletes, and registers the s3:// URL
  scheme. An injectable 'client' (S3ClientLike) makes it testable without AWS;
  tests use an in-memory fake plus a full Replica.sync() -> restore() round-trip.
- CI: use pnpm (pnpm/action-setup + setup-node cache:pnpm, frozen lockfile).
- Document the workspace layout and S3 addon in README and ARCHITECTURE.

Author: mcollina

.github/workflows/ci.yml

@@ -21,17 +21,22 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v7
 
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v6
+        with:
+          version: 12
+
       - name: Setup Node
         uses: actions/setup-node@v7
         with:
           node-version: ${{ matrix.node-version }}
-          cache: npm
+          cache: pnpm
 
       - name: Install dependencies
-        run: npm ci
+        run: pnpm install --frozen-lockfile
 
       - name: Typecheck
-        run: npx tsc -p . --noEmit
+        run: pnpm typecheck:all
 
       - name: Run tests
-        run: npm test
+        run: pnpm test:all

README.md

@@ -27,11 +27,22 @@ No build step. Test files and source run directly under Node.
 
 ## Installation
 
+This repository is a **pnpm workspace**. The core library lives at the root and
+upload strategies (replica backends) live under `packages/`:
+
 ```bash
-npm install sqlite-js-stream
+pnpm install       # link the workspace
 ```
 
-*(Not yet published; install from the repository.)*
+There is no build step: we run TypeScript directly via Node's native type
+stripping.
+
+The core package is published as `sqlite-js-stream` *(not yet published; install
+from the repository)*, and each addon is a separate package:
+
+```bash
+pnpm add sqlite-js-stream sqlite-js-stream-backend-s3
+```
 
 ---
 
@@ -117,6 +128,43 @@ const client = newReplicaClientFromURL('file:///data/replica');
 const replica = new Replica('/data/app.db', client);
 ```
 
+The `file` scheme is built into the core. Additional schemes (S3, etc.) are
+registered by importing the matching addon (see below).
+
+### S3 upload strategy
+
+The `sqlite-js-stream-backend-s3` addon provides an S3 replica client. Import it
+once to register the `s3://` scheme, then create clients from URLs or directly:
+
+```ts
+import { Replica, newReplicaClientFromURL } from 'sqlite-js-stream';
+import { S3ReplicaClient } from 'sqlite-js-stream-backend-s3';
+
+// Via a URL (bucket from the host, prefix from the path).
+const client = newReplicaClientFromURL('s3://my-bucket/backup?region=us-east-1');
+
+// Or directly, with explicit AWS config.
+const s3 = new S3ReplicaClient({
+  bucket: 'my-bucket',
+  prefix: 'backup',
+  region: 'us-east-1',
+  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
+});
+
+const replica = new Replica('/data/app.db', client);
+await replica.run({ intervalMs: 1000 });
+```
+
+Objects are stored as `{prefix}/ltx/{level}/{minTXID}-{maxTXID}.ltx` — the same
+layout as the file backend, so a replica can move between strategies. Creation
+timestamps are preserved as the `ltx-timestamp` object metadata.
+
+> **Testing without AWS:** `S3ReplicaClient` accepts an injected `client`
+> implementing `send(command)`. Pass an in-memory fake (see
+> `packages/backend-s3/test/helpers/memory-s3.ts`) to exercise the addon without
+> network access.
+
 ### Continuous sync
 
 For ongoing replication, `replica.run()` syncs immediately and then repeats on an

docs/ARCHITECTURE.md

@@ -36,9 +36,14 @@ prevents the WAL from being checkpointed past our position.
 | **WAL module** (`src/wal/`) | Parse SQLite WAL header/frames; verify salt + checksum; produce `pageMap()` (latest committed page offsets). |
 | **LocalDatabase** (`src/db.ts`, internal) | Reads a database + `-wal` directly from disk, builds LTX files (base snapshot, incremental L0, level-9 snapshot), and tracks the local position. It holds a short-lived **read-only** `node:sqlite` read transaction to **pin the WAL snapshot** (SQLite won't checkpoint/truncate past an active reader), then reads the files via `fs`. It is an internal implementation detail, not part of the public API. |
 | **Replica** (`src/replica.ts`) | Upload level-0/level-9 LTX files to a client; track the replicated position; `restore()`, `ensureExists()`, `syncStatus()`. |
-| **ReplicaClient** (`src/replica-client.ts`) | An interface to a storage backend (`LTXFiles`, `OpenLTXFile`, `WriteLTXFile`, `DeleteLTXFiles`, `DeleteAll`). |
+| **ReplicaClient** (`src/replica-client.ts`) | An interface to a storage backend (`LTXFiles`, `OpenLTXFile`, `WriteLTXFile`, `DeleteLTXFiles`, `DeleteAll`). Addons implement this to add upload strategies. |
 | **File backend** (`src/backends/file/`) | The simplest backend: store LTX files in `<root>/ltx/<level>/<min>-<max>.ltx`, atomic writes. |
 | **Store** (`src/store.ts`) | Coordinate multiple `Replica`s. |
+| **Addons** (`packages/`) | Upload strategies live as separate workspace packages. Each addon depends on the core `sqlite-js-stream` and registers a URL scheme (e.g. `packages/backend-s3` registers `s3://`). |
+
+> The repository is a **pnpm workspace**: the core package is at the root and
+> upload-strategy addons live under `packages/*`. Addons import the `ReplicaClient`
+> contract (and `registerReplicaClientFactory`) from the core.
 
 ---
 

package-lock.json

@@ -3,6 +3,7 @@
   "version": "0.1.0",
   "description": "Node.js port of Litestream: SQLite streaming replication & disaster recovery via LTX files",
   "type": "module",
+  "packageManager": "pnpm@12.4.2",
   "engines": {
     "node": ">=24"
   },
@@ -16,9 +17,14 @@
   "bugs": {
     "url": "https://github.com/platformatic/sqlite-js-stream/issues"
   },
+  "exports": {
+    ".": "./src/index.ts"
+  },
   "scripts": {
     "test": "node --test 'test/**/*.test.ts'",
-    "typecheck": "tsc -p . --noEmit"
+    "typecheck": "tsc -p . --noEmit",
+    "test:all": "pnpm test && pnpm -r test",
+    "typecheck:all": "pnpm typecheck"
   },
   "dependencies": {
     "lz4js": "^0.2.0"

package.json

@@ -0,0 +1,24 @@
+{
+  "name": "sqlite-js-stream-backend-s3",
+  "version": "0.1.0",
+  "description": "Amazon S3 upload strategy (replica client) for sqlite-js-stream",
+  "type": "module",
+  "engines": {
+    "node": ">=24"
+  },
+  "license": "Apache-2.0",
+  "exports": {
+    ".": "./src/index.ts"
+  },
+  "scripts": {
+    "test": "node --test 'test/**/*.test.ts'"
+  },
+  "dependencies": {
+    "@aws-sdk/client-s3": "^3.1135.0",
+    "sqlite-js-stream": "workspace:*"
+  },
+  "devDependencies": {
+    "@types/node": "^24.0.0",
+    "typescript": "^5.7.0"
+  }
+}

packages/backend-s3/package.json

@@ -0,0 +1,21 @@
+/**
+ * sqlite-js-stream-backend-s3 — Amazon S3 upload strategy.
+ *
+ * Importing this module registers the `s3` scheme with the core's replica URL
+ * factory, so `newReplicaClientFromURL('s3://bucket/prefix')` resolves after
+ * this addon is loaded.
+ */
+
+export {
+  S3ReplicaClient,
+  newReplicaClientFromURL,
+  ReplicaClientType,
+  type S3ClientLike,
+  type S3ReplicaClientOptions,
+} from './replica-client.ts';
+
+import { registerReplicaClientFactory } from 'sqlite-js-stream';
+import { newReplicaClientFromURL } from './replica-client.ts';
+
+// Register the 's3' URL scheme when the addon is imported.
+registerReplicaClientFactory('s3', newReplicaClientFromURL);