SQLR-43 — Go edge/IoT event collector (concurrent-writes showcase)

[joaoh82]

What

The fifth and final app under the SQLR-38 example-apps umbrella: a Go edge/IoT event collector in examples/go-collector/. It accepts telemetry over HTTP from many concurrent producers, buffers each event in a local .sqlrite file, and runs a background uploader goroutine that drains the buffer to a pluggable sink — both writing the same database concurrently via BEGIN CONCURRENT (Phase 11 MVCC, SQLR-22).

Blocker confirmed cleared before starting: SQLR-22 (concurrent writes) is done; engine at v0.10.2; Go SDK exposes siblings + ErrBusy/IsRetryable.

The honest headline (please read)

Concurrent writes here is a capability + correctness feature, not a throughput win. I measured it three ways and shipped the numbers as found:

Workload	serialized (1 mutex)	concurrent (BEGIN CONCURRENT)
Insert throughput	156/s	142/s (0.91×)
Insert p99 latency under a checkpoint writer	164 ms	333 ms (~2× worse)
Disjoint-row batched txns (MVCC best case)	2,655/s	2,018/s (0.76×)

Root causes are all documented v0 limitations: global per-DB Arc<Mutex<Database>>, per-tx double table-clone, per-commit O(N) B-tree rebuild. The genuine win — proven by the load generator — is independent in-process writers + row-level conflict detection + zero dropped events under 32 concurrent producers, which single-writer can't express. The README ships all three tables with reproduce commands and a "why it isn't faster yet" section.

Designed around verified v0 engine sharp edges

Each found by testing and documented inline + in the README: no Go-SDK param binding (inlined/escaped SQL), CREATE TABLE IF NOT EXISTS not honored (SELECT-probe for fresh-vs-reopen), CREATE INDEX rejected under MVCC (DDL before the journal_mode switch), the 4 KiB MVCC commit-batch cap (chunked checkpoint commits), AUTOINCREMENT collisions under MVCC (app-assigned ids).

Included

• cmd/collector (HTTP + uploader), cmd/loadgen (load test + 3 measurement experiments), internal/{store,uploader,server}
• Backpressure (503) + /healthz + /stats; pluggable + flaky sinks
• Dockerfile + docker-compose (with optional read-only sqlrite-mcp sidecar against a snapshot) + Makefile
• Unit/integration tests across all packages (conflict+retry, no-drops, reopen, chunked-checkpoint, backpressure)
• New go-collector CI job (Linux + macOS) against the in-repo engine
• Examples index + sqlritedb.com Examples card + docs/concurrent-writes.md cross-link

Verification

cargo build -p sqlrite-ffi, gofmt -l, go vet ./..., go test ./... all clean. E2E: 627 concurrent POSTs → 627 uploaded → backlog 0, 0 dropped, clean restart-reopen with id continuation. Not run: web /examples typecheck (no node_modules; card mirrors existing entries) and docker build (heavy; reviewed by hand).

Follow-ups (recommended)

• v0 MVCC perf gap (BEGIN CONCURRENT slower than single mutex) — quantified by this example; unblocked by the parked COW-clone + checkpoint-drain work.
• Engine ergonomics: CREATE TABLE IF NOT EXISTS not honored + sqlrite_master not SQL-queryable.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
rust-sqlite	Ready	Preview, Comment	May 31, 2026 5:51pm

[joaoh82]

Amended after self-review: the first push had a failing test (TestLargeCheckpointUnderMVCCCap) — my fixed-size checkpoint chunking wasn't safe under the 4 KiB MVCC commit-record cap, and investigating surfaced a latent hot-path bug (a large enough event payload would overflow the cap on a single-row INSERT at COMMIT).

Fixed two ways and re-verified:

• Ingest guard: event payloads are bounded at maxPayloadBytes (3 KiB compacted) → clean 400, so any one row always commits.
• Adaptive checkpoint: writeAdaptive starts at the configured chunk size and halves on a frame-cap error down to one statement per commit; the one-row floor is guaranteed to fit thanks to the ingest guard.

Added tests for both (oversized-payload rejection + a 300-row checkpoint that would otherwise blow the cap). Full suite green; e2e re-confirmed a 720-event backlog drains to 0 with no frame-cap errors.