Shared per-database DuckDB engine worker (kernel worker host + both extensions)

[qsliu2017]

What

Runs DuckDB analytical work in a shared per-database background worker instead of a DuckDB instance per PostgreSQL backend, bounding memory to one engine per database under concurrency (per-backend instances otherwise OOM). A backend ships its deparsed SQL and MVCC snapshot to the worker over shared memory and streams results back; the worker serves N sessions concurrently, one thread per session on one shared DuckDB instance.

Rebased onto current main. Six commits, reviewable in order:

1. feat: shared-memory transports for the duckdb worker -- session channel (control/result rings + a demux that routes scan_id-prefixed scan replies into per-scan lanes and RPC replies into a metadata lane), session pool (fixed shmem slots, one per concurrent dispatched query; attach refcount, per-acquire generations; a backend that finds the pool full WAITS cancellably for a slot -- capacity never falls back to in-process execution, which would recreate the per-backend memory blow-up under exactly the overload the worker exists to bound), Arrow page pool + PG-datum-to-Arrow codec, scan task queue + ready rings, the RPC protocol, and RelationDesc.
2. feat: the duckdb worker base class in the kernel -- the generic pgddb::DuckdbWorker base class in libpgduckdb/worker/ (registry/spawn/respawn, generation-validated pending sessions, backend service loop, session threads, scan producers, abort/exit cleanup, cancellation propagation) plus the kernel catalog/scan/node integration. DESIGN.md / GLOSSARY.md document it.
3. feat: pg_duckdb duckdb worker -- a thin subclass (pgduckdb::DuckdbWorker): dispatch gate (read-only heap SELECTs), GUCs, entrypoints, tests.
4. feat: pg_ducklake duckdb worker -- subclass (pgducklake::DuckdbWorker) dispatching SELECTs + autocommit DML, with DuckLake metadata reads (MetaQuery) and commit writes (MetaExec) executed on the requesting backend inside its transaction (via ServeFrame), extension SQL 1.2.0 (worker diagnostics), tests.
5. test: TLA+ models of the duckdb worker concurrency protocol -- TLC-checked models of the session-slot lifecycle (including liveness: capacity waits cannot starve), the scan demux, and the scan pool, with bug-mode configs reproducing the failure classes the design prevents.
6. ci: qualify read_parquet in the ClickBench setup -- unrelated CI fix needed for green ClickBench; droppable if the standing CI-fix PR lands first.

Design

• PG-free worker execution. The worker holds no PG transaction or snapshot. Catalog binds go through a DescribeRel RPC answered by the backend; heap scans are inverted (produced on the backend, or by a parallel scan-producer pool over CTID block ranges under the shipped snapshot, into zero-copy Arrow pages); DuckLake metadata SQL is remoted to the backend. That is what makes N concurrent sessions on one PG background worker possible.
• One channel per session, demultiplexed. Scan replies carry the scan id and are routed into per-scan lanes; metadata/catalog replies into a serialized metadata lane. Concurrent scans and mid-execution metadata RPCs (e.g. DuckLake GetFilesForTable and inlined-data reads on DuckDB scheduler threads) share the channel safely -- the earlier "scans must be sequential" constraint that the TLA+ model documented is lifted, and the model now proves the routed version.
• Session resolution off the session thread. Worker-side PG-touching code resolves its session via a thread-local on the session thread and via a ClientContext-keyed registry on scheduler threads; DuckLake's per-transaction nested metadata connection is aliased into the registry for exactly the transaction's lifetime.
• Dispatched DML atomicity (pg_ducklake). An autocommit INSERT/UPDATE/DELETE executes in the worker, but its DuckLake metadata commit runs on the backend, inside the backend's transaction -- commit is atomic with the statement, and commit conflicts surface as TransactionException for DuckLake's retry. Transactional writes and DDL stay in-process by design.
• Lifecycle hardening. Backend transaction/subtransaction abort callbacks and process-exit hooks release open sessions (subtransaction-scoped, so an RPC serviced in a conflict-retried subtransaction survives); session-slot generations reject stale queued sessions; dead workers are detected by pid probe and respawned on demand; the worker drains session threads on any exit and interrupts the DuckDB context of cancelled sessions.

Correctness work

Concurrency testing (a new e2e suite driving concurrent readers/writers through one worker) and TLC model checking found and fixed five real bugs during the rework: thread-local-only session resolution crashing on scheduler threads, a nested-connection alias outliving its transaction (cross-session misrouting via reused context addresses), subtransaction aborts freeing the channel out from under the RPC being serviced, stale pending-ring entries attaching freed/re-acquired slots, and missing bytea support in the Arrow codec (DuckLake inlined tables store text as bytea). The scan transport covers int2/4/8, float4/8, bool, date, timestamp[tz], text, bytea, and decimals; anything else fails loudly naming the column.

Tests

• pg_duckdb regression 66/66 (the shared_worker test runs through the scan-producer pool in CI and covers type coverage, unsupported-type failure, error surfacing, cancellation, worker kill + respawn).
• pg_ducklake regression 52/52 (worker_ping/worker_eval/worker_dispatch/worker_hardening: GUC-off/on parity incl. hybrid + temp-table joins and dispatched DML with read-back, dispatch-gate assertions via ducklake.worker_stats(), failure paths), isolation 4/4 (new worker_concurrent_writes spec), e2e test_shared_worker.py (concurrent readers + writers).
• specs/tla: 10 TLC configs -- 5 PASS proving the invariants, 5 bug-mode configs failing exactly as documented.

🤖 Generated with Claude Code