fix(doctor): wire real memory-DB integrity checks into the default doctor run - #2748
Merged
Conversation
Fixes #2737 Bare `doctor` / `doctor --fix` ran `allChecks`, whose only memory probe was `checkMemoryDatabase` — existsSync()+statSync() only, so it PASSES for any file that exists and can be stat'd, corrupt or not. The real integrity/content/embedding checks added for #2677 (checkMemoryIntegrity/checkMemoryContent/checkMemoryEmbeddingCoverage) were only reachable via `--component memory`, never the default run — so a genuinely corrupt .swarm/memory.db could report "All checks passed! System is healthy." on a bare `doctor` call. Three-part fix: 1. Add `checkMemoryStructuralIntegrity` to the default `allChecks` run — a bounded, native (better-sqlite3, WAL-aware) `PRAGMA quick_check`, explicitly labeled "structural-only" in its name/message. Cheaper than the deep trio (skips UNIQUE/index cross-checks quick_check deliberately omits) but bounded enough to run unconditionally. `checkMemoryDatabase`'s reported row is relabeled "Memory Database Presence" to be honest about its existence-only scope; behavior is unchanged and it stays in `allChecks` too. 2. Strengthen `checkMemoryIntegrity` (deep, `--component memory` only, full #2677 trio unchanged there): prefer native better-sqlite3 `PRAGMA integrity_check` (adds index<->table cross-checking + UNIQUE verification quick_check skips, and is WAL-aware) when better-sqlite3 is installed, falling back to the existing sql.js probe — now explicitly labeled "main-image-only" (WAL-blind) — when it's not. 3. A definitively malformed, unencrypted DB now maps to `fail`, not `warn`, in both checks. Before failing, both sniff the file for the product's RFE1 encryption-at-rest magic bytes (reusing isEncryptedBlob() from encryption/vault.ts, the same detector checkEncryptionAtRest uses) so a legitimately encrypted-at-rest DB is never misclassified as corrupt. better-sqlite3 added to @claude-flow/cli's optionalDependencies (matching the existing optional-dep pattern in @claude-flow/memory and @claude-flow/hooks) with an ambient module declaration in optional-modules.d.ts; every code path that touches it degrades to the sql.js fallback (or a warn) on MODULE_NOT_FOUND. Verified: - New __tests__/doctor-2737-memory-structural.test.ts (7 cases): healthy DB passes including the new check; malformed DB fails the new check and the overall run reports failed>0/exitCode 1 (not "all healthy"); a real RFE1-encrypted blob is not misclassified as corrupt; `--component memory` still runs the full unchanged trio; both encryption-carve-out cases for checkMemoryIntegrity directly. - Full existing doctor-*.test.ts suite (19 tests total) still passes. - Manual repro: `node bin/cli.js doctor` against a scratch dir with a garbage .swarm/memory.db now prints "Memory Structural Integrity (quick_check): ... file is not a database" as a hard FAIL and exits 1 (previously: silent PASS via checkMemoryDatabase, exit 0). Re-verified with a valid DB (exit 0, quick_check: ok) and via --component memory (full trio, native integrity_check path confirmed). Co-Authored-By: RuFlo <ruv@ruv.net>
The doctor memory-integrity fix added a direct better-sqlite3 dependency to @claude-flow/cli/package.json but didn't regenerate the workspace lockfile, causing pnpm install --frozen-lockfile to fail in CI (v3-ci.yml's lockfile-drift guard and Test V3 Packages). Co-Authored-By: RuFlo <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Jul 20, 2026
Merging main (which carries #2748's direct better-sqlite3 dependency on @claude-flow/cli) forward into this branch left the recorded lockfile specifier at the pre-override "^12.9.0" while this branch's own overrides.better-sqlite3 pin ("12.9.0", exact) is now the effective spec pnpm resolves against — a one-line drift causing `pnpm install --frozen-lockfile` to fail across the whole v3-ci.yml matrix. Regenerated to match. Co-Authored-By: RuFlo <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Jul 20, 2026
…cy tree (#2736) (#2746) * fix(security): dedup better-sqlite3 to patched 12.9.0 across dependency tree Fixes #2736 (defense-in-depth mitigation). SQLite's documented WAL-Reset Bug (sqlite.org/wal.html §11) is a race between overlapping checkpoint operations and a WAL-resetting commit that can make a later checkpoint skip a committed transaction, silently losing acknowledged writes and/or corrupting index/table structure. Affects SQLite 3.7.0-3.51.2; fixed in 3.51.3, backported to 3.44.6 and 3.50.7. agentdb's own published dependency floor (`better-sqlite3@^11.8.1`, confirmed in this tree resolving to 11.10.0, which bundles vulnerable SQLite 3.49.2) created a mixed-engine tree alongside this repo's own direct dependency on better-sqlite3 12.9.0 (patched SQLite 3.53.0, via v3/@claude-flow/memory). Multiple concurrent connections (daemon, MCP servers, CLI invocations) issuing wal_checkpoint(PASSIVE) after every memory-bridge store is exactly the race precondition this bug requires. This forces a single deduplicated better-sqlite3 build tree-wide via pnpm overrides, pinned to the exact version this repo's own @claude-flow/memory package already depends on directly (12.9.0, bundling patched SQLite 3.53.0) — no new version to vet, just true deduplication of agentdb's nested resolution into the already-patched copy. Changes (following this repo's existing #2112-lesson override convention — multiple package.json files, each consumed a different way): - v3/package.json: pnpm.overrides (the v3 pnpm workspace root — this is where agentdb's better-sqlite3@11.10.0 actually lived pre-fix) - v3/@claude-flow/cli/package.json: npm-style overrides (effective when @claude-flow/cli is npm-installed standalone via npx) - package.json (root "claude-flow" umbrella): bumped the existing ">=12.8.0" override to an exact "12.9.0" pin, AND added a pnpm.overrides block — the bare "overrides" key alone is silently ignored by pnpm (only read by npm), so the pre-existing override was a no-op for local `pnpm install` at repo root despite being present in package.json for a while - ruflo/package.json: bumped ">=12.8.0" to exact "12.9.0" (npm-style, effective for `npx ruflo@latest`) - v3/pnpm-lock.yaml, pnpm-lock.yaml: regenerated via `pnpm install` Verification: - `pnpm why -r better-sqlite3` (v3/) and `pnpm why better-sqlite3` (root): exactly one resolved version, 12.9.0, everywhere - Confirmed via realpath resolution (not hoisted symlink) from each of the 4 installed agentdb versions' own on-disk directories (1.6.1, 2.0.0-alpha.3.4, 2.0.0-alpha.3.7, 3.0.0-alpha.17) that each now binds the same deduplicated better-sqlite3@12.9.0 copy - `db.prepare('select sqlite_version()').get()` from that binding returns 3.53.0 (>= 3.51.3 patched floor) - v3/@claude-flow/memory test suite: 438/442 passing; the 4 failures are pre-existing Windows path-separator and perf-threshold issues unrelated to SQLite (none of the failing test files touch better-sqlite3/sqlite at all) Not fixed here (upstream, out of reach from this repo): - agentdb's own published package.json floor (`^11.8.1`) is unchanged — this override is a same-repo mitigation, not a fix to agentdb itself. agentdb needs the same bump upstream. Not touched here (explicitly out of scope, separate/riskier change): - The wal_checkpoint(PASSIVE)-after-every-store pattern in memory-bridge.ts (#2558) is left as-is; the issue calls that "defense-in-depth" / optional, and dedup alone removes the mixed-engine race precondition regardless. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(deps): regenerate package-lock.json for better-sqlite3 dedup override package.json pins the better-sqlite3 override to 12.9.0 (patched, fixes the WAL-reset bug) but package-lock.json still resolved a stray top-level better-sqlite3@12.10.0 and was missing the nested better-sqlite3@12.9.0 entries under @claude-flow/memory, agentdb, and agentic-flow — causing npm ci to fail in CI with "Missing: better-sqlite3@12.9.0 from lock file". Co-Authored-By: RuFlo <ruv@ruv.net> * fix(deps): restore better-sqlite3 as direct root optionalDependency The #2736 dedup fix pinned better-sqlite3 to 12.9.0 via a pure "overrides" entry (no direct dependency), which removed the top-level hoisted node_modules/better-sqlite3 that used to exist under the looser ">=12.8.0" override. That silently broke the "graph schema smoke (ADR-130 P1)" CI job: its "npm rebuild better-sqlite3" step (and graph-edge-writer.ts's bare `await import('better-sqlite3')`, which Node resolves by walking up node_modules from the importing file) had nothing at the root to rebuild/resolve, so _openDb() failed closed and insertGraphEdge() silently returned false — 8/22 assertions failing. Restoring the direct optionalDependency (matching what this file already had historically, before the override-only refactor) gives npm a concrete anchor to hoist a single deduplicated copy back to root node_modules, preserving the override's version pin while keeping the graph-edge-writer's resolution path intact. Regenerated package-lock.json accordingly. Verified locally: `npm rebuild better-sqlite3` now finds and rebuilds the root copy, and a direct insertGraphEdge() call against a real better-sqlite3 handle returns true. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(deps): resync v3/pnpm-lock.yaml specifier after merging main Merging main (which carries #2748's direct better-sqlite3 dependency on @claude-flow/cli) forward into this branch left the recorded lockfile specifier at the pre-override "^12.9.0" while this branch's own overrides.better-sqlite3 pin ("12.9.0", exact) is now the effective spec pnpm resolves against — a one-line drift causing `pnpm install --frozen-lockfile` to fail across the whole v3-ci.yml matrix. Regenerated to match. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(memory): force WAL checkpoint before close in graph-edge-writer The "graph schema smoke (ADR-130 P1)" CI job started failing deterministically on Linux runners after the better-sqlite3 dedup (this branch) pinned a single 12.9.0 build tree-wide: TEST 2/3 insert-then-reread assertions found embedding_ref/source_id undefined immediately after _resetBridgeDb() closed the write connection. _resetBridgeDb()'s close() relied on SQLite's implicit "last connection close" PASSIVE auto-checkpoint to flush the WAL into the main file before a same-process sql.js reader re-reads the raw file. That's best-effort, not guaranteed synchronous, and evidently more prone to leaving data in the WAL under Linux CI's I/O/lock timing than on Windows (did not reproduce locally). Not a correctness regression in 12.9.0 itself — a latent timing assumption the dedup made newly observable by changing which exact build backs every connection. Force a blocking `wal_checkpoint(TRUNCATE)` before close so the main file always reflects committed writes before any other reader (same process or otherwise) touches it, regardless of platform timing. Co-Authored-By: RuFlo <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Jul 20, 2026
Patch release covering the statusline/memory-integrity fix batch merged in #2746, #2747, #2748, #2749 (issues #2733, #2735, #2736, #2737, #2742). Also fixes an npm EOVERRIDE conflict this batch introduced: v3/@claude-flow/cli/package.json had gained both a direct optionalDependency on better-sqlite3 (^12.9.0, from #2748) and a self-referential override pinned to an exact "12.9.0" (from #2736) for the same package — npm publish rejects an override that doesn't match its own direct dependency's spec string. Aligned the override to the same "^12.9.0" range so the dedup guarantee holds without the conflict. Co-Authored-By: RuFlo <ruv@ruv.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2737.
@claude-flow/cli'sdoctorcommand has real memory-database integrity checks (checkMemoryIntegrity,checkMemoryContent,checkMemoryEmbeddingCoverage— added for #2677), but they were registered only undercomponentMap['memory'](--component memory), never in theallChecksarray a baredoctorordoctor --fixactually runs. The default run's only memory probe wascheckMemoryDatabase— plainexistsSync+statSync— which reportspassfor any file that exists and can be stat'd, corrupt or not. A genuinely corrupt.swarm/memory.dbcould sail through a baredoctorrun reporting "All checks passed! System is healthy."The wiring gap
allChecks(default run): onlycheckMemoryDatabase(existence/size only).componentMap['memory'](--component memoryonly): the full doctor: --component memory asserts existence, not function — passes a store that is 99.97% empty (and one SQLite calls malformed) #2677 trio.Three-part fix
New default-run check —
checkMemoryStructuralIntegrity, added toallChecks. Uses nativebetter-sqlite3(WAL-aware — SQLite resolves the sibling-walfile itself; the existingsql.jschecks are WAL-blind because they onlyreadFileSync()the main db image) and runsPRAGMA quick_check(bounded — per sqlite.org it skips UNIQUE/index-vs-table cross-checks thatintegrity_checkdoes, cheap enough to run unconditionally). Explicitly labeled "structural-only" in both the row name and every message so it's never confused with the deeper--component memoryprobe.checkMemoryDatabase's reported row is relabeled "Memory Database Presence" to be honest about its existence-only scope — behavior unchanged, it stays inallCheckstoo.Strengthened
checkMemoryIntegrity(deep path,--component memoryonly — the full doctor: --component memory asserts existence, not function — passes a store that is 99.97% empty (and one SQLite calls malformed) #2677 trio stays there, unchanged in registration). Now prefers nativebetter-sqlite3PRAGMA integrity_check(adds the index↔table + UNIQUE verificationquick_checkskips, and is WAL-aware) whenbetter-sqlite3is installed, falling back to the originalsql.jsprobe — now explicitly labeled "main-image-only" (WAL-blind) — when it's not.Definitive corruption now fails, not warns — in both checks, a
"file is not a database"/"malformed"result now maps tofail, WITH an encryption-at-rest carve-out: before concluding "malformed," both checks sniff the file for the product's RFE1 magic bytes (reusingisEncryptedBlob()fromencryption/vault.ts— the exact same detectorcheckEncryptionAtRestuses), so a legitimately encrypted-at-rest DB is never misclassified as corrupt.better-sqlite3is added to@claude-flow/cli'soptionalDependencies(matching the existing pattern in@claude-flow/memoryand@claude-flow/hooks), with an ambient module declaration inoptional-modules.d.ts. Every code path that touches it degrades gracefully (sql.js fallback, or awarn) onMODULE_NOT_FOUND— never a silent pass.What I deliberately did not touch
agentdb_health/memory_stats— out of scope per the issue, tracked separately.doctor --fix's behavior beyond inheriting the new default check (it already runsallChecks).ruflo-doctorskill/plugin docs.checkMemoryContent/checkMemoryEmbeddingCoverage— kept exactly as-is;checkMemoryIntegrityfailing hard is enough to prevent a false "all healthy" (checks are ordered so the earliest chain-break is the first red the user sees).Test plan
__tests__/doctor-2737-memory-structural.test.ts(7 cases), driven throughdoctorCommand.action()(the real entry point, not the check functions in isolation):better-sqlite3) → defaultdoctorrun passes, including the new check.fail; overall run reportsfailed > 0,success: false,exitCode: 1— never the "all healthy" false negative.encryptBuffer) is not misclassified as corrupt.--component memorystill runs the full, unchanged 4-check trio; the new default-run check is deliberately absent from it.checkMemoryIntegritydirectly: fails on malformed+unencrypted, does not hard-fail on legitimately-encrypted.doctor-*.test.tssuite (19 tests total) still passes — no regressions.node bin/cli.js doctoragainst a scratch dir with a garbage.swarm/memory.db:checkMemoryDatabase, exit 0, "All checks passed! System is healthy.").Re-verified against a valid DB (exit 0,
quick_check: ok) and via--component memory(full trio, nativeintegrity_checkpath confirmed with[native, WAL-aware]in the message).Note: while manually verifying
--component memoryend-to-end via the compiled CLI binary, I hit a libuv assertion crash (Assertion failed: !(handle->flags & UV_HANDLE_CLOSING),src\win\async.c) on process exit in this sandbox. I confirmed it reproduces identically on unmodifiedmain(stashed my diff, rebuilt, same crash) — it's a pre-existing Windows/Node-24/native-module environment issue in this sandbox, not something this change introduces, and unrelated to the actual check logic (which completes and prints correct results before the crash occurs during teardown).🤖 Generated with RuFlo