Skip to content

fix(doctor): wire real memory-DB integrity checks into the default doctor run - #2748

Merged
ruvnet merged 2 commits into
mainfrom
fix/2737-doctor-memory-integrity
Jul 20, 2026
Merged

fix(doctor): wire real memory-DB integrity checks into the default doctor run#2748
ruvnet merged 2 commits into
mainfrom
fix/2737-doctor-memory-integrity

Conversation

@ruvnet

@ruvnet ruvnet commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #2737.

@claude-flow/cli's doctor command has real memory-database integrity checks (checkMemoryIntegrity, checkMemoryContent, checkMemoryEmbeddingCoverage — added for #2677), but they were registered only under componentMap['memory'] (--component memory), never in the allChecks array a bare doctor or doctor --fix actually runs. The default run's only memory probe was checkMemoryDatabase — plain existsSync + statSync — which reports pass for any file that exists and can be stat'd, corrupt or not. A genuinely corrupt .swarm/memory.db could sail through a bare doctor run reporting "All checks passed! System is healthy."

The wiring gap

Three-part fix

  1. New default-run checkcheckMemoryStructuralIntegrity, added to allChecks. Uses native better-sqlite3 (WAL-aware — SQLite resolves the sibling -wal file itself; the existing sql.js checks are WAL-blind because they only readFileSync() the main db image) and runs PRAGMA quick_check (bounded — per sqlite.org it skips UNIQUE/index-vs-table cross-checks that integrity_check does, cheap enough to run unconditionally). Explicitly labeled "structural-only" in both the row name and every message so it's never confused with the deeper --component memory probe. checkMemoryDatabase's reported row is relabeled "Memory Database Presence" to be honest about its existence-only scope — behavior unchanged, it stays in allChecks too.

  2. Strengthened checkMemoryIntegrity (deep path, --component memory only — the full doctor: --component memory asserts existence, not function — passes a store that is 99.97% empty (and one SQLite calls malformed) #2677 trio stays there, unchanged in registration). Now prefers native better-sqlite3 PRAGMA integrity_check (adds the index↔table + UNIQUE verification quick_check skips, and is WAL-aware) when better-sqlite3 is installed, falling back to the original sql.js probe — now explicitly labeled "main-image-only" (WAL-blind) — when it's not.

  3. Definitive corruption now fails, not warns — in both checks, a "file is not a database" / "malformed" result now maps to fail, WITH an encryption-at-rest carve-out: before concluding "malformed," both checks sniff the file for the product's RFE1 magic bytes (reusing isEncryptedBlob() from encryption/vault.ts — the exact same detector checkEncryptionAtRest uses), so a legitimately encrypted-at-rest DB is never misclassified as corrupt.

better-sqlite3 is added to @claude-flow/cli's optionalDependencies (matching the existing pattern in @claude-flow/memory and @claude-flow/hooks), with an ambient module declaration in optional-modules.d.ts. Every code path that touches it degrades gracefully (sql.js fallback, or a warn) on MODULE_NOT_FOUND — never a silent pass.

What I deliberately did not touch

  • agentdb_health / memory_stats — out of scope per the issue, tracked separately.
  • doctor --fix's behavior beyond inheriting the new default check (it already runs allChecks).
  • The ruflo-doctor skill/plugin docs.
  • checkMemoryContent / checkMemoryEmbeddingCoverage — kept exactly as-is; checkMemoryIntegrity failing hard is enough to prevent a false "all healthy" (checks are ordered so the earliest chain-break is the first red the user sees).

Test plan

  • New __tests__/doctor-2737-memory-structural.test.ts (7 cases), driven through doctorCommand.action() (the real entry point, not the check functions in isolation):
    • Healthy DB (built with real better-sqlite3) → default doctor run passes, including the new check.
    • Malformed DB (garbage bytes) → new check reports fail; overall run reports failed > 0, success: false, exitCode: 1 — never the "all healthy" false negative.
    • A real RFE1-encrypted blob (built with the product's own encryptBuffer) is not misclassified as corrupt.
    • --component memory still runs the full, unchanged 4-check trio; the new default-run check is deliberately absent from it.
    • checkMemoryIntegrity directly: fails on malformed+unencrypted, does not hard-fail on legitimately-encrypted.
  • Full existing doctor-*.test.ts suite (19 tests total) still passes — no regressions.
  • Manual repro: node bin/cli.js doctor against a scratch dir with a garbage .swarm/memory.db:
    ✓ Memory Database Presence: ...\.swarm\memory.db (0.00 MB) — existence/size only, not a health check (see Memory Structural Integrity)
    ✗ Memory Structural Integrity (quick_check): ...\.swarm\memory.db — quick_check probe threw: file is not a database (unencrypted) [structural-only]
    Summary: 13 passed, 11 warnings, 1 failed
    Some checks failed. Please address the issues above.
    
    Process exit code: 1 (previously: silent pass via checkMemoryDatabase, exit 0, "All checks passed! System is healthy.").
    Re-verified against a valid DB (exit 0, quick_check: ok) and via --component memory (full trio, native integrity_check path confirmed with [native, WAL-aware] in the message).

Note: while manually verifying --component memory end-to-end via the compiled CLI binary, I hit a libuv assertion crash (Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), src\win\async.c) on process exit in this sandbox. I confirmed it reproduces identically on unmodified main (stashed my diff, rebuilt, same crash) — it's a pre-existing Windows/Node-24/native-module environment issue in this sandbox, not something this change introduces, and unrelated to the actual check logic (which completes and prints correct results before the crash occurs during teardown).

🤖 Generated with RuFlo

ruvnet added 2 commits July 20, 2026 16:06
Fixes #2737

Bare `doctor` / `doctor --fix` ran `allChecks`, whose only memory probe
was `checkMemoryDatabase` — existsSync()+statSync() only, so it PASSES
for any file that exists and can be stat'd, corrupt or not. The real
integrity/content/embedding checks added for #2677
(checkMemoryIntegrity/checkMemoryContent/checkMemoryEmbeddingCoverage)
were only reachable via `--component memory`, never the default run —
so a genuinely corrupt .swarm/memory.db could report "All checks
passed! System is healthy." on a bare `doctor` call.

Three-part fix:

1. Add `checkMemoryStructuralIntegrity` to the default `allChecks` run —
   a bounded, native (better-sqlite3, WAL-aware) `PRAGMA quick_check`,
   explicitly labeled "structural-only" in its name/message. Cheaper
   than the deep trio (skips UNIQUE/index cross-checks quick_check
   deliberately omits) but bounded enough to run unconditionally.
   `checkMemoryDatabase`'s reported row is relabeled "Memory Database
   Presence" to be honest about its existence-only scope; behavior is
   unchanged and it stays in `allChecks` too.

2. Strengthen `checkMemoryIntegrity` (deep, `--component memory` only,
   full #2677 trio unchanged there): prefer native better-sqlite3
   `PRAGMA integrity_check` (adds index<->table cross-checking + UNIQUE
   verification quick_check skips, and is WAL-aware) when better-sqlite3
   is installed, falling back to the existing sql.js probe — now
   explicitly labeled "main-image-only" (WAL-blind) — when it's not.

3. A definitively malformed, unencrypted DB now maps to `fail`, not
   `warn`, in both checks. Before failing, both sniff the file for the
   product's RFE1 encryption-at-rest magic bytes (reusing
   isEncryptedBlob() from encryption/vault.ts, the same detector
   checkEncryptionAtRest uses) so a legitimately encrypted-at-rest DB is
   never misclassified as corrupt.

better-sqlite3 added to @claude-flow/cli's optionalDependencies
(matching the existing optional-dep pattern in @claude-flow/memory and
@claude-flow/hooks) with an ambient module declaration in
optional-modules.d.ts; every code path that touches it degrades to the
sql.js fallback (or a warn) on MODULE_NOT_FOUND.

Verified:
- New __tests__/doctor-2737-memory-structural.test.ts (7 cases): healthy
  DB passes including the new check; malformed DB fails the new check
  and the overall run reports failed>0/exitCode 1 (not "all healthy");
  a real RFE1-encrypted blob is not misclassified as corrupt;
  `--component memory` still runs the full unchanged trio; both
  encryption-carve-out cases for checkMemoryIntegrity directly.
- Full existing doctor-*.test.ts suite (19 tests total) still passes.
- Manual repro: `node bin/cli.js doctor` against a scratch dir with a
  garbage .swarm/memory.db now prints "Memory Structural Integrity
  (quick_check): ... file is not a database" as a hard FAIL and exits 1
  (previously: silent PASS via checkMemoryDatabase, exit 0). Re-verified
  with a valid DB (exit 0, quick_check: ok) and via
  --component memory (full trio, native integrity_check path
  confirmed).

Co-Authored-By: RuFlo <ruv@ruv.net>
The doctor memory-integrity fix added a direct better-sqlite3
dependency to @claude-flow/cli/package.json but didn't regenerate
the workspace lockfile, causing pnpm install --frozen-lockfile to
fail in CI (v3-ci.yml's lockfile-drift guard and Test V3 Packages).

Co-Authored-By: RuFlo <ruv@ruv.net>
@ruvnet
ruvnet merged commit c47462e into main Jul 20, 2026
118 checks passed
@ruvnet
ruvnet deleted the fix/2737-doctor-memory-integrity branch July 20, 2026 21:02
ruvnet added a commit that referenced this pull request Jul 20, 2026
Merging main (which carries #2748's direct better-sqlite3 dependency
on @claude-flow/cli) forward into this branch left the recorded
lockfile specifier at the pre-override "^12.9.0" while this branch's
own overrides.better-sqlite3 pin ("12.9.0", exact) is now the
effective spec pnpm resolves against — a one-line drift causing
`pnpm install --frozen-lockfile` to fail across the whole v3-ci.yml
matrix. Regenerated to match.

Co-Authored-By: RuFlo <ruv@ruv.net>
ruvnet added a commit that referenced this pull request Jul 20, 2026
…cy tree (#2736) (#2746)

* fix(security): dedup better-sqlite3 to patched 12.9.0 across dependency tree

Fixes #2736 (defense-in-depth mitigation).

SQLite's documented WAL-Reset Bug (sqlite.org/wal.html §11) is a race
between overlapping checkpoint operations and a WAL-resetting commit
that can make a later checkpoint skip a committed transaction, silently
losing acknowledged writes and/or corrupting index/table structure.
Affects SQLite 3.7.0-3.51.2; fixed in 3.51.3, backported to 3.44.6 and
3.50.7.

agentdb's own published dependency floor (`better-sqlite3@^11.8.1`,
confirmed in this tree resolving to 11.10.0, which bundles vulnerable
SQLite 3.49.2) created a mixed-engine tree alongside this repo's own
direct dependency on better-sqlite3 12.9.0 (patched SQLite 3.53.0, via
v3/@claude-flow/memory). Multiple concurrent connections (daemon, MCP
servers, CLI invocations) issuing wal_checkpoint(PASSIVE) after every
memory-bridge store is exactly the race precondition this bug requires.

This forces a single deduplicated better-sqlite3 build tree-wide via
pnpm overrides, pinned to the exact version this repo's own
@claude-flow/memory package already depends on directly (12.9.0,
bundling patched SQLite 3.53.0) — no new version to vet, just true
deduplication of agentdb's nested resolution into the already-patched
copy.

Changes (following this repo's existing #2112-lesson override
convention — multiple package.json files, each consumed a different
way):
- v3/package.json: pnpm.overrides (the v3 pnpm workspace root — this
  is where agentdb's better-sqlite3@11.10.0 actually lived pre-fix)
- v3/@claude-flow/cli/package.json: npm-style overrides (effective
  when @claude-flow/cli is npm-installed standalone via npx)
- package.json (root "claude-flow" umbrella): bumped the existing
  ">=12.8.0" override to an exact "12.9.0" pin, AND added a
  pnpm.overrides block — the bare "overrides" key alone is silently
  ignored by pnpm (only read by npm), so the pre-existing override was
  a no-op for local `pnpm install` at repo root despite being present
  in package.json for a while
- ruflo/package.json: bumped ">=12.8.0" to exact "12.9.0" (npm-style,
  effective for `npx ruflo@latest`)
- v3/pnpm-lock.yaml, pnpm-lock.yaml: regenerated via `pnpm install`

Verification:
- `pnpm why -r better-sqlite3` (v3/) and `pnpm why better-sqlite3`
  (root): exactly one resolved version, 12.9.0, everywhere
- Confirmed via realpath resolution (not hoisted symlink) from each of
  the 4 installed agentdb versions' own on-disk directories
  (1.6.1, 2.0.0-alpha.3.4, 2.0.0-alpha.3.7, 3.0.0-alpha.17) that each
  now binds the same deduplicated better-sqlite3@12.9.0 copy
- `db.prepare('select sqlite_version()').get()` from that binding
  returns 3.53.0 (>= 3.51.3 patched floor)
- v3/@claude-flow/memory test suite: 438/442 passing; the 4 failures
  are pre-existing Windows path-separator and perf-threshold issues
  unrelated to SQLite (none of the failing test files touch
  better-sqlite3/sqlite at all)

Not fixed here (upstream, out of reach from this repo):
- agentdb's own published package.json floor (`^11.8.1`) is
  unchanged — this override is a same-repo mitigation, not a fix to
  agentdb itself. agentdb needs the same bump upstream.

Not touched here (explicitly out of scope, separate/riskier change):
- The wal_checkpoint(PASSIVE)-after-every-store pattern in
  memory-bridge.ts (#2558) is left as-is; the issue calls that
  "defense-in-depth" / optional, and dedup alone removes the
  mixed-engine race precondition regardless.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(deps): regenerate package-lock.json for better-sqlite3 dedup override

package.json pins the better-sqlite3 override to 12.9.0 (patched,
fixes the WAL-reset bug) but package-lock.json still resolved a stray
top-level better-sqlite3@12.10.0 and was missing the nested
better-sqlite3@12.9.0 entries under @claude-flow/memory, agentdb, and
agentic-flow — causing npm ci to fail in CI with "Missing:
better-sqlite3@12.9.0 from lock file".

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(deps): restore better-sqlite3 as direct root optionalDependency

The #2736 dedup fix pinned better-sqlite3 to 12.9.0 via a pure
"overrides" entry (no direct dependency), which removed the top-level
hoisted node_modules/better-sqlite3 that used to exist under the
looser ">=12.8.0" override. That silently broke the "graph schema
smoke (ADR-130 P1)" CI job: its "npm rebuild better-sqlite3" step
(and graph-edge-writer.ts's bare `await import('better-sqlite3')`,
which Node resolves by walking up node_modules from the importing
file) had nothing at the root to rebuild/resolve, so _openDb() failed
closed and insertGraphEdge() silently returned false — 8/22 assertions
failing.

Restoring the direct optionalDependency (matching what this file
already had historically, before the override-only refactor) gives
npm a concrete anchor to hoist a single deduplicated copy back to
root node_modules, preserving the override's version pin while
keeping the graph-edge-writer's resolution path intact. Regenerated
package-lock.json accordingly.

Verified locally: `npm rebuild better-sqlite3` now finds and rebuilds
the root copy, and a direct insertGraphEdge() call against a real
better-sqlite3 handle returns true.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(deps): resync v3/pnpm-lock.yaml specifier after merging main

Merging main (which carries #2748's direct better-sqlite3 dependency
on @claude-flow/cli) forward into this branch left the recorded
lockfile specifier at the pre-override "^12.9.0" while this branch's
own overrides.better-sqlite3 pin ("12.9.0", exact) is now the
effective spec pnpm resolves against — a one-line drift causing
`pnpm install --frozen-lockfile` to fail across the whole v3-ci.yml
matrix. Regenerated to match.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(memory): force WAL checkpoint before close in graph-edge-writer

The "graph schema smoke (ADR-130 P1)" CI job started failing
deterministically on Linux runners after the better-sqlite3 dedup
(this branch) pinned a single 12.9.0 build tree-wide: TEST 2/3
insert-then-reread assertions found embedding_ref/source_id undefined
immediately after _resetBridgeDb() closed the write connection.

_resetBridgeDb()'s close() relied on SQLite's implicit "last
connection close" PASSIVE auto-checkpoint to flush the WAL into the
main file before a same-process sql.js reader re-reads the raw file.
That's best-effort, not guaranteed synchronous, and evidently more
prone to leaving data in the WAL under Linux CI's I/O/lock timing than
on Windows (did not reproduce locally). Not a correctness regression
in 12.9.0 itself — a latent timing assumption the dedup made newly
observable by changing which exact build backs every connection.

Force a blocking `wal_checkpoint(TRUNCATE)` before close so the main
file always reflects committed writes before any other reader (same
process or otherwise) touches it, regardless of platform timing.

Co-Authored-By: RuFlo <ruv@ruv.net>
ruvnet added a commit that referenced this pull request Jul 20, 2026
Patch release covering the statusline/memory-integrity fix batch
merged in #2746, #2747, #2748, #2749 (issues #2733, #2735, #2736,
#2737, #2742).

Also fixes an npm EOVERRIDE conflict this batch introduced:
v3/@claude-flow/cli/package.json had gained both a direct
optionalDependency on better-sqlite3 (^12.9.0, from #2748) and a
self-referential override pinned to an exact "12.9.0" (from #2736)
for the same package — npm publish rejects an override that doesn't
match its own direct dependency's spec string. Aligned the override
to the same "^12.9.0" range so the dedup guarantee holds without the
conflict.

Co-Authored-By: RuFlo <ruv@ruv.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default doctor run never executes its memory integrity checks (a corrupt DB still ends "All checks passed! System is healthy.")

1 participant