fix: rebuild index when mode adds capabilities

[astandrik]

Summary

• persist the effective index capability level in the Project node as index_mode
• force a full rebuild when the requested mode requires capabilities missing from the stored index
• preserve additive data on downgrades by separating requested discovery mode from stored effective extraction mode
• migrate legacy, malformed, or non-exact mode metadata with one conservative rebuild
• keep existing persistent artifacts synchronized after metadata-only downgrades and full mode upgrades

Root cause

Incremental routing originally considered stored file hashes and the discovered-file threshold, but not the capability level of the stored index. This caused unchanged fast → moderate upgrades to take the incremental no-op path.

After the first fix, changed-file downgrades still re-extracted with the weaker requested mode. That could remove stored capabilities such as SIMILAR_TO edges or full-mode Macro nodes.

The metadata parser also compared yyjson strings with strcmp(). A valid JSON string containing "full" + U+0000 + "garbage" therefore matched "full" at the embedded NUL and could incorrectly select incremental/no-op instead of the conservative rebuild route.

Mode transitions

Stored	Requested	Route / effective behavior
fast	moderate / full	full rebuild
moderate	full	full rebuild
same mode	same mode	existing incremental/no-op path
more capable	less capable	requested mode controls discovery; stored mode is the effective mode for changed-file extraction and capability post-passes
missing / unknown / malformed / non-exact	any	one conservative full rebuild

Downgrades keep the stored index_mode; p->mode and Project metadata are not weakened. Files outside requested discovery continue to use the existing mode-skipped/hash mechanism. A no-change downgrade refreshes coverage metadata without rebuilding graph data. If a persistent artifact already exists, both metadata-only downgrades and full mode upgrades refresh it after the SQLite update.

The public MCP API and SQLite schema are unchanged. Existing changed-file threshold and ADR preservation behavior are unchanged.

Regression coverage

• unchanged fast → moderate: fail-before incremental no-op left SIMILAR_TO at 0; pass-after rebuild creates similarity data and records index_mode=moderate
• changed-file moderate → fast: preserves SIMILAR_TO and index_mode=moderate; the following moderate no-op remains stable
• changed-file full → moderate: preserves full-only Macro nodes and index_mode=full; the following full no-op remains stable
• legacy Project metadata {} triggers a rebuild and restores index_mode
• a NUL-suffixed index_mode is rejected with length-aware yyjson comparison, rebuilt once, and normalized
• metadata-only downgrade round-trips through an existing persistent artifact with requested coverage and the stronger stored mode intact
• full mode upgrade refreshes an artifact created by an earlier run even when persistence is not re-enabled on the upgrade call

Verification

Rebased onto main at 7235a4d; PR head is bfe8ca6.

• fail-before downgrade suite: 222 passed, 2 failed
• fail-before embedded-NUL regression: 224 passed, 1 failed
• fail-before stale-artifact regressions: each failed on the expected missing metadata/capability assertion
• ASAN_OPTIONS=detect_leaks=0 build/c/test-runner pipeline — 226 passed
• scripts/test.sh — passed on bfe8ca6, including production build, watchdog regressions, and security allow-list checks
• make -f Makefile.cbm cbm — passed
• make -f Makefile.cbm lint-format lint-no-suppress — passed
• git diff --check upstream/main...HEAD — passed

scripts/lint.sh was attempted locally. The environment has no cppcheck, and its Homebrew clang-tidy reports broad pre-existing errors outside this diff. The independent format/no-suppression gates passed; the GitHub lint job is the authoritative full lint check.

[DeusData]

Thanks for the mode-capability fix and regression coverage. This is triaged as a normal-priority 0.9.1-rc correctness PR. Review will focus on the capability ordering, rebuilding only when a requested mode adds unavailable graph data, preserving valid incremental state, and keeping same-mode reindex behavior unchanged.

[astandrik]

Thanks for the mode-capability fix and regression coverage. This is triaged as a normal-priority 0.9.1-rc correctness PR. Review will focus on the capability ordering, rebuilding only when a requested mode adds unavailable graph data, preserving valid incremental state, and keeping same-mode reindex behavior unchanged.

@DeusData I ran into this while working with a really large repository: https://github.com/ydb-platform/ydb

Version 8 wasn’t able to index it at all, but version 9 managed to handle it successfully. Thanks for the improvements — I really love your codebase memory tool!

[Copilot]

Pull request overview

This PR fixes incremental routing so it accounts for the capability level of the stored index (via Project.index_mode), forcing a full rebuild when a requested mode requires capabilities the existing index does not have, while preserving stronger stored capabilities during requested-mode downgrades.

Changes:

• Persist index_mode on the Project node during structure pass and parse it during routing to detect capability upgrades.
• Route fast → moderate/full (and moderate → full) transitions to a full rebuild instead of an incremental no-op.
• Run incremental changed-file extraction using an explicit effective_mode (stored mode) to preserve additive capabilities across downgrades, and add regression tests for mode transitions and malformed/NUL-suffixed metadata.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/test_pipeline.c	Adds regression tests covering mode upgrade rebuilds, downgrade preservation behavior, and malformed/NUL-suffixed index_mode metadata handling.
src/pipeline/pipeline.c	Persists Project.index_mode, parses stored mode, and blocks incremental routing when requested mode needs missing capabilities.
src/pipeline/pipeline_internal.h	Updates incremental pipeline contract to accept an effective_mode for changed-file extraction.
src/pipeline/pipeline_incremental.c	Applies effective_mode to incremental extraction (incl. macro gating) while keeping requested mode for discovery/exclusions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DeusData]

Thanks @astandrik — reviewed carefully and we want this: the capability-aware rebuild is correct (the FULL<MODERATE<FAST ordering and index_mode_covers logic check out), the NUL-suffix guard via length-aware yyjson_equals_str is a real fix, and the RED suite is genuine. Persisting index_mode on the Project node and the downgrade-preserve semantics are both accepted — they fix the real 'reindex produces a smaller graph' symptom (this overlaps #1037's second half).

One thing before merge: the base is ~53 commits behind main and pipeline_incremental.c/pipeline.c have moved substantially (kernel-perf work), and the cbm_pipeline_run_incremental signature you changed is exactly the kind of thing that'll conflict — please rebase onto current main and re-run the RED/GREEN suite so we can re-review at the new SHA. Then we'll merge (with a merge commit, to keep your DCO sign-off).

[astandrik]

@DeusData
Rebased onto current main (7235a4d) at bfe8ca6 and reran the validation chain.

The rebased revision also covers the persistence edge cases found during review: no-change downgrades refresh requested coverage while retaining the stronger stored/effective mode, changed files are re-extracted with effective_mode, and existing artifacts are refreshed after both metadata-only downgrades and full mode upgrades. The added regressions have fail-before/pass-after local DB and artifact round-trip coverage.

Local results on bfe8ca6: pipeline suite 226 passed; scripts/test.sh passed including production build, watchdogs, and security checks; clang-format/NOLINT and git diff --check passed. Full local scripts/lint.sh remains environment/baseline-blocked by missing cppcheck and pre-existing repo-wide clang-tidy findings; GitHub lint is running on the exact head.