docs: record CodeGraph worktree discovery

[joe-crick]

Stage 0 for the CodeGraph worktree-aware integration plan.\n\nChanges:\n- Documents current CCE path-based storage identity.\n- Documents CodeGraph worktree mismatch behavior and machine-readable surfaces.\n- Records selected Git base-ref strategy and MVP limitations.\n\nStack note:\n- Upstream push permission is denied for joe-crick, so later stacked stages are open in the fork with bases chained branch-to-branch: joe-crick#1 through https://github.com/joe-crick/code-context-engine/pull/9.\n\nChecks:\n- uv run python -m pytest -o addopts='' tests/test_project_storage_dir.py tests/integration/test_git_context.py

[rajkumarsakthivel]

Review

Thanks for the work here. The architecture is clean (git identity -> storage layout -> overlay backend -> fusion -> MCP) and the test coverage for the new modules is solid. This is a meaningful feature addition.

A few things to address before merging:

Scope and labeling

This PR adds ~2960 lines of new feature code across 37 files (new modules, MCP server changes, CLI commands, config additions). The title says docs: but this is a feat: PR. Please relabel so the changelog and release notes reflect what actually shipped.

Code concerns

1. WorktreeOverlayBackend doesn't implement StorageBackend formally. The codebase uses getattr checks for count_chunks, needs_overlay_index, etc. throughout cli.py and mcp_server.py. If StorageBackend gains a new required method, this will silently break. Consider subclassing or a protocol.

2. Private attribute access. _count_chunks falls back to backend._vector_store.count(). This pattern is duplicated in multiple files. Should be a proper interface method on StorageBackend.

3. No file size or type filtering. _source_for_worktree_path reads entire files into memory for every modified/added path on each context_search call. Large binary or generated files would be read and scanned with no guard.

4. Blocking subprocess in async context. CodeGraphClient._run_json and git helpers use synchronous subprocess.run but are called from async code paths. This blocks the event loop. Consider asyncio.create_subprocess_exec or run_in_executor.

5. Token estimation. _structural_token_count uses len(text) / 3.3 as a heuristic. The existing codebase has proper token counting. Using a different method here creates inconsistency.

6. Absolute paths in discovery doc. docs/codegraph-worktree-integration-discovery.md references /home/joe/Webstorm_Projects/codegraph/.... Please remove machine-specific paths.

What's good

• Backward compatible: structural_provider defaults to "off", existing users see no behavior change
• Clean module separation with proper __init__.py exports
• Good test coverage for overlay backend, fusion, merge, codegraph provider, and budgeting
• The layered architecture makes sense

Worth pursuing, just needs the above addressed first.

[Copilot]

Pull request overview

This PR lays groundwork for a worktree-aware CodeGraph integration by adding Git repository/worktree identity, shared-base + per-worktree overlay storage semantics, and optional CodeGraph-derived “structural context” that is fused into MCP context_search output—alongside Phase 0 discovery documentation.

Changes:

• Add Git repository/worktree identification, base SHA resolution, and worktree diff modeling; introduce worktree-aware storage layout plus diagnostics/benchmark CLI commands.
• Introduce a semantic base+overlay storage backend and retrieval fusion with token budgeting; wire into MCP server indexing and context_search.
• Add a CodeGraph structural provider adapter + structural merge logic, document new structural configuration, and record discovery notes.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_worktree_doctor.py	Adds tests for worktree-aware doctor report and CLI JSON output.
tests/test_worktree_benchmark.py	Adds tests for worktree overlay benchmark helper and CLI output.
tests/test_git_storage_layout.py	Verifies shared repo base + per-worktree overlay storage layout behavior.
tests/test_git_repository_context.py	Adds tests for repository/worktree identity, base SHA strategy, and diff modeling.
tests/test_config.py	Adds tests for new structural configuration defaults and YAML mapping.
tests/structural/test_merge.py	Tests structural base+overlay merge semantics (sources/relationships/impact).
tests/structural/test_codegraph_provider.py	Tests CodeGraph client/provider JSON handling and mapping to structural models.
tests/storage/test_worktree_overlay.py	Tests WorktreeOverlayBackend behaviors (shadowing, merging, neighbors, caching).
tests/retrieval/test_fusion.py	Tests semantic+structural retrieval fusion and token budgeting behavior.
tests/retrieval/test_budgeting.py	Tests disclosure-level token budget resolution.
tests/integration/test_mcp_server.py	Extends MCP integration tests for overlay correctness, structural context, and indexing storage base override.
tests/integration/test_mcp_empty_index.py	Updates empty-index behavior test to use count_chunks() and storage override wiring.
src/context_engine/utils.py	Adds resolve_project_storage_dir() helper and refactors storage dir resolution.
src/context_engine/structural/models.py	Introduces provider-neutral structural models (symbols, sources, relationships, context).
src/context_engine/structural/merge.py	Adds structural merge logic for base+worktree overlay with shadowing rules.
src/context_engine/structural/codegraph.py	Adds CodeGraph client wrapper and base structural provider adapter.
src/context_engine/structural/base.py	Defines structural provider protocol and provider status model.
src/context_engine/structural/init.py	Exposes structural API surface (models/providers/merge).
src/context_engine/storage/worktree_overlay.py	Adds overlay storage backend that hides shadowed base paths and merges results.
src/context_engine/retrieval/fusion.py	Adds fusion of overlay/base chunks + structural context with token budgeting.
src/context_engine/retrieval/budgeting.py	Defines progressive disclosure token budgets and resolution logic.
src/context_engine/retrieval/init.py	Exports retrieval fusion and budgeting APIs.
src/context_engine/integration/mcp_server.py	Wires structural context into context_search, adds indexing storage override, and uses backend count_chunks().
src/context_engine/indexer/pipeline.py	Adds storage_base_override parameter to indexing pipeline to support shared-base indexing.
src/context_engine/git/storage.py	Introduces repository-scoped storage layout for base + worktree overlays.
src/context_engine/git/repository.py	Adds Git repository/worktree identity resolution and safe base SHA selection.
src/context_engine/git/doctor.py	Adds worktree-aware “doctor” diagnostic report generation.
src/context_engine/git/diff.py	Adds Git worktree diff model capturing modified/added/deleted/renamed/untracked.
src/context_engine/git/benchmark.py	Adds helper to benchmark diff discovery cost.
src/context_engine/git/init.py	Exposes new git/worktree helpers as a package API.
src/context_engine/data/tools_reference.md	Documents that context_search may include structural context when enabled.
src/context_engine/data/instructions.md	Documents worktree-aware/structural behavior of context_search.
src/context_engine/config.py	Adds structural.* configuration fields and YAML mapping.
src/context_engine/cli.py	Adds doctor and worktree-benchmark commands; switches serve/index paths to worktree-aware storage base/backend.
README.md	Links the new CodeGraph worktree discovery document from Documentation section.
docs/wiki/Configuration.md	Documents structural config and structural context behavior in context_search.
docs/codegraph-worktree-integration-discovery.md	Adds Phase 0 discovery notes and selected base-ref strategy write-up.

Suppressed comments (1)

src/context_engine/integration/mcp_server.py:429

• _source_for_worktree_path() reads the entire file into memory just to build a small excerpt. For large files this can add significant per-query overhead; a bounded read is enough for excerpting/searching.

    path = worktree_root / rel_path
    try:
        content = path.read_text(encoding="utf-8", errors="ignore")
    except OSError:
        return None

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rajkumarsakthivel]

Copilot flagged additional items worth addressing alongside the earlier review:

1. _worktree_structural_overlay() reads every changed file per query (mcp_server.py:420). No size cap, no binary filtering. For large worktrees this makes context_search O(changed_files). Add a file-size limit and skip binary extensions.

2. _symbol_ref() line number check (mcp_server.py:407). if line: treats line 0 as falsy. Use if line is not None: instead.

3. refresh_diff() uses stale git context (cli.py:3608). The GitRepositoryContext captured at startup has a fixed head_sha. If HEAD moves after startup, get_worktree_diff() stops including new commits. Re-resolve the git context on refresh.

4. Discovery doc base SHA description (codegraph-worktree-integration-discovery.md:158). Doc says "no base SHA when no safe base can be established" but the code falls back to HEAD. Update the doc to match implementation.

5. _source_for_worktree_path() (mcp_server.py:429) reads entire files via path.read_text() just for excerpting. Use bounded reads for the excerpt use case.

These are on top of the items from the earlier review.