feat(extract): add --no-dedup so incremental merges can skip fuzzy dedup (#2881)

[rajarshidattapy]

Fixes #2881.

Implements the issue's primary request: a CLI opt-out from entity dedup, so incremental updates on large graphs can skip the fuzzy pass and get the shrink guard for free.

The gap

build_merge() already accepts dedup: bool = True, but the CLI hardcoded dedup=True on the incremental path (cli.py, the if incremental_mode: branch) with no flag to override it. That matters because the fuzzy pass runs over the combined node set — existing graph plus new chunk — so a small diff merged into a large graph compares every new node against every pre-existing one.

And as the issue points out, the safety net for exactly this case is inverted:

# Safety check: refuse to SILENTLY drop nodes (#479, reworked in #2497)
if had_graph and not dedup:

The guard is disabled in the only mode reachable from the CLI, and armed in the mode that was unreachable from it.

What this adds

graphify extract <path> --no-dedup:

• passes dedup=False to both build() (full build) and build_merge() (incremental merge);
• therefore arms build_merge's Prevent partial-chunk overwrite in --update: pre-build node-count assertion + build_merge() helper #479 shrink guard, which refuses to drop nodes from sources this run neither re-extracted nor pruned;
• surfaces that refusal as a clean [graphify extract] … error and exit 1 rather than an unhandled ValueError traceback. graph.json on disk is untouched when it fires, so the existing graph survives the refusal;
• is rejected together with --dedup-llm (exit 2). The LLM tiebreaker is pass 3 inside the dedup pipeline, so combining them would be a silent no-op that still demands an API key.

Documented in the README flag list and in graphify extract's usage line.

One note on the traced root cause

The issue was filed against 0.8.28, and on current v8 both dedup passes now skip code symbols outright:

# Code symbols are excluded from fuzzy matching too: two functions with
# similar long names in different files … must not be fuzzy-merged (#1205)
if _is_code(node):
    continue

So the specific losses reported — AdminPanel.tsx, LoginModal.tsx, Python builtins like date/int — should no longer be reachable through pass 2 on the current release; #1205, #1243, #2182, #1284 and #2576 each narrowed that path after 0.8.28. I have deliberately not claimed this PR fixes those merges, since I could not reproduce them on v8.

What is verifiably still true is the part this PR addresses: the incremental path had no opt-out, and the shrink guard was unreachable from the CLI. If the reporter still sees shrinkage on 0.9.x, --no-dedup now turns a silent 5-10% loss into a loud refusal naming the first unexplained node — which is the diagnostic the issue was missing.

Not included

Scoping pass 2's candidate set to the current diff plus its direct neighbours (the issue's alternative suggestion). That changes dedup quality for every user on every run and wants its own benchmarking against the corpora that motivated the existing thresholds; it did not belong in the same change as an opt-out flag.

graphify update needs no flag: it routes through watch._rebuild_code → build_from_json, which never calls deduplicate_entities.

Tests

tests/test_no_dedup_flag.py — the flag reaches build(), dedup stays on by default, the flag reaches build_merge() on a genuine second (incremental) run, and the --dedup-llm conflict exits 2. Three of the four fail without the change.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Adds a --no-dedup flag to graphify extract that passes dedup=False through both the build and incremental build_merge paths (the latter previously hardcoded dedup=True), and arms build_merge's #479 shrink guard so an incremental merge can't silently drop untouched files' nodes. Rejects --no-dedup combined with --dedup-llm as mutually exclusive (exit 2), and catches the shrink guard's ValueError to print the refusal instead of a traceback. Covers the flag, the default-on behavior, the merge path, and the conflict in tests/test_no_dedup_flag.py.

Worth a look

• Tests patch build/build_merge on graphify.build but dispatch_command calls _build/_build_merge aliases — tests/test_no_dedup_flag.py:38 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
• Dedup tests monkeypatch the wrong build symbols — tests/test_no_dedup_flag.py:44 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 267 functions depend on the 86 functions this change touches.

Health — this change adds coupling hotspots:

• new: dispatch_command() — 2 callers, 117 callees
• new: _stale_graph_sources() — 7 callers, 6 callees
• new: _run_hook_guard() — 4 callers, 7 callees
• new: test_poisoned_manifest_is_healed() — 0 callers, 6 callees

Verification — 267 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 224 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify dispatch\_command.

The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)

· 4 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Adds a --no-dedup flag to graphify extract that passes dedup=not no_dedup through both the _build and _build_merge paths in dispatch_command, and rejects the flag alongside --dedup-llm with exit 2. On the incremental merge path, --no-dedup arms build_merge's shrink guard, and its ValueError is now caught and reported as a one-line error (exit 1) instead of a traceback. Documents the flag in the README and covers the default-on, flag-off, incremental-merge, and mutual-exclusion cases in the new tests/test_no_dedup_flag.py.

No blocking issues surfaced. 1 lower-confidence candidate did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 269 functions depend on the 88 functions this change touches.

Health — this change adds coupling hotspots:

• new: dispatch_command() — 2 callers, 117 callees
• new: _stale_graph_sources() — 7 callers, 6 callees
• new: _run_hook_guard() — 4 callers, 7 callees
• new: test_poisoned_manifest_is_healed() — 0 callers, 6 callees

Verification — 269 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 226 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify dispatch\_command.

The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)

· 4 more finding(s) on lines outside this diff (see the check run).

[safishamsi]

Shipped in v0.9.48 via authorship-preserving cherry-pick, with an added graph-level test locking the exact-id-collapse invariant. Thanks @rajarshidattapy! Release: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.48