feat: NeuG embedded graph DB, Cypher CLI, and GDS Leiden clustering (opt-in) - #2895
feat: NeuG embedded graph DB, Cypher CLI, and GDS Leiden clustering (opt-in)#2895BingqingLyu wants to merge 1 commit into
Conversation
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an opt-in NeuG embedded-graph-DB pipeline: introduces graphify cypher and graphify delta-cluster CLI commands, a GRAPHIFY_NEUG=1 path in extract that builds graph.db and runs GDS Leiden clustering, plus a --cluster-on-files flag. Wires up storage.py (init/ingest/export, god-node/god-file/surprising-connection queries, Leiden subgraph clustering, freeze-assign delta analysis) and extends serve.py with NeuG-backed context filtering and graph-stats/query tools. Updates README and ARCHITECTURE docs for the new neug extra and commands, and adds storage tests.
Worth a look
- --allow-partial guard removed from no-cluster overwrite path —
graphify/cli.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- NeuG clustered global merge path no longer calls global_add —
graphify/cli.py:4004· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- --global no longer merges in the Neug clustered extract path —
graphify/cli.py:4237· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- MCP tool executes arbitrary client-supplied Cypher against graph.db —
graphify/serve.py:1983· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
- --clear-ast no longer propagated when saving no-cluster manifest —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 706 functions depend on the 331 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 124 callees - new:
_query_graph_text()— 20 callers, 9 callees - new:
_score_query()— 15 callers, 5 callees - new:
_query_terms()— 20 callers, 3 callees - new:
delta_analyze()— 5 callers, 10 callees - new:
run_benchmark()— 16 callers, 3 callees - new:
_stale_graph_sources()— 7 callers, 6 callees - new:
_build_server()— 2 callers, 18 callees - …and 25 more — each is listed as a finding
Verification — 706 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 706 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_run\_cli (not a proof).
The verifier ran both versions of \_run\_cli on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_build\_server.
The verifier did not have enough to check \_build\_server, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ImportError — names the real obstacle, not a sampling gap)
· 14 grounded finding(s) anchored inline below; 19 more finding(s) on lines outside this diff (see the check run).
| return node_types | ||
|
|
||
|
|
||
| def ingest_extraction( |
There was a problem hiding this comment.
ingest_extraction()
12 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| shutil.copy2(node_csv, dest_dir / f"{tag}_nodes.csv") | ||
|
|
||
|
|
||
| def cluster_on_files( |
There was a problem hiding this comment.
cluster_on_files()
high coupling complexity (Ca·Ce = 25).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def find_god_nodes(conn: object, top_n: int = 10) -> list[dict]: |
There was a problem hiding this comment.
find_god_nodes()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return score, reasons | ||
|
|
||
|
|
||
| def find_surprising_connections( |
There was a problem hiding this comment.
find_surprising_connections()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def cluster_by_neug( |
There was a problem hiding this comment.
cluster_by_neug()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_find_surprising_connections(tmp_db): |
There was a problem hiding this comment.
test_find_surprising_connections()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_label_communities_by_hub(tmp_db): |
There was a problem hiding this comment.
test_label_communities_by_hub()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --- incremental delta analysis (freeze-assign leiden) --- | ||
|
|
||
|
|
||
| def test_run_leiden_freeze_assign(tmp_db): |
There was a problem hiding this comment.
test_run_leiden_freeze_assign()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_run_leiden_freeze_assign_resolution(tmp_db): |
There was a problem hiding this comment.
test_run_leiden_freeze_assign_resolution()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert changes["dissolved_communities"][0]["old_size"] == 3 | ||
|
|
||
|
|
||
| def test_delta_analyze(tmp_db): |
There was a problem hiding this comment.
test_delta_analyze()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
21778bb to
ca05c66
Compare
|
Addressing the review findings:
Also fixed a duplicate |
|
Regarding the inline coupling findings: These are expected given the design of the NeuG adapter layer:
These coupling metrics reflect the inherent complexity of integrating an embedded graph database with the existing pipeline. The functions are cohesive (each has a single responsibility) even if they have multiple dependencies. |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) alter behavior, breaking input(s) attached.
Behavior changes: to\_html changes behavior, here is the input that shows it.
The verifier found a concrete input on which to\_html behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.
Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.
Evidence: On input \{"G":"\(lambda \_g: \(\_g\.add\_nodes\_from\(\[\(1, \{\}\), \(2, \{\}\), \(3, \{\}\)\]\), \_g\.add\_edges\_from\(\[\(1, 2, \{\}\), \(1, 3, \{\}\), \(2, 3, \{\}\)\]\), \_g\)\[\-1\]\)\(\_\_import\_\_\('networkx'\)\.Graph\(\)\)","communities":"\{'k': 'v'\}","output\_path":"'racecar'","community\_labels":"\{'k': 'v'\}","member\_counts":"\{'a': 1\}","node\_limit":"2","learning\_overlay":"\{'a': 1, 'b…, the old code produced False but the new code produces None. Paste that input straight into a regression test.
Graphify review — findings
Adds an opt-in NeuG embedded graph-database pipeline: new graphify cypher and graphify delta-cluster CLI commands in dispatch_command, a --cluster-on-files extract flag, and GRAPHIFY_NEUG=1 extract handling that builds graph.db and runs freeze-assign Leiden clustering. Introduces graphify/storage.py (init_db, ingest_extraction, ingest_communities, delta_analyze, find_surprising_connections, etc.) with matching tests/test_storage.py coverage. Documents the neug extra and new commands in README and ARCHITECTURE.
Worth a look
- MCP tool executes caller-supplied Cypher without read-only validation —
graphify/serve.py:1984· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- File-level delta analysis uses fixed temp table names —
graphify/storage.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- --no-cluster no-change fast path prunes only graph_stale_sources, not excluded_files —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Default extract no longer writes the manifest —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Early graph.json prune skips DB graph manifest invalidation —
graphify/cli.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 707 functions depend on the 332 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 124 callees - new:
_query_graph_text()— 20 callers, 9 callees - new:
_score_query()— 15 callers, 5 callees - new:
_query_terms()— 20 callers, 3 callees - new:
delta_analyze()— 5 callers, 10 callees - new:
run_benchmark()— 16 callers, 3 callees - new:
_stale_graph_sources()— 7 callers, 6 callees - new:
_build_server()— 2 callers, 18 callees - …and 25 more — each is listed as a finding
Verification — 707 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 707 function(s) in the blast radius were not formally verified this run
Formal verification
Behavior changes: to\_html changes behavior, here is the input that shows it.
The verifier found a concrete input on which to\_html behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.
Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.
Evidence: On input \{"G":"\(lambda \_g: \(\_g\.add\_nodes\_from\(\[\(1, \{\}\), \(2, \{\}\), \(3, \{\}\)\]\), \_g\.add\_edges\_from\(\[\(1, 2, \{\}\), \(1, 3, \{\}\), \(2, 3, \{\}\)\]\), \_g\)\[\-1\]\)\(\_\_import\_\_\('networkx'\)\.Graph\(\)\)","communities":"\{'k': 'v'\}","output\_path":"'racecar'","community\_labels":"\{'k': 'v'\}","member\_counts":"\{'a': 1\}","node\_limit":"2","learning\_overlay":"\{'a': 1, 'b…, the old code produced False but the new code produces None. Paste that input straight into a regression test.
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_obsidian\_safe\_stem (not a proof).
The verifier ran both versions of \_obsidian\_safe\_stem on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in to\_graphml (not a proof).
The verifier ran both versions of to\_graphml on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify extract.
The verifier did not have enough to check extract, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `cache_root` is annotated `Path | None` — outside the synthesizable primitive/collection set
Could not verify: Could not verify extract\_cpp.
The verifier did not have enough to check extract\_cpp, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_extract\_generic.
The verifier did not have enough to check \_extract\_generic, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify extract\_markdown.
The verifier did not have enough to check extract\_markdown, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_resolve\_markdown\_link.
The verifier did not have enough to check \_resolve\_markdown\_link, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `source_dir` is annotated `Path` — outside the synthesizable primitive/collection set
No difference found (not proven): No behavior difference found in \_bedrock\_response\_text (not a proof).
The verifier ran both versions of \_bedrock\_response\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Verification did not run: Verification did not run for \_call\_azure.
The verification could not execute (an environment/toolchain issue, not a statement about the code).
Guarantee: No guarantee, the check itself did not complete.
Note: Detail: harness produced no verdict (rc=124): timeout after 30s
Could not verify: Could not verify \_call\_bedrock.
The verifier did not have enough to check \_call\_bedrock, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ParamValidationError — names the real obstacle, not a sampling gap)
Verification did not run: Verification did not run for \_call\_claude.
The verification could not execute (an environment/toolchain issue, not a statement about the code).
Guarantee: No guarantee, the check itself did not complete.
Note: Detail: harness produced no verdict (rc=124): timeout after 30s
Could not verify: Could not verify \_call\_claude\_cli.
The verifier did not have enough to check \_call\_claude\_cli, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly RuntimeError — names the real obstacle, not a sampling gap)
Verification did not run: Verification did not run for \_call\_openai\_compat.
The verification could not execute (an environment/toolchain issue, not a statement about the code).
Guarantee: No guarantee, the check itself did not complete.
Note: Detail: harness produced no verdict (rc=124): timeout after 30s
· 14 grounded finding(s) anchored inline below; 19 more finding(s) on lines outside this diff (see the check run).
| return node_types | ||
|
|
||
|
|
||
| def ingest_extraction( |
There was a problem hiding this comment.
ingest_extraction()
12 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| shutil.copy2(node_csv, dest_dir / f"{tag}_nodes.csv") | ||
|
|
||
|
|
||
| def cluster_on_files( |
There was a problem hiding this comment.
cluster_on_files()
high coupling complexity (Ca·Ce = 25).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def find_god_nodes(conn: object, top_n: int = 10) -> list[dict]: |
There was a problem hiding this comment.
find_god_nodes()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return score, reasons | ||
|
|
||
|
|
||
| def find_surprising_connections( |
There was a problem hiding this comment.
find_surprising_connections()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def cluster_by_neug( |
There was a problem hiding this comment.
cluster_by_neug()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_find_surprising_connections(tmp_db): |
There was a problem hiding this comment.
test_find_surprising_connections()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_label_communities_by_hub(tmp_db): |
There was a problem hiding this comment.
test_label_communities_by_hub()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| # --- incremental delta analysis (freeze-assign leiden) --- | ||
|
|
||
|
|
||
| def test_run_leiden_freeze_assign(tmp_db): |
There was a problem hiding this comment.
test_run_leiden_freeze_assign()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _close(db, conn) | ||
|
|
||
|
|
||
| def test_run_leiden_freeze_assign_resolution(tmp_db): |
There was a problem hiding this comment.
test_run_leiden_freeze_assign_resolution()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert changes["dissolved_communities"][0]["old_size"] == 3 | ||
|
|
||
|
|
||
| def test_delta_analyze(tmp_db): |
There was a problem hiding this comment.
test_delta_analyze()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
Summary
graphify cypher) and MCP server (cypher_querytool)graph.dbdelta-cluster: incremental community analysis command;--baselineseeds from an external clusteringGRAPHIFY_NEUG=1or an existinggraph.db; the default extract path is unchangedMotivation
Graphify currently uses NetworkX +
graph.jsonas its core graph storage. This architecture has bottlenecks:graph.json→ merge → rebuild → re-serialize (O(full graph) even for single-file changes)Why NeuG?
NeuG is a lightweight embedded graph database (C++ core, Python bindings):
pip install neugis all it takes)Architecture
Dual-engine coexistence, each independently consuming extraction data:
When
GRAPHIFY_NEUG=1is set (orgraph.dbalready exists), the NeuG pipeline also runs:graph.dbvia COPY FROM (bulk) or MERGE (incremental)graph.dbgraph.jsonfromgraph.db(not dual-write) — ensures downstream tools (wiki generation, HTML visualization, community labeling) continue to work unchanged, maintaining full backward compatibilityChanges
graphify/storage.pygraphify/cli.pydelta-clustersubcommand, segfault guard for tiny graphsgraphify/__main__.pygraphify cypherCLI commandgraphify/serve.pycypher_queryMCP tool for AI agentsgraphify/llm.pypyproject.tomlneug>=0.1.3optional dependency (neugextra +allextra)README.mdARCHITECTURE.mdstorage.pymodule descriptiontests/test_storage.py,test_cypher_cli.py)Usage
Test Plan
pytest tests/test_storage.py tests/test_cypher_cli.py -v— all tests passedtree-sitter-hcl)cypher_querytool end-to-end verifiedgraphify extract .runs normally (silent skip)Note
This PR builds on the NeuG integration proposed in #1056. While #1056 established the core storage layer (graph.db, Cypher queries, MCP tool), this PR adds GDS extension support for native and incremental community detection.
Review Findings Addressed
The following review findings have been fixed:
--allow-partialguard missing in NeuG clustered path — Added shrink guard before_cluster_by_neugcall for RT-parity with NetworkX path_save_manifestcall withclear_astparameterclear_astnot propagated in NeuG no-cluster path — Addedclear_astparameter to_save_manifestcallneugkey in pyproject.toml — Removed duplicate entryThe MCP
cypher_querytool executing arbitrary Cypher is by design — it's intended for AI agents to flexibly query the knowledge graph. Since it's a local MCP server, the security risk is limited.