Skip to content

Latest commit

 

History

History
229 lines (186 loc) · 12.4 KB

architecture.md

File metadata and controls

229 lines (186 loc) · 12.4 KB

Architecture

sciogen has exactly two phases. Indexing turns source code into a typed, confidence-scored knowledge graph plus multi-granularity embeddings. Querying answers questions against those stores deterministically. There is no LLM anywhere, no token awareness, and no ML routing — the query mode is fixed by which API method the caller invokes.

                         Phase 1 — Indexing
 ┌──────────┐  ┌────────────┐  ┌────────────┐  ┌──────────┐  ┌─────────┐
 │ discover │→ │ tree-sitter│→ │ normalizer │→ │ resolver │→ │  graph  │
 │ + differ │  │   parse    │  │  (AST→IR)  │  │ (refs→   │  │ builder │
 └──────────┘  └────────────┘  └────────────┘  │  edges)  │  └─────────┘
      │              parallel across cores     └──────────┘       │
      │                                              │            ▼
      ▼                                              │       ┌─────────┐
 ┌──────────┐                                        │       │ KuzuDB  │
 │  SQLite  │◄───── hashes, symbols, refs ───────────┘       └─────────┘
 └──────────┘
      │              ┌───────────┐               ┌──────────┐
      └─ embed dedup │ embedder  │─── vectors ──►│ ChromaDB │
                     └───────────┘               └──────────┘

                         Phase 2 — Querying
 structural  → KuzuDB Cypher/BFS      (get_symbol, get_callers, analyze_impact,
                                       get_dependencies, get_diff_impact)
 semantic    → ChromaDB cosine        (search mode="semantic")
 hybrid      → ChromaDB seeds + KuzuDB expansion
                                      (search mode="hybrid", get_context_for_task)

How it is used

sciogen (bare, no subcommand) indexes the current working directory: it prints the banner, runs the pipeline with live progress, and ends with the ✓ Done! N nodes · M edges summary. A coding agent reads that output to understand the codebase; no Python API call is needed. In the same workspace, sciogen explore opens the browser GUI over the stored graph, showing every node and its interconnections. The Python API and MCP server are alternative front doors to the same query engine — the CLI is the primary one.

Phase 1 — Indexing

1. Discovery + two-tier differ (sciogen/discovery.py)

One os.walk over the project, pruning ignored/hidden directories in place. Change detection is two-tier — this is the hot path of re-indexing:

  1. Stat fast path: if a file's (size, mtime_ns) matches what SQLite recorded, the file is never opened. Re-indexing a repo with no changes costs one stat per file, not a read + hash. (Measured on sciogen itself: first index 25.4s, no-op re-index 0.2s.)
  2. SHA256 truth: only stat-changed files are read and hashed. If the hash still matches (a touch, a branch switch that restored content), the stat signature is refreshed and the file is not re-parsed.

mtime is never trusted to declare a file changed — only to prove it unchanged against a recorded signature. SHA256 stays the source of truth, which is what makes this correct in CI/CD. The watchdog file watcher is opt-in dev-mode sugar, never the primary mechanism.

2. Tree-sitter parse (sciogen/parsing/)

tree-sitter-language-pack provides grammars for 100+ languages behind one API. Parsers are cached per process and per language.

3. AST normalizer (sciogen/normalize/)

Converts language-specific ASTs into a universal IR (sciogen/ir.py): Node (kinds: Module/File/Class/Function/Type/Chunk/External), Edge, CallSite, ImportRecord, MutationRecord, bundled per file into a FileIR.

This is the only layer with language knowledge. Python, JS, and TS normalizers produce identical IR shapes; everything downstream is language-agnostic. Adding a language is one normalizer class registered in sciogen/normalize/__init__.py; unsupported languages fall back to a generic normalizer that still yields a searchable FileNode.

Normalizers extract: classes (bases, docstrings), functions (params, return type, cyclomatic complexity, docstring), imports in all forms, call sites (with the enclosing class for self/this dispatch), attribute mutations, and — for functions over 40 lines — ChunkNodes whose embedding_text is a normalized token summary (identifiers, strings, comments), never raw code.

4. Symbol resolver (sciogen/resolve/resolver.py)

Runs after parsing, before graph building — tree-sitter gives syntax, not semantics; without this step a CALLS edge is a string, not a pointer. For each file the resolver builds a scope table from its imports, then traces every recorded reference to the exact definition:

Resolution path Confidence
Local definition, sibling method (self.x()), exact test-name match 1.0
Through an explicit import to an indexed file 0.9
Through a module alias (svc.login() after import svc) 0.85
Method through an imported class 0.8
Unique project-wide name match (no import proves it) 0.7
Dynamic receiver, method name unique project-wide 0.4
External / unresolvable 0.2–0.5

Unresolvable references are kept as edges to external:* placeholder nodes with low confidence — dropping them would fabricate a misleadingly clean graph. Static analysis is incomplete, not wrong; the score preserves the information and the caller picks the threshold. The one deliberate exception: stdlib-noise methods on unknown receivers (x.append(), p.then()) are elided, because external:append edges carry no signal.

The resolver consumes a language-neutral FileRefs bundle rather than the FileIR directly — see incremental re-resolution.

5. Graph builder (KuzuDB, sciogen/stores/graph.py)

Materializes nodes and typed edges via Cypher. Confidence is never filtered at write time — the graph stores everything; filtering happens at query time.

Phase 2 — Querying

All queries are deterministic and return typed dataclasses (sciogen/query/results.py) — never raw source. The caller decides whether to open a file; sciogen hands it exact file:line_start-line_end references.

  • Structural — KuzuDB only. Multi-hop traversals (get_callers, analyze_impact) run as a Python BFS over prepared single-hop Cypher queries, giving exact hop counts and path confidence (the minimum edge confidence along the best path — a chain is only as strong as its weakest hop).
  • Semantic — ChromaDB cosine similarity over the granularity the API call selects (or all four, merged by score).
  • Hybrid — vector hits become seed nodes; the graph expands their neighborhood; expanded nodes inherit a decayed score (seed × 0.7 × edge confidence) so ranking stays meaningful.

get_diff_impact exists separately from analyze_impact because a diff is a multi-symbol change set: hunk ranges are intersected with symbol line ranges, and one traversal runs from all changed symbols at once — deduplication is free and hops means "distance to the nearest changed symbol".

Storage: triple store

Store Role Location
KuzuDB Graph topology, edge confidence .sciogen/graph.kuzu/
ChromaDB Embeddings at 4 granularities .sciogen/vectors/
SQLite Hashes, symbol table, reference records, embed hashes, schema version, parse errors .sciogen/meta.db

All three are embedded — no server processes anywhere in sciogen.

ChromaDB granularities (one collection each): file-level answers "how is the auth module structured?", class-level "classes that handle sessions", function-level "functions that validate JWTs", chunk-level "the block that rate-limits login". No single granularity answers all of those well.

Schema version lives in SQLite with a forward-migration registry (stores/meta.py), because the graph schema will change; migrations beat forcing a full rebuild on every upgrade.

Performance architecture

The deliberate speed decisions, each with its rationale. The contract they serve: first index fast, re-index near-instant, queries sub-second.

  • Parallel parsing. Parsing + normalization is CPU-bound and per-file independent. Batches above Config.min_files_for_parallel (32) fan out to a ProcessPoolExecutor; FileIR is a plain picklable dataclass. Small batches stay serial — on Windows, pool startup costs more than parsing a handful of files. Tune with SCIOGEN_WORKERS.
  • One node table in KuzuDB. All node kinds share a single Symbol table with a kind column: one primary-key hash index instead of six, uniform DETACH DELETE WHERE file = $p for incremental updates, and simpler Cypher. Edges stay typed (one rel table per kind) because traversals are always kind-scoped. Node types remain first-class in the API; the discriminator is an implementation detail.
  • Batched transactional writes. Cached prepared statements inside one explicit transaction per pipeline stage — not one auto-commit per row.
  • Embedding deduplication. Embedding is the slowest stage by an order of magnitude. Each node's embedding text is content-hashed and compared against the stored hash; only changed summaries are re-embedded.
  • Incremental re-resolution. When file A changes, call edges out of unchanged file B may be stale. sciogen persists every file's reference records in SQLite, so B is re-resolved from stored records without re-parsing. Importer discovery over-approximates by module basename — a false positive costs one cheap re-resolve; a miss would leave a stale edge, so the bias is deliberate.
  • Query LRU keyed by index version. Every index run bumps a monotonic index_version; the cache key includes it, so one bump invalidates every cached result and repeated agent queries on an unchanged index are dict lookups.
  • Lazy imports. kuzu/chromadb import only when a store is first touched; sciogen --help never pays the multi-second chromadb import.

Embeddings

Embedders are pluggable (sciogen/embed/):

  • nomic-embed-code (default when sciogen[embeddings] is installed) — local, free, code-optimized. Loaded lazily; nothing leaves the machine.
  • HashingEmbedder (automatic fallback, or SCIOGEN_EMBEDDER=hash) — deterministic feature hashing of identifier n-grams. Zero deps, instant, keyword-level retrieval quality. Used in tests.

What gets embedded is always the normalized text summary of a node (name + file + params + docstring + call list) — never raw source, which embeds noise. The embedder identity is persisted with the index; opening with a different embedder raises instead of silently mixing vector spaces.

sciogen explore

Serializes the full graph (nodes, edges, in-degrees) to JSON, injects it into a single self-contained HTML template (sciogen/cli/explore/template.html, kept as a static asset with a {{GRAPH_JSON}} placeholder), and opens it in the browser. No server, no port — consistent with the embedded-everything design. The view is collapsed to files by default; clicking a file expands its classes and functions, clicking a function reveals its call edges. All progressive disclosure is client-side over the already-loaded JSON. Node color encodes kind, node size encodes in-degree (hotspots), edge dash style encodes confidence (solid ≥0.8, dashed 0.5–0.8, dotted <0.5). It is a snapshot: re-run sciogen index then sciogen explore after code changes.

Hard constraints

  • No LLM calls anywhere inside sciogen.
  • No token counting, budgeting, or context-window awareness.
  • Query methods return typed node objects, never raw source.
  • No confidence filtering at write time — store everything, filter at query time.
  • The MCP layer contains no logic; capabilities land in the Python API first.
  • The file watcher is opt-in dev mode; the SHA256 differ is the CI-safe mechanism.
  • Embed normalized summaries, never raw code.