Code indexing

Code Indexing ETL

This document describes how the code indexing ETL pipeline works. Unlike SDLC entities, code versions can exist in parallel across branches, so the same file can look different on main vs. a feature branch.

If we want Orbit to answer questions about code, it needs to understand relationships at any given branch and commit.

The ETL pipeline:

Reads code from GitLab repositories through the Rails internal API
Transforms it into a call graph and file system hierarchy
Writes entities and relationships to ClickHouse

What is a call graph?

Here is an example of a call graph between two files:

// fileA.ts
export class Foo {
  hello() {
    const bar = new Bar();
    bar.myMethod();
  }
}

// fileB.ts
export class Bar {
  myMethod() {
    console.log("Hello from Bar.myMethod()");
  }
}

Explanation:

fileA.ts defines class Foo with a method hello().
Inside hello(), it creates an instance of Bar and calls myMethod().
fileB.ts defines class Bar with the myMethod() implementation.

We create a call graph between the two files, and the result looks like this:

graph LR
    %% Define subgraphs
    subgraph DIR["my_directory -"]
        subgraph FILEA["fileA.ts"]
            A1["class Foo"]
            A2["hello()"]
        end

        subgraph FILEB["fileB.ts"]
            B1["class Bar"]
            B2["myMethod()"]
        end
    end

    %% Edges
    FILEA --> A1
    FILEB --> B1
    A1 -->|"method"| A2
    B1 -->|"method"| B2
    A2 -.->|"calls"| B2

    %% Styling
    %% Lighter directory background
    style DIR fill:#fafafa,stroke:#bbb,stroke-width:1px,color:#111,font-weight:bold
    style FILEA fill:#dce9ff,stroke:#90c2ff,stroke-width:1px,color:#111,font-weight:bold
    style FILEB fill:#d9f7d9,stroke:#90e090,stroke-width:1px,color:#111,font-weight:bold

    %% Class and method nodes (colored distinctly)
    style A1 fill:#cce0ff,stroke:#4080ff,stroke-width:1px,color:#003366
    style A2 fill:#dde9ff,stroke:#4080ff,stroke-width:1px,color:#003366
    style B1 fill:#ccffcc,stroke:#33aa33,stroke-width:1px,color:#004400
    style B2 fill:#ddffdd,stroke:#33aa33,stroke-width:1px,color:#004400

    %% Edge styling
    linkStyle 0,1,2,3 stroke:#444,stroke-width:1.5px
    linkStyle 4 stroke:#5b32ff,stroke-width:2.5px,stroke-dasharray: 4 2,background-color:#f0f0ff

Architecture overview

graph LR
    Siphon["Siphon CDC"] --> ExtNATS["NATS JetStream<br/>(Siphon stream)"]
    ExtNATS --> Dispatcher["SiphonCodeIndexingTaskDispatcher<br/>(DispatchIndexing mode)"]
    ExtNATS --> Backfill["NamespaceCodeBackfillDispatcher<br/>(DispatchIndexing mode)"]
    Dispatcher --> IntNATS["NATS JetStream<br/>(GKG_INDEXER stream)"]
    Backfill --> IntNATS
    IntNATS --> Handler["CodeIndexingTaskHandler<br/>(Indexer mode)"]
    Handler --> Rails["Rails internal API<br/>(archive download)"]
    Rails --> CodeGraph["code-graph<br/>(parse + analyze)"]
    CodeGraph --> Arrow["ArrowConverter"]
    Arrow --> ClickHouse["ClickHouse"]

Components

Component	Description
`code-graph` crate	Single crate containing the v2 pipeline stack in `src/v2/` and shared code-indexing utilities
`treesitter-visit` crate	Tree-sitter wrapper crate kept separate for compile-time isolation and used by the generic v2 language pipelines
`indexer` crate	NATS consumer, ETL engine, Siphon task dispatcher, namespace backfill dispatcher, code indexing task handler, Arrow conversion, ClickHouse writes
`orbit-server`	Server binary, runs in Indexer mode for code indexing
NATS JetStream	Message broker with durable delivery between Siphon and GKG
NATS KV	Distributed lock store to prevent concurrent indexing of the same project
ClickHouse	Columnar OLAP database storing the data lake and the property graph
Rails internal API	Proxies repository operations: project info lookups, archive downloads, and blob streaming for content resolution (length-delimited protobuf)

For background on Siphon CDC, NATS, and ClickHouse architecture, see the SDLC indexing design document.

Data storage

Orbit code data is stored in a separate ClickHouse database.

For .com this is expected to run in a separate instance.
For small dedicated environments and self-hosted instances, this can be done in the same instance as the main ClickHouse database. This choice ultimately depends on what the operators think is best for their environment.

Some numbers

As of November 2025, the GitLab monolith has over 4000 branches considered "active" (committed to within the last 3 months). Even more are considered "stale" (last committed more than 3 months ago).

Locally, with the limited support for Ruby. We currently index about 300,000 definitions and over 1,000,000 relationships.

For simplicity's sake, let's say we want to keep an active code index for branches that are considered "active". This would require us to index (300,000 definitions 4000 branches) = 1.2 billion definitions. It would also require (1,000,000 relationships 4000 branches) = 4 billion relationships just for the GitLab monolith. This is simply not feasible if we extrapolate this to all the repositories in .com.

Use cases

The scale problem depends on what we actually need from the graph. The target use cases:

Detect whether a merge request changes existing behavior in unexpected ways
Assess the blast radius of a merge request on the rest of the codebase
Help developers explore unfamiliar code and understand architectural patterns
Guide refactoring and feature work by surfacing dependencies
Trace how a vulnerability propagates through the call graph
Expose queryable APIs for code exploration and analysis
Generate documentation from the indexed codebase

None of these require indexing every active branch for every repository.

Indexing the main branch

The first milestone is indexing the main branch for every repository. This covers the majority of the use cases listed above. A strategy for active branches is described in a later section.

Extract

The extract phase uses a two-hop dispatch model to decouple Siphon CDC consumption from indexing.

Rails writes to a dedicated p_knowledge_graph_code_indexing_tasks table only when a push lands on the default branch of a namespace with Orbit indexing enabled. These rows are replicated via Siphon CDC to NATS JetStream:

gkg_siphon_stream.p_knowledge_graph_code_indexing_tasks

Each task carries project_id, ref, commit_sha, and traversal_path directly. So the handler does not need to call Rails for default branch validation or query ClickHouse for the namespace hierarchy.

For the full rationale behind this approach and the alternatives that were considered, see ADR 005: PostgreSQL task table for code indexing triggers.

Dispatch

The SiphonCodeIndexingTaskDispatcher runs as a ScheduledTask in DispatchIndexing mode. On each run it batch-pulls pending Siphon CDC messages and decodes the protobuf LogicalReplicationEvents. It then publishes a CodeIndexingTaskRequest (JSON) per event to the internal GKG_INDEXER NATS stream. The subject pattern code.task.indexing.requested.<project_id>.<branch> combined with max_messages_per_subject: 1 deduplicates in-flight requests per project and branch.

This separation lets task dispatching be stopped independently from the indexer. The handler keeps draining whatever was already dispatched, but no new work enters the pipeline.

Namespace backfill dispatch

When a namespace first enables Orbit indexing, its existing projects need to be indexed even though no push events have occurred yet. The NamespaceCodeBackfillDispatcher handles this by consuming knowledge_graph_enabled_namespaces CDC events from Siphon. For each newly enabled namespace it:

Resolves the namespace's traversal path from namespace_traversal_paths
Queries project_namespace_traversal_paths to find all projects under that namespace
Publishes a CodeIndexingTaskRequest for each project to the GKG_INDEXER stream

Pending projects are published in bounded batches, not after every namespace has been enumerated. A fleet-wide sweep therefore does not have to hold the whole fleet's projects before the first publish. Every schema migration produces one such sweep, since the new version's checkpoint table starts empty. Each namespace contributes at most its share of one batch. The share is the batch size (schedule.tasks.code-backfill.publish_window) divided by the namespaces that still had pending projects on the previous run. Each batch is shuffled before it is published. A namespace with a large backlog therefore takes its turn alongside every other namespace, instead of filling the queue. It is topped back up on each run as its projects are indexed.

These backfill requests omit branch and commit_sha. The handler resolves the default branch from the Rails internal API at processing time.

The backfill dispatches a project that has no indexed_at, fewer than MAX_CODE_ATTEMPTS (5) attempts, and no attempt in the last CODE_RETRY_AFTER (1 hour). A project at 5 attempts is a gap, and only a new push indexes it. A run whose failure NATS redelivers gives its attempt back, if it still holds the project lock. So attempts counts the runs that ended the message or never returned.

A namespace whose every project is checkpointed is drained. Once per schema version, the backfill tick then tombstones the namespace's code rows that are older than their project's checkpoint. These are rows left by a previous version or by a run that died before checkpointing. The sweep runs inside the tick. So it is capped at schedule.tasks.code-backfill.stale_sweeps_per_tick namespaces per tick, and the rest wait for later ticks. Without the cap a fleet-wide re-index held dispatch for the length of that burst while indexers sat idle. Such a re-index drains thousands of small namespaces within minutes.

Handler

The CodeIndexingTaskHandler runs in Indexer mode and subscribes to CodeIndexingTaskRequest messages from the GKG_INDEXER stream. It deserializes the JSON request and, if no branch was provided (e.g. from a namespace backfill), resolves the default branch from the Rails internal API. It then acquires a lock on the project + branch combination to prevent other workers from indexing the same branch concurrently. If the lock is held, the handler returns the task to NATS for the time left on the lease plus one renew interval. That delivery spends no checkpoint attempt and skips the engine retry check. NATS still counts it against nats.max_deliver (5 by default). So at the defaults a task waits behind about 23 minutes of lock contention. Then NATS stops redelivery, and the max-deliveries reconciler deletes the message. The next push indexes the newest commit. A deferred task also arrives with a higher NATS delivery count. So under the job-timeout policy (2 attempts), its first budget timeout dead-letters it with no retry. The checkpoint check skips the task later if the other run already indexed past it.

Example NATS KV:

Key: project.{project_id}.{base64_encoded_branch}
Value: { "worker_id": String, "started_at": Instant }
TTL: 60 seconds

After acquiring the lock, the service downloads the full repository archive from the Rails internal API. The request passes include_lfs_blobs=false, so Gitaly does not run its LFS smudge filter. The archive then carries each LFS file's pointer text instead of its object contents. Nothing in the graph is built from LFS content. The extractor charges its total-bytes cap before filtering. So resolving those pointers would spend object-storage reads and cap budget on data that is then discarded. Each run extracts into its own unique directory, keyed by a per-run id, not just project + branch. So two workers racing the same repository never share an extraction tree and cannot clobber each other's files. That race happens when the ack deadline lapses mid-unpack and the task is redelivered before the first worker finishes. During archive extraction, the Gitaly archive root directory (<slug>-<ref>/) is stripped. This makes indexed paths repo-relative, so they match the paths used by content resolution's list_blobs revisions. That run's extraction directory is owned by the cached repository handle. So it is removed from disk as soon as the handle drops. This happens whether indexing completes, errors, or the whole job is dropped mid-run on the wall-clock timeout. A hard pod crash is caught by the startup purge that clears the cache root on boot. This prevents unbounded storage growth across indexer pods.

The extractor records a repository file inventory from archive metadata before it applies byte-level filtering. That inventory drives File and Directory node creation, so files do not need to be unpacked or parsed to appear in the graph. After inventory recording, extraction filters each regular file in two phases. The header phase uses an exclusion denylist (code_graph::v2::config::is_excluded_from_indexing) and the configured per-file size ceiling. It drops obvious binary/media/archive/document/datastore blobs or oversized blobs without reading any content. Files that survive the header phase are sniffed. The first 8000 bytes are inspected, and the file is dropped as binary when a NUL byte is present (code_graph::v2::config::looks_binary). This mirrors git's heuristic, with a UTF BOM rescue so BOM-marked UTF-16/32 text is kept. This catches binary blobs that fall under the size ceiling and carry no telltale extension. The content phase also recognizes Git LFS pointers, matching Gitaly's git.IsLFSPointer. Pointer text is valid UTF-8 and would otherwise be classified by extension and parsed as source. The size the archive reports for an LFS file is the pointer's size. So an LFS file's size_bytes on its File node is the pointer size, not the size of the object it stands for.

Source files, manifests, lockfiles, dotfiles, unknown extensions, and resolver inputs are intentionally allowed through unless they exceed the size ceiling or look binary. Dropped files remain in the inventory and still appear as File nodes; they are simply never written to disk and so are never parsed. Symlinks, hardlinks, and directories bypass the byte filter. Symlinks cost negligible disk and may legitimately point at parsable files. Directories are created lazily as files are unpacked.

The reason and byte volume of skipped entries are exposed via the gkg.indexer.code.archive.entries.skipped and gkg.indexer.code.archive.bytes.skipped counters. Operators can then quantify the disk savings per indexing run.

Transform (call graph construction)

Incremental engine behind a feature flag

The code_graph_incremental feature flag selects VFS archive loading in RepositoryResolver::resolve. It defaults to off and supports a root-namespace allowlist. The enabled path uses the archive source and a separate content policy, without creating an extraction tree. Filtered files remain cataloged, but their content is unavailable. The policy retains source and resolver inputs. The incremental engine groups source files by language family and reads their bytes and manifests from the VFS.

The code pipeline settings source_memory_budget_bytes and source_compress_spill control retained content. The memory budget is small, so most repositories stay in memory and a large one spills the rest to one temporary file. A job that waits for a lane holds at most that much source. Zero spills all retained content. Spill compression defaults to off. The existing file-size and total-byte limits also apply, with total bytes charged before filtering.

Each archive starts a fresh incremental-engine run; the server does not persist engine snapshots between jobs. An engine error that cannot succeed on retry dead-letters the job at once, as a fatal legacy pipeline error does. Examples are a rules load error and an export schema mismatch. A cancelled run and a failed insert stay retryable. File and directory rows are emitted once across language families, including files rejected by the content policy. The server reuses the embedded export mapping with ontology-derived output schemas. The envelope supplies typed values for extra schema columns and overrides header constants such as traversal path. The engine validates mapped columns against the ontology and output schema. Required envelope values must be present. The server adds branch relationships and endpoint tags before it inserts the batches. Both engines share locking, heartbeats, stale-data cleanup, and checkpoint handling. Write failures and cancelled runs do not advance the checkpoint. Tree-sitter polls cancellation and the per-file parse deadline while parsing.

Parser architecture

The code-graph crate contains the v2 pipeline stack under src/v2/. Code indexing uses several parser and analysis backends:

JavaScript and TypeScript in the v2 custom JS pipeline use OXC for parsing and semantic analysis. The same pipeline uses parser-side SSA for local value flow and member-call resolution. It treats JSX/TSX component opening elements as call sites while ignoring intrinsic tags. It uses oxc_resolver for cross-file module resolution. It honors tsconfig.json and jsconfig.json path mappings. It statically evaluates explicit webpack alias modules and their local require() dependencies. It skips minified files. It keeps ESM import resolution separate from CommonJS require() resolution, so package export conditions are evaluated in the correct mode.
Vue sources are handled by extracting script blocks into virtual JavaScript or TypeScript sources and routing them through the same OXC-based JS pipeline.
File-backed JS asset imports such as GraphQL, GQL, and JSON are indexed as module-like files. Each gets a synthetic primary export that resolves back to the file node. This keeps module resolution accurate for frontend repositories without pretending those assets contain parsed code definitions.
Webpack alias evaluation is deliberately partial and bounded. It only evaluates explicit local config modules. It enforces file-count, byte, statement, and recursion budgets. It treats process.env as an empty object. It allows filesystem probes such as fs.existsSync() only for repo-contained paths that remain under the checkout root after normalization.
Rust uses a rust-analyzer-backed custom v2 pipeline with a shell-free synthetic repo-local Cargo workspace model. The loader stays inside the checked-out tree. It attaches a pinned embedded Rust 1.95.0 sysroot project plus baked server-side cfg/target data. It disables proc macros and build-script execution. It layers parser-time SSA over rust-analyzer for local callable flow. That flow covers aliases, rebindings, destructuring, tuple/record field slots, and branch joins. rust-analyzer resolves callable semantics for functions, methods, macros, operators, ?, and await. rust-analyzer keeps its type values in process-global interners that only an explicit sweep frees. So a Rust job holds a process-wide read lock while it owns a rust-analyzer database. The job that finishes with no other Rust job in that phase runs the sweep. If enough Rust jobs pass without one, the next job waits a bounded time for exclusive access before it starts. It then proceeds either way.
Python, Kotlin, Java, C#, Go, Ruby, C, C++, PHP, Bash, Elixir, Swift, Lua, HCL, and YAML use tree-sitter grammars. The declarative DSL in crates/code-graph/src/v2/langs/generic/ drives them. Contributors adding a new language should follow docs/dev/adding-a-language.md. Lua indexes top-level, local, and global function declarations as Function or Method definitions. It extracts require() calls as runtime imports. setmetatable-based OOP is out of scope for v1.

Automatic v2 language dispatch is extension-based. The JavaScript pipeline owns .js, .jsx, .mjs, .cjs, .vue, .graphql, .gql, and .json. The TypeScript pipeline owns .ts, .tsx, .mts, and .cts. The Rust pipeline owns .rs. Swift owns .swift. Lua owns .lua. Ruby, JavaScript/TypeScript, Python, Kotlin, Java, PHP, Elixir, Swift, and Lua support full reference extraction. PHP resolves $this/self/parent/static member and static calls, trait/interface inheritance via Extends, and use imports. Swift resolves self/super member calls. Lua resolves dot and colon method calls with receiver tracking. Rust emits CALLS edges from rust-analyzer semantic resolution and local SSA flow; it does not currently materialize arbitrary non-call reference edges. C# currently supports definitions and imports only. Bash/Shell supports function definitions and source/. sourced-script imports only; cross-file call resolution, variable bindings, and aliases are out of scope. Swift v1 indexes classes, structs, enums, protocols, functions, constructors, and enum entries; extension-scope reattachment is a planned follow-up. YAML (.yml, .yaml) indexes anchors and aliases everywhere. Mapping keys become definitions and imports only where a document-type config under crates/code-graph/src/v2/langs/generic/yaml/document_types/ claims them. Those config types are GitLab CI, ArgoCD, Helm charts and values, and Docker Compose. Unclaimed YAML and files above the YAML size gate keep their File node only. No cross-file resolution is performed.

For each file, the parser extracts three categories of information:

Definitions such as classes, modules, methods, functions, constants, and interfaces. Each carries a fully qualified name (FQN), source range, and language-specific type.
Imported symbols with their import path, identifier, optional alias, and scope.
References including call sites and property accesses. A reference can be resolved to a single target, ambiguous across multiple candidates, or unresolved.

For JavaScript and TypeScript, phase 1 also populates the normal v2 CodeGraph and a JS-local module index together. Each source file synthesizes a top-level Module definition keyed by the repository-relative file path, plus export-member definitions. These let several constructs reuse the same nested and member resolution machinery as other v2 definitions. The constructs are namespace imports, primary exports, named exports, star re-exports, and module-level cross-file navigation. They do this without exposing a magic synthetic prefix as the user-facing identity. A second OXC-driven pass records invocation sites, including React and Next.js JSX/TSX component usages. It feeds local bindings through the shared SSA engine. It resolves intrafile targets through the generic v2 FileResolver. It leaves JS-specific cross-file import and module resolution in the custom JS resolver layer. An imported call sometimes cannot resolve to a repository-local definition. Then the graph preserves the call as a Definition to ImportedSymbol CALLS edge, instead of dropping the call site.

Orbit Local rebuilds its shared DuckDB graph when the code-index revision changes, even if repository commits have not changed. Repositories are re-indexed as used; Orbit Remote schema versions are unaffected.

The incremental engine stores trees as Tree<Mutable<Node>> or Tree<Compact<Node>>. Storage owns its records and exposes node reads and updates through an associated node type. Generic cursor navigation works with any payload; code-specific helpers specialize on Node. Mutable uses indextree for structural rewrites. Compact stores node payloads and five 32-bit links in a contiguous vector. Both use the same cursor operations. Parsing keeps source-backed text as byte spans until a rule needs a symbol. Canonicalization interns surviving text, releases the source, and builds compact storage directly from live nodes in preorder, before linking assigns edges. The compact vector reserves the live-node count rather than growing during conversion. The consuming From conversions preserve node IDs in both directions, including removed slots. compact_and_remap instead removes holes and renumbers nodes; use it before linking. Compact storage supports symbol and tag updates in place. Display uses the shared matcher to apply tag-only rules directly to compact trees. Rules that append or replace nodes convert one file at a time to mutable storage and back. Incremental indexing builds new trees for changed files and retains compact trees for unchanged files. Snapshot loading builds compact storage directly.

The incremental engine distinguishes an undefined name from a local value whose target is unknown. Unknown locals block fallback to same-named imports or functions. Rust identifier initializers use ordinary bindings. Unit struct declarations have an anonymous binding whose initializer constructs the struct. The linker follows that declaration rather than capitalization. Imported bindings use the existing type-flow path; unresolved references do not become calls. Direct Rust tuple destructuring reads the source bindings before assigning the destination bindings, including positions discarded with _. Rust pattern rules lower tuple and record components to ordinary declarations with field reads. Nested patterns repeat the same rewrite. A temporary binding holds the source before destination names become visible. Field initializers also use temporaries so later expressions cannot change earlier captured values. The tree_node_id transform supplies unique names within the rewritten file; these temporary names are internal and do not become graph definitions. Record fields and tuple-struct positions use separate SSA bindings within each function. Copies retain the current field values; whole-value replacement clears old fields. Calls through fields or loop joins retain their reaching values until the linker seals the loop back-edges.

Language YAML groups external framework providers under config.frameworks. Each group has a unique name and a providers list using the same schema as config.stdlib. Both lists compile into the existing provider lookup tables. Ruby declares Rails library roots with runtime precedence to prevent unrelated repository paths from capturing framework imports. Explicit aliases and relative imports retain their existing behavior. Framework methods are not implicit globals; application methods with names such as find, create, and render still resolve. Framework groups are configured by the language rules, not detected from manifests.

Ruby rules rewrite send, public_send, and __send__ when the first argument is a literal symbol or a string without interpolation. The resulting call keeps its receiver, remaining arguments, and block. Dynamic names keep the original dispatch call. These rules assume the standard Ruby dispatch methods; they do not detect custom overrides of those methods.

Language rules preserve lexical blocks as __scope nodes. The linker maps names to declaration identities, which also identify SSA variables and field slots. The linker allocates these keys per file; they do not enter the shared symbol interner or snapshots. Class values retain declaration node IDs through joins. Leaving a block removes its name mappings without restoring values. Assignments to outer bindings therefore survive block exit.

Incremental snapshots store live tree nodes in traversal order, with their arena IDs and parent IDs. Loading preserves those IDs and sibling order, including when nodes were added or removed after linking. Edges, tags, and cached resolver locations therefore keep their targets. Removed slots have no serialized node payload. Snapshot version 15 requires a fresh index for older saved graphs. Snapshots also retain original import paths so resolved paths cannot change standard-library classification on reindex. Language YAML distinguishes builtin callable names, external module roots, and shadowable standard-library roots. Loading rebuilds the file index from saved trees and manifests before edits apply.

Import discovery scans retained files only when the file index gains keys. Existing imports are checked separately for changed targets. Dirty files and redirected imports seed one reverse-dependency traversal with a shared visited set. Each affected file resolves once, including when dependency paths overlap or cycle.

An empty __declaration marker introduces a binding after its initializer. A named marker selects a matching scope label. The linker registers these names before walking that scope, so reads before assignment cannot fall back to outer names. TypeScript rules use block labels for let and const, and function labels for var. Python assignment rules use function labels. Rust declarations take effect in source order. Scope labels are YAML symbols; the engine does not branch on language names.

A text label such as (__rhs @$V) holds only the text of the node it names, and the subtree of V is dropped. A rule writes (__rhs @$V $V) to place the node under its label as well. A call, lambda or array inside that expression then survives canonicalization and is walked for edges. The rewriter records that the text and the child come from the same capture. Canonicalization marks a node it hoists out of a node it drops as lifted. A lifted node under such a label is walked, but it is never the value of the label. So x = f() types x by the call, and x = a + f() leaves x opaque. A language rule that replaces a transparent wrapper by its call, as Rust does for f()?, f().await, &f() and (f()), makes the call the value.

A __qualifier marker on the object of a member names a module or type, as in ptr::write. The linker resolves that object through declared items and imports only. A parameter or local with the same name does not shadow it. The same marker on a binding, as on a bounded type parameter, adds that binding to those items. A __qualifier binding declares that item only. The linker does not walk its right-hand side, so a language rule must not put a call under such a binding.

A __ssa_loop has one header shape. The loop variable is a __binding with a __declaration that the loop rule places directly under the loop. The children before it are the header, evaluated once before the loop. A bare __declaration scopes the loop variable to the loop; a labelled one binds it where the label says. A loop without a declared variable has no header, and all of its children run in the loop block.

C and C++ compound statements, Java and C# blocks, and Lua blocks retain lexical boundaries. Their local declarations introduce bindings independently of assignments to outer names. JavaScript-family iteration rules retain the source expression and bind the loop variable in its block or function scope. Rust blocks use the existing hoisting tag to register local items before walking statements.

An __alias on a named declaration selects an enclosing scope with that label. The linker registers these aliases before local declarations. Python global and PHP global select the module; Python nonlocal selects an outer function. PHP rules register variable reads in their function scope so undeclared locals cannot inherit file-level values. Bash variable bindings use their $ prefix to keep them separate from literal command names.

Inventory-driven indexing pipeline

The indexing pipeline uses a repository inventory as the single file list. Pipeline callers must provide the inventory; the parser grouping, structural graph, and stats all derive from that same list. The stages are:

Repository inventory supplies the complete set of file entries from the Git tree. Server indexing reads this from Gitaly archive metadata for the indexed revision before extraction filters run; local CLI indexing reads present, non-ignored files from Gitalisk.
Materialized-file checks use the inventory paths to find files that were unpacked or are present on disk and can be read by parsers and resolvers. No second directory walk runs.
Extension filtering runs parsable_language over materialized files, then groups parseable files by language.
Structural graph emission creates Directory, File, and containment edges from the repository inventory. Non-parsable files use language = "unknown" and do not produce definitions or imports.
Async file reads load parseable file contents with bounded IO concurrency.
CPU-bound parsing runs on a thread pool with a semaphore to cap parallelism based on available cores.
Analysis groups parsed results by language and builds definition, import, call, and inheritance relationships that attach to the structural file nodes.

IO reads and CPU-bound parsing are bounded independently. File reads use a concurrency limit proportional to the worker thread count. Parsing uses a semaphore sized to the number of available CPU cores. This separation prevents IO-heavy repositories from starving the parser and vice versa. The pipeline outputs a graph structure consumed by the load phase. The defaults scale with the number of available cores.

Graph data model

After parsing, the analysis phase groups results by language and builds a graph containing:

Node type	Description
Directory	Directory in the repository tree
File	Repository file with path, language, extension. Non-parsable files use `unknown` for language.
Definition	Code definition with FQN, type, source range, file path
Imported symbol	Import statement with path, type, identifier, source range

Relationship types

The graph captures fine-grained relationships across several categories:

Containment tracks which directories contain other directories or files.
Definitions link files to the code entities they define, and capture the nesting hierarchy (e.g., a class containing methods, a module containing classes).
Imports connect files and symbols to the definitions or files they import.
References represent call sites, property accesses, and ambiguous calls where the target cannot be resolved to a single definition. Unresolved imported calls are retained as calls to imported symbols when the parser can associate the call with an import.

Internally the graph uses roughly 50 fine-grained relationship types spread across these categories. For example, it distinguishes a method call from a property access, or a re-export from a direct import. During the load phase, these fine-grained types are collapsed into five high-level ontology labels: CONTAINS, DEFINES, IMPORTS, CALLS, and EXTENDS. This simplification keeps the query layer consistent while the internal graph retains full detail for analysis.

Load

ETL engine and module system

The indexer provides a general-purpose ETL engine shared by both code and SDLC indexing. Each indexing pipeline is a module plugged into this engine.

The engine subscribes to NATS topics, routes messages to the appropriate module's handler, and manages acknowledgments. A global worker pool with optional per-module concurrency limits prevents any single module from starving others.

Pluggable storage

Handlers don't write to ClickHouse directly. They receive a trait-based storage abstraction and create table-specific writers on demand. The abstraction has two implementations. One is a production writer that serializes Arrow record batches and streams them to ClickHouse. The other is a mock that captures writes in memory for test assertions. Because handlers only depend on the trait, they are database-independent and can be tested without any external infrastructure.

Code indexing task flow

The code indexing handler subscribes to CodeIndexingTaskRequest messages from the internal GKG_INDEXER NATS stream. On each task the handler:

Checks the checkpoint to skip already-indexed commits
Acquires a distributed lock via NATS KV to prevent concurrent indexing of the same project and branch
Downloads the full repository archive via the Rails internal API
Runs the indexing pipeline (parse, convert to Arrow, insert into ClickHouse)
Deletes the downloaded repository from disk
Waits for the inserts, cleans up stale data, updates the checkpoint, and releases the lock

Storage in ClickHouse

The graph is converted to Apache Arrow record batches and written to ClickHouse. Node batches use the branch, directory, file, definition, and imported symbol tables. Edge batches route by relationship kind and both endpoint kinds. The resolved ontology variant selects gl_code_edge for source-code relationships, including the Project and Branch bridge. Every row carries base columns for the namespace hierarchy path, project ID, branch, and a version timestamp used for stale data cleanup.

Record batches are serialized to Arrow IPC format and streamed to ClickHouse.

Writes

Each job inserts its own rows. The parse emits one Arrow batch per table. The job cuts a batch into slices of at most write_slice_rows (default 1M). It starts one insert per slice while the parse continues. At most write_max_concurrent (default 8) inserts of one job are in flight. The parse pauses until a slot frees, so a job holds at most that many slices in memory. The default is above the number of code tables. So a small job sends every table's insert at once and waits one server flush, not two. Every insert sets async_insert=1 and wait_for_async_insert=1. ClickHouse pools small inserts from all jobs and all pods per table into one part per flush. It answers each insert only after its rows are in a part. The flush fires at the latest on the server's async_insert_busy_timeout_max_ms, so a small job waits about that long for its rows. The operator can raise it in the code pool's graph.insert_settings to pool more inserts per part at the cost of a longer wait. A cluster with insert_quorum cannot use async inserts; there every job writes one part per table.

Each insert is retried with bounded backoff (2 s then 4 s) before it fails. So a transient ClickHouse blip does not cost the project a full re-index. The retry runs through the shared engine retry harness (engine/retry.rs), the same vocabulary the SDLC datalake read, the schema-readiness wait, and the job-timeout policy use.

Archive downloads also share a budget per Gitaly storage across all indexer pods. fetch_concurrency is per pod, and KEDA adds pods as the backlog grows. Each download takes one of archive_storage_budget (default 4) slot keys for its storage in the indexing_locks KV bucket, named archive.<base64url storage>.<slot>. A slot is the same renewable lock the project lock uses. A pod takes it with create-as-CAS and keeps it with a heartbeat every third of archive_lease_ttl_secs (default 30). The heartbeat runs beside the download, so a slow renewal never pauses it. A crashed pod's slot is taken over once its embedded expiry passes. Only the download and its extraction hold a slot, not the retry backoff or the parse.

When a renewal reports the slot lost, or renewals fail until the lease expires, the download future is dropped. The legacy extractor reads the stream on a blocking thread that outlives the future. So it also carries a cancel flag, set on drop, that fails its next read. Together these stop the Gitaly stream, and a second pod cannot overlap it. The attempt then retries through the normal archive retry.

The storage comes from repository_storage on the task row when GitLab fills it, so those push tasks make no lookup. Otherwise it comes from the /info response. Backfill tasks reuse the lookup they already make to resolve the default branch. Push tasks without the column make one lookup, which the one-hour project-info cache usually serves. Either way a task makes at most one lookup, whatever the cache holds. After a storage move the key follows the value on the newest task row, or the cache for tasks without one. GitLab versions whose /info omits the field, and failed lookups, use a separate tenant-wide budget, archive_tenant_budget, under the keys archive.tenant.<slot>. Its default is 8. That is above the per-storage default, so a tenant with a few storages is not starved before the Rails field ships. It is far below today's worst case of fetch_concurrency × pods (about 40), so an Orbit upgrade alone already bounds the load. During a mixed rollout the two key spaces are independent. A storage can then see its storage budget plus the tenant budget at once, because some projects resolve a storage and others do not.

A job queues for a slot for at most archive_lease_max_wait_secs (default 30), while it holds one of the pod's fetch slots. A longer wait would let one busy storage pin every fetch slot of a pod. Projects on idle storages would starve. After the wait the job goes back to the queue without spending a checkpoint attempt, and it does not count as a fetch error. When the project lock is lost it also goes back, but the attempt stays counted, the same as a lost lock during the lane wait. It does spend a JetStream delivery, though. NATS counts every redelivery toward max_deliver (5 by default), and each NAK waits retry_interval_secs (60) first. A storage that stays saturated for several minutes therefore drops the task from the stream. The project then waits for the dispatcher cadence, up to the one-hour backfill retry window. Under sustained saturation the lease behaves as dispatcher-paced retry, not as a queue. Each cycle also costs two checkpoint writes, a project-lock acquire, and up to 60 polls of every slot key. A budget of 0 turns the lease off for that key space. The key is the storage name. Under Praefect it covers a set of Gitaly nodes, so the budget is stricter than needed.

Concurrency is split into two lanes by parsable-file count. The count includes only files that the content filter keeps and that have a supported language. So a repository of images or lockfiles is not misclassified as large. Both engines use the same count, so a repository takes the same lane under either engine. A repository with more than small_repo_max_files such files takes the big lane. Unless they are set, the lane sizes come from the worker budget, which the CPU count and the container memory limit bound. A part of the indexing share, at least one, is the reserved lane (big_indexing_slots). The remainder, at least one, is the wide lane (small_indexing_slots) that serves the small-repo tail. The reserved lane means a flood of small repositories cannot starve monorepos. A job that finds its lane busy waits for up to the time left on its project lock lease. The heartbeat renews the lock and refreshes the NATS delivery during the wait. If no lane frees in that time, the job goes back to the queue without spending an attempt.

A project is checkpointed only after its rows are durable. When the parse ends, the job deletes its extraction tree, releases its intake permit, and waits for its inserts. If any insert fails after its retries, the job fails without a checkpoint. NATS redelivers it, and the backfill sweep dispatches it again one hour after its last attempt (re-indexing is idempotent). A crash before the rows land has the same effect. So does a job that hits its wall-clock timeout; its run future is dropped, which aborts its inserts, and the project is re-indexed. Once the rows land, the job tombstones the prior version's stale rows, then writes the checkpoint. The handler holds the project lock until the checkpoint is saved, and acks the task only then. This keeps the checkpoint the single durable record, with no watermark and no durability handshake. The handler heartbeat renews the lock, and a lost lock cancels the job.

Parts that span many namespaces are safe to query. The edge sort key is (traversal_path, relationship_kind, …), so ClickHouse still prunes by primary key within a mixed part.

Checkpoint tracking

The code_indexing_checkpoint table records the last successfully indexed point per namespace, project, and branch (keyed on traversal_path, project_id, branch). The code indexing task handler checks it to skip already-indexed commits. Before a first index, the handler counts an attempt. A completed index resets the count to 0.

Projects whose archive endpoint returns 404 or a plain 500 are checkpointed with no commit and treated as terminal "indexed empty". Workhorse maps Gitaly's not-found, invalid-argument and failed-precondition to 404. Backfill tasks resolve the branch through project_info first, so the project exists and a 404 means it has no repository or no refs. That is why 404 stays terminal for tasks without a commit SHA. A plain 500 stays terminal too. Rails currently answers 500 for a missing repository (NoRepository). This prevents retries and DLQ churn for projects with no content. Once the project is populated, subsequent siphon tasks arrive with a larger task_id than the stored checkpoint and are re-processed normally. The gkg.indexer.code.repository.empty counter (labelled reason=not_found|server_error|empty_archive|repository_too_large) tracks how often this short-circuit fires.

A 429, 502, 503 or 504 from the archive endpoint is transient overload, not an empty repository. Workhorse answers 500 for Gitaly's ResourceExhausted today, which is treated like a missing repository; a 429 is handled as overload. Overload is never checkpointed, for backfill and CDC tasks alike. The indexer skips its local retries, nacks the task, and increments gkg.indexer.code.errors with stage=repository_fetch_overload. Redeliveries are retry_interval_secs (60 s) apart, so counting each one would use up MAX_CODE_ATTEMPTS within minutes. The handler refunds the checkpoint attempt on every delivery except the first, so one dispatch costs at most one attempt whatever max_attempts is. The backfill dispatcher re-sends the project after CODE_RETRY_AFTER (1 h). After MAX_CODE_ATTEMPTS counted attempts it stops, leaving an unindexed checkpoint row (indexed_at null) that is visible and repairable. The project is never marked indexed-empty, so a later push event still indexes it. The archive lease is released before the nack, so an overloaded task never holds a slot while it waits for redelivery.

Tasks with no branch field resolve the default branch via GET /api/v4/internal/orbit/project/:id/info. That endpoint sometimes returns 404, when the project is deleted in Rails but still referenced by the dispatcher's datalake view. Then the task is acked with the same empty_repository{reason=not_found} counter, and no checkpoint is stored. The branch is unknown, so there is no key to write under. The ack avoids DLQ churn; the dispatcher stopping emission for deleted projects is tracked separately.

A job that exceeds its hard wall-clock budget (job_timeout_secs, default 1500s) gets one retry. The retry absorbs a transient slowdown, such as a busy pod or a slow Gitaly fetch. A job that times out a second time is treated as structurally stuck and dead-lettered. This avoids burning the full delivery budget on a repo that will almost certainly time out again, and wasting an indexing slot each time. (A transient write failure is different: that is retried in the writer with backoff, above, and does not dead-letter the job.) Abandoning the job also stops the indexing work it started, so the pod releases the CPU and memory rather than holding them until it restarts. The stop is not instantaneous. Work already in flight on one file finishes first. A Rust job waiting for the interner sweep described above finishes that wait first.

Flow visual representation

  Push events:                          Namespace enabled:
  Rails (p_knowledge_graph_code_        Rails (knowledge_graph_enabled_
         indexing_tasks)                       namespaces)
        |                                      |
        v                                      v
  Siphon CDC → NATS JetStream (Siphon stream, two subjects)
        |                                      |
        v                                      v
  SiphonCodeIndexingTask               NamespaceCodeBackfill
  Dispatcher                           Dispatcher
        |- Batch-pull messages                 |- Batch-pull messages
        |- Decode protobuf                     |- Resolve namespace traversal path
        \- Publish per task                    \- Publish per project in namespace
        |                                      |
        +------------------+-------------------+
                           |
                           v
                 NATS JetStream (GKG_INDEXER stream)
                           |
                           v
                 CodeIndexingTaskHandler (Indexer mode)
                           |
                           |- 1. Deserialize CodeIndexingTaskRequest
                           |- 2. Resolve default branch from Rails (if not provided)
                           |- 3. Check checkpoint (skip already-indexed commits)
                           |- 4. Acquire distributed lock via NATS KV
                           |- 5. Download full repository archive
                           |- 6. Run indexing pipeline
                           |       |- Repository file inventory
                           |       |- Parser file discovery
                           |       |- Async file reads
                           |       |- CPU-bound parsing (bounded parallelism)
                           |       |- Analysis phase -> graph
                           |       |- Convert graph to Arrow record batches
                           |       \- Insert into ClickHouse (6 tables)
                           |- 7. Delete downloaded repository from disk
                           |- 8. Wait for its inserts, tombstone stale rows,
                           |      then save the checkpoint
                           \- 9. Release lock and ack

The handler acks only after the project's rows are written and its checkpoint is saved. Nothing in the indexer delays the write: the wait ends when ClickHouse answers the job's own inserts. An insert that hangs holds the job, its lock, and its NATS delivery. The heartbeat progress acks keep the delivery under nats.ack_wait_secs and renew the lock during the wait. On shutdown the engine stops pulling messages and joins the in-flight jobs. A job that waits for its inserts still lands them and checkpoints before the process exits.

Write data flow

@startuml
skinparam shadowing false
skinparam ParticipantPadding 20

participant "code job" as J
database ClickHouse as CH

J -> J : parse; cut each table batch\ninto write_slice_rows slices
J -> CH : insert slice (async_insert,\nwait_for_async_insert)
note right of CH : pools inserts from all jobs and pods\nper table; one part per flush
J -> J : delete extraction tree,\nrelease intake permit
CH --> J : rows in a part (about 1 s)
J -> CH : tombstone stale rows (re-index only)
J -> CH : save checkpoint
J -> J : release lock, ack
@enduml

Differences from the original local tool

The original Knowledge Graph (at gitlab-org/rust/knowledge-graph) was a local desktop tool. Here are the main architectural differences in the current service:

Aspect	Original (local)	Current (service)
Graph database	lbug (embedded)	ClickHouse
Code access	Local filesystem	Rails internal API (full archive download, no disk cache)
Event trigger	Filesystem watcher (`watchexec`)	Siphon CDC → dispatcher → internal NATS stream
Storage format	Parquet -> lbug bulk import	Arrow IPC -> ClickHouse
Multi-tenancy	Single user, single repo	Namespace-scoped via `traversal_path`
Authorization	None (local tool)	Rails gRPC delegation
Parser crate	External `parser-core` dependency	In-tree `code-graph` crate, `src/v2/`
Graph builder	External `indexer` crate	In-tree `code-graph/src/v2/linker/`
Concurrency	Streaming model (Rayon + semaphore)	Same streaming model (preserved)

Indexing the active branches

The problem

The main branch is the most common branch to index, but a strategy for active branches is worth documenting. Orbit also includes a local version that customers can use to query code against their local repository at any version.

The core issue with indexing active branches is volume: billions of definitions and relationships for repositories the size of the GitLab monolith. The initial release focuses on main-branch indexing; branch-level and commit-level support are planned as follow-on work.

A future strategy

After the initial deployment, metrics and customer feedback will determine whether branch-level indexing is worth the storage and compute cost. The approach below outlines one viable path.

As stated above GitLab has the concept of a branch being "active" or "stale". An active branch is one that has been committed to within the last 3 months. A stale branch is one that has not been committed to in the last 3 months.

The data is large, and query distribution is uneven, since some branches are never going to be queried. So it's best we don't keep the data against the main branches in the same database. That would result in a lot of wasted storage and compute resources.

Ideally, we would re-use the same indexing strategy as the main branch. We can index the active branches by listening to code indexing tasks from NATS. But instead of loading the data into ClickHouse, we would store the data in cold storage (like S3 or GCS).

On request, we would load the data into ClickHouse from cold storage in materialized tables. This would then let us query the data in ClickHouse during the current session. It would then unload the data from ClickHouse after the session is complete (based on a variable TTL).

Flow visual representation

graph TB
    subgraph "Active Branch Processing"
        J[Push Event] --> K[NATS Event Stream]
        K --> L[Index Data]
        L --> M[Cold Storage<br/>S3/GCS]
    end

    subgraph "On-Demand Loading"
        N[User Request] --> O{Data in ClickHouse?}
        O -->|No| P[Load from Cold Storage]
        O -->|Yes| Q[Query Data]
        P --> R[Materialized<br/>in ClickHouse]
        R --> Q
        Q --> S[Session Complete]
        S --> T[Unload Branch Data After TTL]
        T --> M
    end

    style M fill:#2196F3
    style P fill:#2196F3

Cleaning up

Once the branch either becomes stale or is deleted, we should clean up the data in our cold storage. This would be done by a separate job that would run periodically and clean up the data based on the latest state of the branches.

Alternative approach

Consider an alternative approach if the time to first response is not critical. We would index the active branches, then index the stale branches on demand. This depends on the indexing speed on the servers. We would save the temporary data in ClickHouse, then dispose of it after the session is complete or at a later time. This would eliminate the need to manage cold storage and the associated costs.

Indexing the stale branches

Stale branches are in most cases branches that have been abandoned by the original author. They are not actively being worked. If we were to index them, we could follow the same strategy as the alternative approach described for active branches.

Zero-Downtime Schema Changes

Code Indexing is going to follow the same schema migration strategy as the main branch as described in Zero-Downtime Schema Changes.

How code querying works today

The Orbit team originally built dedicated MCP tools for code querying. Each tool wraps a focused workflow on top of the indexed call graph. Reference documentation lives at Knowledge Graph: tools.

The available tools:

list_projects enumerates indexed repositories for agent discovery.
search_codebase_definitions searches definition nodes by name, FQN, or partial match and returns signatures plus context.
get_definition resolves a usage line to its definition or import by following call graph edges.
get_references walks the graph in the other direction to list every reference to a definition, with contextual snippets.
read_definitions batches definition bodies so agents can retrieve implementations efficiently.
repo_map walks the directory nodes and summarizes contained definitions, using the graph to stay .gitignore-aware.
index_project triggers the repository indexer inside the MCP process for on-demand reindexing.

These tools use a shared query library and the same database connections used during indexing. Many also supplement database hits with filesystem reads to include code snippets based on byte offsets from the graph.

Important Note: We intend to replace the above tools, where it makes sense, with our Graph Query Engine technology. This would let agents and analytics traverse the graph using tools that will be shared with SDLC querying. Agents will never write or execute raw queries themselves. They can only interact with the graph through these exposed, parameterized tools, which enforce security and access controls.

FilesExpand file tree

code_indexing.md

Latest commit

History

code_indexing.md

File metadata and controls

Code indexing

Code Indexing ETL

What is a call graph?

Architecture overview

Components

Data storage

Some numbers

Use cases

Indexing the main branch

Extract

Dispatch

Namespace backfill dispatch

Handler

Transform (call graph construction)

Incremental engine behind a feature flag

Parser architecture

Inventory-driven indexing pipeline

Graph data model

Relationship types

Load

ETL engine and module system

Pluggable storage

Code indexing task flow

Storage in ClickHouse

Writes

Checkpoint tracking

Flow visual representation

Write data flow

Differences from the original local tool

Indexing the active branches

The problem

A future strategy

Flow visual representation

Cleaning up

Alternative approach

Indexing the stale branches

Zero-Downtime Schema Changes

How code querying works today