Skip to content

fix(pipeline): stable USAGE-edge ownership for directory-module languages (#787)#828

Closed
spde wants to merge 2 commits into
DeusData:mainfrom
spde:fix/787-stable-usage-ownership
Closed

fix(pipeline): stable USAGE-edge ownership for directory-module languages (#787)#828
spde wants to merge 2 commits into
DeusData:mainfrom
spde:fix/787-stable-usage-ownership

Conversation

@spde

@spde spde commented Jul 3, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes #787: since #667, fresh indexes of the same Java tree attached USAGE edges to different (wrong) source files on every run.

Reproduce-first, per the triage note on the issue: the new tests/repro/repro_issue787.c asserts stable USAGE source ownership across repeated indexes — exact distinct-source set, 5 sequential re-indexes, and a >50-file variant that forces the parallel merge path. All three cases fail on unfixed HEAD (the parallel case surfaces a random filler file as a USAGE source, which is the race made visible).

Root cause

#667's directory-based Java/Go module QNs collide with the package's Folder node QN (or Project for root files), while every file still emits a Module def with that QN. Three code paths then corrupted source attribution on the shared node:

  1. cbm_gbuf_upsert_node (graph_buffer.c) — the QA: reproduce-first bug suite + LSP/grammar extraction fixes (5-platform board green) #667 guard kept the Folder/Project label on a colliding Module def but still clobbered name/file_path/line-range, so the shared node's file_path became whichever same-package file was processed last.
  2. merge_update_existing (graph_buffer.c) — the worker-gbuf merge had no guard at all: unconditional "src wins", including relabelling the Folder node to Module. Worker merge order varies per run — this is the nondeterminism.
  3. The three source-node finders (find_enclosing_node / find_source_node / calls_find_source) look up class-level references' enclosing QN directly, landing on that one shared directory node — so all same-package class-level USAGE edges (e.g. @Autowired fields) collapsed onto a single source with an arbitrary file_path.

Fix

  • Both graph-buffer paths now skip the update entirely when a Module def collides with a Folder/Project node (the merge path keeps the ID remap so worker edges still redirect to the canonical node).
  • New cbm_pipeline_node_is_dir_container() predicate; the three finders treat a directory-container hit as a miss and fall through to the per-file File node.

Verification

  • New repro: 3/3 green, stable across 10 consecutive runs (was 0/3 on HEAD).
  • Full gating suite: 5748 passed, 0 failed.
  • spring-petclinic, 3 independent fresh indexes: OwnerRepository inbound USAGE sources = exactly the 7 ground-truth files, identical every run (HEAD gave 4–6 with run-varying false positives).
  • A private Spring Boot service (~1.3k Java files, 142 beans): grep-verified bean-user recall back to 757/757 = 100% (HEAD: ~16%).
  • Incremental reindexing behavior from fix(pipeline): preserve inbound cross-file edges on incremental reindex #528 preserved (inbound USAGE survives an edit+reindex cycle).

Two adjacent direct enclosing-QN lookups were left out of scope (env-access CONFIGURES sources and resolve_file_throws); noted here rather than folded in, happy to follow up if you want them handled the same way.

Tommy Corbett added 2 commits July 4, 2026 00:31
…rce misattribution across same-package files

Three cases in the reproduce-first suite: exact distinct-source-file set for
a class-level type reference (sequential path), stability of that set across
5 independent indexes, and the same assertions on a >50-file fixture that
takes the parallel worker/merge path (where the collapse varies run to run).
Set semantics mirror the spring-petclinic oracle: OwnerRepository must have
exactly 7 distinct USAGE user files.
…ages (DeusData#787)

Since the directory-based Java/Go module QNs (DeusData#667), every file in a package
shares its module QN with the package's Folder node (or the Project node for
root files). That collision corrupted USAGE/CALLS source attribution in three
places:

- cbm_gbuf_upsert_node kept the Folder/Project label on a colliding per-file
  Module def but still clobbered name/file_path/range in place, so the shared
  node's file_path became whichever file was processed last.
- merge_update_existing (parallel worker-gbuf merge) applied unconditional
  "src wins", even relabelling the Folder node to Module — and worker merge
  order varies per run, which is the reported run-to-run nondeterminism.
- The source-node finders (pass_usages/pass_parallel/pass_calls) looked up a
  class-level reference's enclosing QN (== the module/directory QN) directly,
  landing on that shared directory node: all same-package class-level
  references collapsed onto ONE source with an arbitrary file_path.

Fix: the structural directory node is never touched by a colliding Module def
(upsert AND merge paths, same guard), and the finders treat a Folder/Project
hit as a miss, falling back to the per-file File node — so each file's
class-level usages attribute to that file, deterministically.

spring-petclinic: OwnerRepository USAGE users are again exactly the 7
grep-verified files on every fresh index (HEAD returned 4-6 with bogus,
run-varying entries); 142-bean sweep on a Spring repo stays at 757/757
user-file recall; incremental re-index preserves inbound USAGE counts.
@spde spde requested a review from DeusData as a code owner July 3, 2026 22:35
@DeusData DeusData added bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Jul 4, 2026
@DeusData

DeusData commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Thanks for the stable USAGE-edge ownership fix for #787. Triage: high-priority parsing correctness.

Review will focus on the reproduce-first fixture across repeated indexes and the graph-buffer merge/upsert changes. The important invariant is deterministic source ownership for directory-module languages without regressing the #667 package/module behavior.

timothybrush pushed a commit to timothybrush/codebase-memory-mcp that referenced this pull request Jul 4, 2026
Since DeusData#667 every file in a Java/Go package shares a directory-based module
QN that collides with the pipeline's Folder/Project node. Class-level
references resolved their enclosing scope to that shared node, whose
file_path was clobbered by each file's always-emitted Module def — last
writer wins (sequential upsert) or last-merged worker wins (parallel merge)
— so USAGE-edge sources conflated across same-package files and varied run
to run.

Three coordinated changes:
- cbm_gbuf_upsert_node: a Module def colliding with a Project/Folder node
  no longer updates ANY field (the DeusData#667 guard only protected the label).
- merge_update_existing: same guard on the parallel worker-gbuf merge path;
  the ID remap stays outside the guard so worker edges still redirect onto
  the canonical node.
- Source-node finders (pass_usages, pass_parallel, pass_calls) treat a
  lookup landing on a Folder/Project container as a miss and fall back to
  the per-file File node (new cbm_pipeline_node_is_dir_container helper).

Adds tests/repro/repro_issue787.c: exact-source-set, 5-run stability, and
>50-file parallel-path guards (red on the unfixed code in both sequential
and parallel flavors).

Distilled from DeusData#828: the author's commits lacked DCO sign-off trailers and
could not be cherry-picked; content applied verbatim plus one clang-format
fix in graph_buffer.c.

Closes DeusData#787.

Co-authored-by: Tommy Corbett <spde89@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

DeusData commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Thank you — this was a precise, well-diagnosed fix. All three mechanisms you identified (the label-only guard leaving name/file_path/props clobbered, the unguarded merge path, and class-level refs landing on the shared directory node) were verified exactly as you described, and your 3-part reproduction (exact source set / 5-run stability / >50-file parallel) was model reproduce-first work. We carried it over the line as 896aa91 (PR #844) with you credited as co-author — the only changes were adding DCO sign-off (your commits lacked the trailer) and one clang-format fix. Closes #787. Closing in favor of the distill — thanks again for the excellent write-up!

@DeusData DeusData closed this Jul 4, 2026
DeusData pushed a commit that referenced this pull request Jul 9, 2026
… directory-module collisions

Follow-up to #787/#828, disclosed during #828's review: the same clobber
class also affects two call sites #828 deliberately left out of scope.

create_env_configures_for_file (pass_definitions.c) and resolve_file_throws
(pass_parallel.c) both resolved their edge source via a direct
cbm_gbuf_find_by_qn(enclosing_func_qn) lookup with no guard against the
lookup landing on the shared Folder/Project node for directory-module
languages (Java/Go) -- the exact #787 root cause, just unguarded at two
more sites. A class-level construct (no enclosing function) has its
enclosing_func_qn fall back to the MODULE QN, which for a directory-based
module collides with the package's Folder node, conflating every
same-package file's CONFIGURES/THROWS edges onto one arbitrary source.

- create_env_configures_for_file: added the same
  cbm_pipeline_node_is_dir_container guard already used by the three
  #828-fixed finders, falling back to the per-file File node on a
  directory-container hit.
- resolve_file_throws: was missing the `rel` parameter its sibling
  resolve_file_calls/usages/rw already carry, so it inlined a raw lookup
  instead of calling the shared, already-guarded find_source_node. Added
  `rel` and switched to find_source_node, matching its siblings exactly.

Regression guard: tests/repro/repro_issue842.c, reproduce-first per the
issue's request for "an #828-style multi-order red fixture":

- CONFIGURES reproduces on a small 5-file sequential fixture (Go, since
  Java's System.getenv detection is separately broken -- extract_env_key_from_call
  reads a "function" field Java's method_invocation node doesn't have; noted
  in the test, out of scope here).
- THROWS only reproduces past MIN_FILES_FOR_PARALLEL=50 (the worker-merge
  path) -- confirmed empirically that the small sequential fixture already
  attributes correctly even on unfixed HEAD, mirroring #787's own two-faced
  (sequential/parallel) bug. Padded with 60 filler classes, following
  repro_issue787.c's build_parallel_fixture precedent.

RED on unfixed HEAD (both tests): 0 passed, 2 failed. GREEN with this fix:
2 passed, 0 failed. Full scripts/test.sh (5958 cases) passes clean with
this change; scripts/lint.sh's two pre-existing failures (pass_lsp_cross.h
parameter-name mismatch, extract_defs.c cppcheck AST parse) are unrelated
and present identically on unfixed main.

Fixes #842

Signed-off-by: Dhruvy0804 <dhruvgarg553@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nondeterministic Java USAGE-edge misattribution since #667 — edges attach to wrong source files on every fresh index

2 participants