docs(#136): memory distillation pipeline first-run post-mortem

[alexosugo]

Summary

Adds docs/memory-pipeline-postmortem.md - the consolidated post-mortem of the first production
run of the memory distillation pipeline (scraper -> filter -> distiller -> open-review-pr), run
against merged medic/cht-core PRs and surfaced as the 8 open chore(memory): promotion PRs
(#120, #121, #122, #123, #129, #130, #131, #132).

~107 distilled memories were reviewed, spot-checked against
the live cht-core API.

Key finding

Distillation prose is good; the failures are in plumbing. One defect dominates:

PR-number-as-issue-number (distiller.ts:267). The scraper only parses body
Fixes/Closes/Resolves #N, but CHT encodes the issue in the PR title type(#N):, which is
never parsed - so the ?? pr.prNumber fallback fires and id/issueNumber/issueUrl name the
merge PR. ~60/107 drafts affected. The /issues/N -> /pull/N redirect masks it, so
validate-schema passed while shipping wrong data. This defeats the planned #135 de-dup consumer.

What's in the report

• Overall patterns + recurring failure classes across all 8 PRs
• Root-cause analysis mapped to specific prompt lines / truncation limits / schema fields
• Prompt + pipeline improvements, each mapped to the feedback it addresses
• An 8-item prioritized roadmap (by impact)
• A 3-layer evaluation strategy + golden-set plan
• A 20-scenario regression catalog (each row = a reviewer-caught defect as a permanent test)
• Documentation updates for the next iteration
• Per-PR appendix

Related

• Tracked by Memory distillation pipeline: first-run post-mortem findings & next-iteration improvements #136
• Addresses Eval foundation: golden dataset from memory pilot #104 (Eval foundation: golden dataset from memory pilot)
• Feeds Context analysis agent doesn't load memory-pipeline drafts (wrong path + schema) #135 (Context Analysis Agent consumer / de-dup-by-issue-id)

Test plan

• Docs-only change; no code affected. Review the report for accuracy against the source PRs.

[sugat009]

Comprehensive review of the post-mortem, every claim re-checked against the live GitHub API and source, unbiased. The core is solid and this is a valuable doc: the prose-vs-plumbing thesis, every source-code reference, all 15 PR->issue resolutions, and the backport/cross-domain analysis all hold up. A handful of inline corrections before it becomes the reference gating #135/#104: two factual errors (the resolution priority chain is inverted vs the shipped code; the 10623 hallucination example is wrong), and three staleness/labeling items (#129's shipped scope, the roadmap already living in #138, and the "60/107 corpus-wide" scope). All non-blocking, all inline.

[sugat009]

issue: The recommended resolution order here (body Fixes/Closes/Resolves -> title type(#N): -> closingIssuesReferences) is inverted relative to the shipped code. issue-linkage.ts (byte-identical on #129 and #138) orders closing-ref -> title -> body as explicit "descending authority", with closingIssuesReferences the most authoritative and linkedIssues[0] canonical. A closing-ref is GitHub's authoritative link, so it should win over an author-typed body keyword, not lose to it. In the divergence case (title #N, empty closing-refs, body Closes #M) this doc picks #M while the code correctly picks the authoritative signal. The same ordering flows into the Part 2 eval chain (L213-214) and the Part 3 title vs body precedence fixture (L254), which would then encode the wrong canonical issue. Suggest flipping to closingIssuesReferences -> title -> body throughout to match the implementation.

[alexosugo]

Good catch — verified against issue-linkage.ts on the #129/#138 branches (collectLinkedIssueRefs pushes closing-ref → title → body, "descending authority", first entry canonical). Flipped the chain to closingIssuesReferences > title > body throughout: section 4.1, the Part 2 unit chain, the Part 3 fixture (now "title outranks body", issueNumber=8026), and the Part 4 precedence line. Fixed in ba5b5ce.

[sugat009]

issue: This example does not hold up. PR 10623 does change webapp/src/ts/pipes/message.pipe.ts and webapp/src/ts/reducers/tasks.ts (plus reducers/global.ts and their specs) per gh api repos/medic/cht-core/pulls/10623/files, so the draft's file list is accurate, not hallucinated. Since 10623 is the flagship "hallucinated paths" case (also cited in the Part 3 fixture at L258 and the appendix at L399-400), it should be dropped or swapped for a real case. The other prose examples (Slack L341, "cf. Seed 4" L382) do check out, so the point survives, just not this instance.

[alexosugo]

Confirmed via gh api .../pulls/10623/files: message.pipe.ts and reducers/tasks.ts are both in the real PR. But the draft (10623-feat10577-…​.md) also lists services/user-contact.service.ts, which is not in the PR — so 10623 is still a valid hallucination, just misdescribed. Kept it as the exemplar and corrected the file facts at L118/L267/L409. Fixed in ba5b5ce.

[sugat009]

issue (stale): This understates what #129 already shipped. On the #129 branch: the nested 10399 -> 10182(PR) -> 10183 case is handled and regression-tested (relink-issues.spec.ts:96 asserts to === 10183); title parsing is wired into the forward path (scraper.ts:305 collectLinkedIssueRefs(prTitle, ...)); and the ?? pr.prNumber alias is gone from the forward write path (distiller.ts:268 uses linkedIssues[0]?.number and throws/flags when absent). Of the three gaps listed, only deduplication is genuinely still open in #129. Suggest narrowing this to dedup.

[alexosugo]

Verified on #129 head (58e1e0c): scraper.ts wires collectLinkedIssueRefs(prTitle, …), distiller.ts:268 dropped the ?? pr.prNumber alias and flags no-issue PRs, and relink-issues.spec.ts:96 covers the nested 10399→10182→10183 case. Narrowed the paragraph to deduplication as the one open gap. Fixed in ba5b5ce.

[sugat009]

note: Most of this roadmap is already implemented in the open #138 (same author). Ranks 1-3 and 5-8 map to dedup.ts (ciGuardReason, dedupeByIssueId + source_prs[]), reconcile.ts (hallucinationRate), the CONSTRAINTS block, stripBoilerplate + ISSUE_BODY_LIMIT_EXPANDED, and computeConfidence. Worth reframing as "tracking #138" so it reads as in-flight rather than net-new. One caveat: rank 4's specific design (nullable issueNumber, /pull URLs, cht-core-pr-<n>) is contradicted by #138, which keeps issueNumber required (distiller.ts:329 still emits /issues/${issueNumber}) and flags/skips no-issue PRs instead. Rank 4 should be rewritten to match the require-an-issue-or-skip approach that actually shipped.

[alexosugo]

Agreed. Added a status note framing the roadmap as tracking #138 (ranks 1-3, 5-8 → dedup.ts/reconcile.ts/CONSTRAINTS/stripBoilerplate/computeConfidence). Rewrote rank 4 and section 4.2 to the shipped require-an-issue-or-skip approach (issueNumber stays required, no-issue PRs flagged), and fixed the Part 4 invariant. Fixed in ba5b5ce.

[sugat009]

nitpick: "~60 of 107 drafts corpus-wide" is really the four fully-reviewed domains, not the corpus. The 8-domain table above sums to 255 memories; 60/107 is exactly contacts+auth+config+data-sync (44+39+10+14 = 107; mislinks 31+19+4+6 = 60). Section 1.1 (L63) then lists tasks (15/35) in the same sentence but still totals 60/107. Suggest relabeling "60/107 across the four fully-reviewed domains" (corpus-wide would be higher, out of 255) to keep the headline stat precise.

[alexosugo]

Relabeled: "~60 of 107 across the four fully-reviewed domains (contacts/auth/config/data-sync)" at L41, and reworked the 1.1 line so 60/107 is the four domains with tasks/messaging/forms/infra called out as additional. Fixed in ba5b5ce.

[sugat009]

nitpick (minor, non-blocking): A few small accuracy items in the appendix/fixtures: #10036 is x4 not x3 (merged PRs 10070/10061/10056/10043). "all 44 ... confidence: medium / related_issues: []" (L345) holds for the distilled drafts but not 4 legacy subDomain-schema drafts that carry neither field. Two Part 3 fixture titles (8843, 8675 at L252/255) are paraphrased rather than the verbatim PR titles. And CouchDB 9960/10014 (L421) close different issues (9882 vs 10027), so they are duplicated effort, not a single-issue backport pair.

[alexosugo]

All four fixed in ba5b5ce: #10036 → x4 (10070/10061/10056/10043); noted the 4 legacy subDomain-schema drafts lack confidence/related_issues; used the verbatim 8675/8843 titles; and clarified CouchDB #9960/#10014 close different issues (#9882 vs #10027) — duplicated effort, not a backport pair.

[sugat009]

Fact check of the document against the code on main and the GitHub record. Each finding was verified twice: by an independent adversarial pass, then by direct command output. The analysis is valuable, and much of it was accurate for the 2026-07-01 date. The 15 inline comments give the corrections.

Two rendering defects have no inline anchor: a line-initial - at lines 249-250 splits a bold span and creates a stray bullet, and lines 3-9 render as one paragraph.

[sugat009]

issue (blocking): The count contradicts the table below.

The table sums to 255 (44+39+10+14+35+47+49+17). The number 107 is the sum of the four fully-reviewed domains only. Line 65 confirms this scope: tasks, messaging, forms, and infra "add more on top". Suggested fix: "255 distilled memories across the 8 domains, ~107 fully reviewed in four of them". The headline ~60/107 rate then reads correctly as a four-domain figure.

[sugat009]

issue: The "Reviewer" column holds participants, not reviewers.

GitHub shows Hareet authored all 8 promotion PRs and submitted no reviews on any of them before this document's date. All formal reviews to 2026-07-01 are from sugat009. The cells match the real participation on that date, so the data is correct. The header is not. Fix: rename the column to "Participants", or list only review submitters. Note: Hareet later submitted one COMMENTED review, on #123 on 2026-07-21, and #123 is the one row that omits him.

[sugat009]

issue: baseRefName is not captured.

The scraper's --json field list does not request it, and ScrapedPR has no such field. It has zero occurrences in src/ at this document's code and on main. #138 adds it, which confirms the gap. The other two markers (body cherry-pick marker, title (#M) suffix) are captured. Scope "captured but unused" to those two, and note that baseRefName needs a scraper change plus a type change (also lines 165, 326).

[sugat009]

issue (blocking): The dedup design misses collisions with the merged corpus.

The design at lines 162-166 ("across ALL seeders, before promotion") compares only drafts inside the promotion batch. Two #123 drafts collided with memories already on main before this document's date: 10432 vs cht-core-10344 (on main since 2026-06-04) and 9975 vs cht-core-9974 (on main since 2026-03-31). Batch-internal dedup cannot catch these. Also, #6543 had a third seeder in #123 (9099-feat6543), beyond the two this row names. Fix: check the promotion set against the merged corpus, and extend the line 279 fixture past the 132/131 pair.

[sugat009]

issue: Prompt fixes 3 and 4 must also edit DRAFT_SHAPE, or CLI mode keeps the old text.

The field descriptions exist twice: in buildPrompt (~188, ~199) and in DRAFT_SHAPE (distiller.ts:80, :91). CLI mode appends DRAFT_SHAPE after the prompt (structured-cli.ts:52-55), so the old wording lands last. The rootCause copy in DRAFT_SHAPE (<why it was wrong>) is the vaguer of the two. #138 currently edits only buildPrompt, so this miss is already in that diff. Name both edit sites here.

[sugat009]

issue: The expected value 'high' is not reachable.

computeConfidence in #138 is typed 'medium' | 'low', and its docstring reserves 'high' for human-reviewed entries. Line 307 of this document also says the plan is only to downgrade. Change the expectation to 'medium' for the rich draft. Also, "a fallback-to-PR draft" is a state the merged code can no longer produce; use a flagged no-issue PR as the low-side fixture instead.

[sugat009]

suggestion (non-blocking): The input label points at the wrong mechanism.

The success criterion names the startsWith guard in sameRepoClosingRefs, which only sees sidebar metadata, never body text. A body link cannot reach it: bodyIssueRefs needs a Fixes/Closes/Resolves keyword, and its pattern cannot match the four-segment attacker path at all. Change "body link" to "closingIssuesReferences entry". Note: main already ships this exact test with this URL (test/scripts/issue-linkage.spec.ts:74-85), so this row points at an existing guard rather than adding a new test.

[sugat009]

issue: The root cause is incomplete. The body also had the correct reference.

PR 10198's body contains Closes medic/cht-core#8026. The regex accepts only a bare #N or a full URL after the keyword, so the qualified owner/repo#N form silently drops (verified by execution; the bare form matches). The number was not available only in the title. The gap is still in main's bodyIssueRefs. Add this form to the remediation list, next to the Fixes: #N colon form from the 10914 case.

[sugat009]

issue (blocking): The cause here is a linkage miss, not truncation.

The PR body reads Fixes: #10912 with a colon. The scraper regex requires whitespace directly after the keyword, so it did not match (verified by execution). linkedIssues was empty, and issue #10912 was never fetched. Proof: the as-distilled draft (commit bf92766) has issueNumber: 10914, so the PR-number fallback fired. Truncation cannot remove text that was never fetched.

Two follow-ups. First, the truncation scenario is now live on main: the merged linkage chain resolves #10912 (sidebar and title), and the mechanism sentence sits at character 589, past ISSUE_BODY_LIMIT=500. Keep rank 6, and recast this case's history as a linkage miss (also lines 87-89, 105, 178, 204, 287, 313-316). Second, no remediation names the keyword-colon form, and main's bodyIssueRefs still misses it. Small correction: the PHI comment is 343 characters, not ~246 (lines 87, 178, 314).

[sugat009]

issue: Both the count and the example are wrong.

Only 2 of 49 infra PRs truly close no issue: 8693 and 10689. PR 10488 closes #10481, as line 431 of this document says, so the two remediations for it conflict. PR 8996 closes #9118 through a body keyword. Separately, the measured mislink rate in this batch is 23 of 49, so ~12 understates the defect. Write "2/49 close no issue (8693, 10689)" and add the 23/49 mislink figure.