feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session memory: a cache-safe <skill-memory> block is appended to each skill tool RESULT with intent-scoped recall (embeddings + FTS) and historian auto-extraction into global notes. Ships the opt-in distill-skill-memory dreamer task, ctx_skill_note/ctx_skill_recall tools, TUI/CLI stats, and DB migrations to v52 with a self-heal path.

• New Features

  • Transparent per-skill recall: the skill tool optionally accepts intent; before-hook stashes it and after-hook injects recall; ctx_skill_note writes and ctx_skill_recall reads; Status/TUI show per-project totals and pinned counts.
  • Ranking + embeddings: cosine blend over intent_embedding + delta_embedding with FTS5 fallback; programmatic re-embed backfills NULL/stale vectors before distill-skill-memory.
  • Historian auto-extraction: TC: skill(<name>) markers → <skill_observations>; both runners promote via promoteSkillObservations as global '*' notes (source_type='historian').
  • Schema/migrations/self-heal: skill_memory table + FTS vtable/columns; LATEST_SUPPORTED_VERSION=52; initializeDatabase re-creates missing objects and rebuilds skill_memory_fts when needed; dashboard/CLI/config include the distill-skill-memory task (off by default).

• Bug Fixes

  • Cross-partition recall: union a skill’s partition with global '*' so project-local skills surface historian-written notes; writes/dedup remain exact-partition.
  • Disabled-path safety and routing: guard ctx_skill_note when the plugin is disabled; correctly classify singular ~/.config/opencode/skill/** as global; normalize Windows paths.
  • Frontmatter parsing: accept inline skill-memory: { enabled: true }; anchor to start-of-file; tolerate leading BOM/whitespace; strip inline # from unquoted scalars.
  • Historian marker + intent stash: emit TC: skill(<name>) verbatim (CR/LF/tab only are unsafe); key stashed intents by sessionID:callID and prune on session delete.
  • Prompt guidance + observability: drop ctx_memory cross-reference in skill guidance when memory is off; log recall errors; escape apostrophes in SQL.
  • Token budgeting + tier derivation: count per-note XML framing and clamp pinned budget; derive tier via dirname(resolvedPath) for robustness.

^{Written for commit 9c62039. Summary will update on new commits.}

Greptile Summary

This PR introduces skill-memory — per-skill cross-session recall that attaches a <skill-memory> block to each skill tool result, driven by a 4-rung retrieval cascade (cosine embeddings → FTS5 → flat recency) and historian auto-extraction. The feature is fully opt-in per skill via SKILL.md frontmatter and cache-safe by design (appended to tool results, never to the cached system prefix).

• Core feature (P1–P3b): skill_memory table + three DB migrations (v50–v52); ctx_skill_note / ctx_skill_recall tools; before/after hook augmentation with session-scoped intent stash; promoteSkillObservations writes historian-extracted notes as global '*'-tier entries recallable from any project.
• Recall correctness: recallPartitionPredicate unions the skill's own partition with the global '*' partition so project-local skills also surface historian-written notes — the previous P1 (orphaned historian notes) is fixed.
• Session lifecycle: intentByCallId is now keyed sessionID:callID and pruned by prefix in onSessionDeleted — the previous P1 (cross-session eviction) is fixed. The initializeDatabase self-heal backfills skill_memory_fts when rows exist but the FTS index is empty — the previous P1 (missing rebuild) is also fixed.

Confidence Score: 5/5

Safe to merge; all three correctness issues surfaced in earlier rounds are verifiably fixed in this revision.

The intent stash is now keyed sessionID:callID with per-session prefix pruning, initializeDatabase fires the guarded FTS rebuild, and recallPartitionPredicate unions the own partition with the global '*' partition so historian-written notes reach project-local skills. The remaining findings are maintenance-quality observations — searchSkillMemoryFts calling recallPartitionPredicate twice, the pre-computed part value being passed where raw projectIdentity is expected (idempotent, never wrong), and a cosmetic inaccuracy in the ctx_skill_note success message for global-tier skills.

No files require special attention. The recall.ts and storage.ts recallPartitionPredicate call-site patterns are worth tidying but carry no correctness risk.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	Core DB storage layer for skill_memory; introduces recallPartitionPredicate for cross-partition recall. All read paths union the skill's own partition with the global '*' partition; write/dedup paths stay exact-partition. Idiomatic and well-guarded.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	Implements the 4-rung recall cascade (cosine, FTS5, flat). Computes part = partitionKey(scope, projectIdentity) then passes it to storage functions that internally re-apply partitionKey — idempotent so never wrong, but confusing against the API contract. searchSkillMemoryFts also calls recallPartitionPredicate twice (clause + binds) which is a maintenance hazard.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	Historian-extraction write path: promotes observations as global '*' notes with dedup via normalized hash. Best-effort per item, logs errors. Clean and correct.
packages/plugin/src/features/magic-context/skill-memory/frontmatter.ts	Custom YAML frontmatter parser for skill-memory config. Correctly handles inline flow-mapping, block form, inline comments, BOM/leading whitespace, and malformed input. Anchored to start-of-file to avoid mid-document horizontal rules. Well-tested.
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	Parses 'Base directory for this skill:' provenance line using the last regex match (avoids content-shadowing), uses fileURLToPath for cross-platform correctness, and correctly classifies global vs project tiers including singular ~/.config/opencode/skill/ path.
packages/plugin/src/features/magic-context/migrations.ts	Adds migrations v50 (skill_memory table), v51 (delta_embedding, recall_count, FTS5 vtable + triggers + backfill rebuild), v52 (origin_project, source_type, collision-merge global-tier notes to '*'). Migration bodies use ensureColumn/IF NOT EXISTS for idempotency. v52 runs in a transaction.
packages/plugin/src/features/magic-context/storage-db.ts	initializeDatabase self-heal extended: creates skill_memory with all columns, skill_memory_fts and triggers IF NOT EXISTS, then guards an FTS rebuild (ftsCount=0 && rowCount>0). ensureColumn calls added for all v51/v52 columns.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	After-hook now correctly keys intent stash as sessionID:callID (previous P1 fixed), hoists the single await import(…provenance) above both try blocks (previous comment addressed). Session-scoped registry population and skill-memory injection path both look correct.
packages/plugin/src/tools/ctx-skill-note/tools.ts	Records skill-specific notes with hash dedup and semantic dedup via cosine similarity. Success message says 'in this project' for all tiers but global-tier notes are cross-project. Passes already-partitioned part to storage functions that internally re-apply partitionKey (idempotent).
packages/plugin/src/hooks/magic-context/hook.ts	Session lifecycle correctly prunes intentByCallId by prefix via pruneIntentsForSession — the previous P1 (clear-all-on-delete) is fixed. skillLoadRegistry pruned by sessionId prefix. All per-session maps cleaned up in onSessionDeleted.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant SkillTool as skill tool
    participant BeforeHook as tool.execute.before
    participant AfterHook as tool.execute.after
    participant Registry as SkillLoadRegistry
    participant RecallCore as recallSkillMemoryBlock
    participant DB as SQLite (skill_memory)
    participant Historian

    Agent->>SkillTool: skill(name, intent)
    SkillTool->>BeforeHook: stash intent (sessionID:callID)
    SkillTool-->>Agent: tool output (Base directory: file:///...)
    AfterHook->>AfterHook: parseSkillProvenance(output)
    AfterHook->>AfterHook: parseFrontmatterConfig(SKILL.md)
    AfterHook->>Registry: set(sessionID:skillId, provenance+config)
    AfterHook->>AfterHook: getAndDeleteIntent(sessionID:callID)
    AfterHook->>RecallCore: recallSkillMemoryBlock(skill, intent, scope, projectIdentity)
    RecallCore->>DB: "embedTextForProject(intent) -> vector"
    alt embeddings available
        RecallCore->>DB: "getRankingCandidates -> cosine rank"
    else FTS fallback
        RecallCore->>DB: searchSkillMemoryFts(sanitized intent)
    else flat fallback
        RecallCore->>DB: getSkillMemoryNotes (recency x hit)
    end
    RecallCore->>DB: bumpRecallCountByIds
    RecallCore-->>AfterHook: skill-memory block
    AfterHook->>AfterHook: append block to output.output
    Agent->>DB: "ctx_skill_note -> insertSkillMemoryNote / bumpHitCount"
    Historian->>Historian: detect TC: skill(name) markers
    Historian->>DB: "promoteSkillObservations (tier=global, project_identity='*')"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant SkillTool as skill tool
    participant BeforeHook as tool.execute.before
    participant AfterHook as tool.execute.after
    participant Registry as SkillLoadRegistry
    participant RecallCore as recallSkillMemoryBlock
    participant DB as SQLite (skill_memory)
    participant Historian

    Agent->>SkillTool: skill(name, intent)
    SkillTool->>BeforeHook: stash intent (sessionID:callID)
    SkillTool-->>Agent: tool output (Base directory: file:///...)
    AfterHook->>AfterHook: parseSkillProvenance(output)
    AfterHook->>AfterHook: parseFrontmatterConfig(SKILL.md)
    AfterHook->>Registry: set(sessionID:skillId, provenance+config)
    AfterHook->>AfterHook: getAndDeleteIntent(sessionID:callID)
    AfterHook->>RecallCore: recallSkillMemoryBlock(skill, intent, scope, projectIdentity)
    RecallCore->>DB: "embedTextForProject(intent) -> vector"
    alt embeddings available
        RecallCore->>DB: "getRankingCandidates -> cosine rank"
    else FTS fallback
        RecallCore->>DB: searchSkillMemoryFts(sanitized intent)
    else flat fallback
        RecallCore->>DB: getSkillMemoryNotes (recency x hit)
    end
    RecallCore->>DB: bumpRecallCountByIds
    RecallCore-->>AfterHook: skill-memory block
    AfterHook->>AfterHook: append block to output.output
    Agent->>DB: "ctx_skill_note -> insertSkillMemoryNote / bumpHitCount"
    Historian->>Historian: detect TC: skill(name) markers
    Historian->>DB: "promoteSkillObservations (tier=global, project_identity='*')"

Loading

Comments Outside Diff (1)

1. packages/plugin/src/features/magic-context/skill-memory/recall.ts, line 481-484 (link)

   Bare catch {} makes recall errors invisible

   Any exception thrown inside recallSkillMemoryBlock — a SQLite error, embedding provider crash, or a coding mistake — is silently swallowed and returns an empty string. From the caller's perspective this is indistinguishable from "no notes exist" or "skill-memory disabled". At minimum a sessionLog or console.warn call here would let a developer diagnose why the recall block is absent without attaching a debugger.

_{Reviews (6): Last reviewed commit: "fix(skill-memory): gate ctx_memory cross..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/plugin/src/features/magic-context/dreamer/task-executor.ts, line 419:

<comment>Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</comment>

<file context>
@@ -412,7 +413,18 @@ export function createDreamTaskExecutor(deps: DreamTaskExecutorDeps): TaskExecut
+            // Skill-memory P2: re-embed stale/NULL skill-note vectors BEFORE the
+            // distill agentic pass so the prompt sees a fully-embedded corpus and
+            // P1-era notes promote from the FTS rung to the cosine rung.
+            if (config.task === "distill-skill-memory") {
+                try {
+                    await reembedStaleSkillNotes(db, projectIdentity);
</file context>

[iceteaSA]

Holding this one as working-as-intended. The catch is not silent — it logs [dreamer] distill-skill-memory reembed pre-step failed: ${e} (task-executor.ts, the line directly inside the catch). The suppression is deliberate: re-embedding is a best-effort optimization pre-step, not a prerequisite — if it fails, the affected notes simply stay on the FTS recall rung instead of the cosine rung, and the distill report still runs correctly. Letting a reembed failure fail the whole task would block a report that doesn't depend on it. The failure is observable via the log; happy to escalate to a metric/warning if you'd prefer louder signal.

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>