feat: layered, track-aware workspace memorize & retrieve#466
Merged
Conversation
Add an optional `track` column to the Resource model that records which workspace subtree a file came from: files under chat/ → "chat", under agent/ → "skill", everything else → "workspace". Classification happens in memorize_workspace and threads through the workflow as the `resource_track` state key. Legacy single-file memorize leaves it None. The column is nullable across all three backends (in-memory, SQLite, Postgres), mirroring how RecallFile.track was added — created via metadata.create_all(), no migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-file `resource → entry → file` workspace pipeline (plus the `generate_skills` bypass) with a direct `resource → file` synthesis for the chat and skill tracks: - Two-step extraction: (a) route the source to the set of files to update/create given existing files' names+descriptions, then (b) synthesize each target file's body in parallel. No RecallEntry is created on this path. - Track routing: chat → memory-track RecallFile, skill → skill-track RecallFile, workspace → resource-only (no file, no entry). - Add a RecallFileResource(resource_id, category_id) join table mirroring RecallFileEntry across all three backends (inmemory/sqlite/postgres), recording resource → file provenance; cascade-delete unlinks it. - Single-file `memorize` (resource → entry → file) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce RecallFileSegment (ADR 0007 L2 item): a searchable slice of a RecallFile carrying text + embedding + recall_file_id. Each file has 1..n segments, embedded as the retrieval unit that rolls up to its file. No ordinal — slicing is track-specific and not necessarily sequential. Threaded through the full stack like RecallFileResource: core model + inmemory/postgres/sqlite models and schemas (file_segments table), RecallFileSegmentRepo protocol and three backend repos, DatabaseState, interfaces, and store wiring. Segments are (re)built after workspace file synthesis: - skill track: one "name/description" segment per file - memory track: one segment per content line, skipping blanks and headings On update, diff existing vs new segment texts and drop-and-add only the difference so unchanged lines keep their embeddings. Cleared alongside categories in the crud clear path.
Rework retrieve_workspace to return segments/files/resources: - segments: RecallFileSegment ranked by embedding, file.top_k - files: roll-up of the files those segments point to (max segment score), not a ranked search - resources: workspace-track resources ranked by embedding, resource.top_k The entry layer is off (config retained but ignored). Add a denormalized `track` to RecallFileSegment, mirrored from the owning file at creation, so segment retrieval filters by track with a plain column predicate instead of a join. Track is immutable (segments are drop-and-recreated on re-slice), so it never drifts from the file. Threaded through create_segment across the interface and all three backends; _ws_recall_segments applies file.tracks as a track__in filter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks the workspace memorize/retrieve pipeline into a layered, track-aware design (ADR 0007). Source resources now synthesize directly into files, files are sliced into embedded segments as the retrieval unit, and both resources and files carry a
tracktag that scopes retrieval. Built up across four commits:feat: track resource provenance in workspace memorize — Add an optional
trackcolumn toResourcerecording the workspace subtree a file came from (chat/→chat,agent/→skill, elseworkspace). Classified inmemorize_workspaceand threaded through the workflow asresource_track. Legacy single-file memorize leaves itNone. Nullable across all three backends, no migration.feat: resource → file workspace memorize — Replace the
resource → entry → fileworkspace pipeline (and thegenerate_skillsbypass) with directresource → filesynthesis for chat and skill tracks. Two-step extraction: route the source to the target files, then synthesize each file body in parallel — noRecallEntrycreated. Adds aRecallFileResource(resource_id, category_id)join table recording resource → file provenance with cascade-delete. Single-filememorizeunchanged.feat: add RecallFileSegment layer for file retrieval — Introduce
RecallFileSegment(ADR 0007 L2 item): a searchable slice of aRecallFilecarrying text + embedding +recall_file_id, embedded as the retrieval unit that rolls up to its file. Threaded through the full stack (core model, three backends, repos, schemas,file_segmentstable). Segments are (re)built after file synthesis — one per file for skills, one per content line for memory — with a diff-based drop-and-add so unchanged lines keep their embeddings.feat: workspace retrieve over segments; denormalize track onto segment — Rework
retrieve_workspaceto return segments (ranked by embedding,file.top_k), files (roll-up of the files those segments point to, max segment score), and resources (workspace-track resources ranked by embedding,resource.top_k). The entry layer is off. DenormalizetrackontoRecallFileSegment(mirrored from the owning file at creation, immutable) so segment retrieval filters by track with a plain column predicate instead of a join.Test plan
tests/test_skill_track.pytests/test_folder_memorize.py🤖 Generated with Claude Code