refactor(database,app): unify INDEX/MEMORY/SKILL on Resource + Entry backbone

docs/adr/0006-unified-resource-entry-lane-backbone.md

@@ -0,0 +1,121 @@
+# ADR 0006: Unify INDEX / MEMORY / SKILL onto a Resource + Entry Lane Backbone
+
+- Status: Accepted
+- Date: 2026-06-25
+
+## Context
+
+memU historically modeled structured memory as four record types — `Resource`,
+`MemoryItem`, `MemoryCategory`, `CategoryItem` — with retrieval running a fixed
+`category -> item -> resource` waterfall. Separately, the read-only `memory_fs`
+exporter projected markdown trees that were decoupled from retrieval.
+
+This produced asymmetric concepts:
+
+- INDEX: `Resource.caption` + verbatim `resource/` copies
+- MEMORY: `MemoryCategory.summary` + `memory/<slug>.md`
+
+We want INDEX, MEMORY, and SKILL to share **one backbone** with **consistent
+storage and retrieval**, all derived from the same per-resource canonical text.
+Each lane is the same processing track; lanes differ only in their entry-type
+set, extraction prompts, and how entries are grouped into coarse lane docs.
+
+## Decision
+
+Collapse the model to **two first-class, lane-tagged entities plus one edge**.
+
+### Lane
+
+A `lane` discriminator: `index`, `memory`, and `skill`. (Raw inputs use
+`lane="source"`.) The lanes are parallel, structurally identical processing
+tracks over a shared trunk; they differ only in *what the extractor pulls out*
+(per-lane `entry_type` set and prompts) and *the entry→resource grouping
+cardinality*:
+
+- `index` — `entry_type` ∈ {`description`}, **`per_resource`** grouping: one
+  coarse description doc per source resource (1:1, no LLM grouping).
+- `memory` — `entry_type` ∈ {`profile`, `event`, `knowledge`}, **`adaptive`**
+  grouping: the extractor proposes group names and a summarized category doc is
+  synthesized per group.
+- `skill` — `entry_type` ∈ {`tool`, `log`}, **`adaptive`** grouping: entries are
+  grouped into summarized skill docs (analogous to memory categories).
+
+Per-lane behavior is configured via `MemorizeConfig.lanes` (a `dict[str,
+LaneConfig]`); all three lanes are enabled by default. Each adaptive lane has its
+own summary prompt / target length / LLM profile.
+
+### Entities
+
+1. **`Resource`** (lane-tagged, one physical table — "everything is a resource"):
+   - Raw source artifacts (`lane="source"`, `modality` = video/image/audio/
+     conversation/document); multimodal preprocessing fills `content` (the
+     canonical, modality-agnostic text — the shared trunk).
+   - Generated coarse docs (`lane` ∈ {index, memory, skill},
+     `modality="markdown"`), each rendered as a file under the `resource/` root:
+     - `resource/index/<slug>.md` — a description page linking to a raw resource
+     - `resource/memory/<slug>.md` — a category page
+     - `resource/skill/<slug>.md` — a skill page
+   - Carries `embedding` (for coarse recall) and `resource_refs` provenance back
+     to the raw sources it derives from.
+   - This **absorbs the former `MemoryCategory`** (a category is just a
+     `lane="memory"` markdown resource).
+
+2. **`Entry`** (lane-tagged, one physical table — the searchable atom):
+   - index → a resource description; memory → a memory item; skill → a tool/log.
+   - Carries `text`, `embedding`, `entry_type` (the per-lane sub-type, which
+     selects the extraction prompt), `extra`, and `source_path` — a back-link to
+     the originating raw resource, relative to the `resource/` root. (This
+     **generalizes the former `MemoryItem`**.)
+
+3. **`ResourceEntry`** (edge): membership of an `Entry` in its coarse lane
+   `Resource` (memory item ∈ category page, description ∈ index page, tool/log ∈
+   skill page). Many-to-many. (This **generalizes the former `CategoryItem`**.)
+
+### Links / provenance
+
+- `Entry.source_path` points only at the originating raw resource (relative to
+  the `resource/` root).
+- A coarse `Resource`'s provenance (`resource_refs`) is the union of its member
+  entries' sources, stored redundantly to avoid a query-time join.
+- All paths are relative to the single `resource/` root, so the same value works
+  for both the retrieval API and the exported tree.
+
+### Pipelines
+
+- **memorize**: `ingest -> preprocess_multimodal (-> Resource.content) ->
+  extract_lanes (per enabled lane: index/memory/skill extractors) ->
+  embed_entries -> persist lane resources (per_resource 1:1 doc or adaptive
+  grouped+summarized docs) -> build_response`.
+- **retrieve**: a single `route_intention` pass, then for each enabled lane,
+  `Resource` recall (stored embedding, `where lane=`) → `Entry` recall (stored
+  embedding, `where lane=`), plus a `source`-lane resource recall, returning a
+  per-lane shape `{lanes: {index, memory, skill}, resources: [...]}` (with
+  backward-compatible top-level `categories`/`items` mirroring the memory lane).
+  All lanes traverse the same code path; only the `lane` filter differs. Both
+  `rag` and `llm` ranking methods are supported.
+
+### Naming
+
+`lane`, `Resource`, `Entry`, `ResourceEntry`, `content` (canonical text),
+`source_path`, `resource_refs`. The former `Doc`/`LaneDoc` concept is dropped:
+a "doc" is just a markdown `Resource`, which avoids mislabeling a video/image as
+a document.
+
+## Consequences
+
+Positive:
+
+- One storage schema and one retrieval path for all lanes (true
+  storage/retrieval consistency).
+- Every entry and coarse resource is traceable back to its raw source.
+- "Everything is a resource" keeps the mental model and the on-disk tree aligned.
+
+Negative / risk:
+
+- Breaking schema change: `MemoryCategory` folds into `Resource`; `MemoryItem` →
+  `Entry`; `CategoryItem` → `ResourceEntry` (field renames included). All three
+  backends (`inmemory`, `sqlite`, `postgres`) and the app layer (`memorize`,
+  `retrieve`, `crud`) must be migrated together.
+- `retrieve` is category-centric today and needs a substantial rewrite.
+- Stored vs query-time category embeddings are unified onto stored embeddings,
+  changing recall behavior slightly.

docs/architecture.md

@@ -8,12 +8,28 @@ The repository also describes a hosted Cloud product in `README.md`, but this do
 
 ## System overview
 
-memU implements structured agent memory with four persistent record types:
-
-- `Resource`: raw source artifacts (conversation/document/image/video/audio)
-- `MemoryItem`: extracted atomic memories with embeddings
-- `MemoryCategory`: grouped topic summaries
-- `CategoryItem`: item-category relation edges
+memU implements structured agent memory on a unified, lane-tagged backbone
+(see ADR 0006). "Everything is a resource", with two first-class record types
+plus one edge:
+
+- `Resource`: a `lane`-tagged node — either a raw source artifact
+  (`lane="source"`, modality conversation/document/image/video/audio; the
+  canonical preprocessing text lives in `content`) or a generated markdown doc
+  (`lane` ∈ {index, memory, skill}, `modality="markdown"`). A memory-lane
+  `Resource` is the former "category".
+- `Entry`: a `lane`-tagged searchable atom with an embedding (index → a
+  description, memory → a memory item, skill → a tool/log). Its per-lane
+  `entry_type` (memory: profile/event/knowledge, skill: tool/log, index:
+  description) selects the extraction prompt during memorize. Links to its origin
+  via `source_id`/`source_path`.
+- `ResourceEntry`: membership edge from an `Entry` to its coarse (lane) `Resource`.
+
+The retrieval lanes (index/memory/skill) are parallel, structurally identical
+tracks over the shared `Resource.content` trunk; retrieval runs the same
+`Resource recall → Entry recall` per lane, parameterized by `lane`. Per-lane
+extraction/grouping is configured via `MemorizeConfig.lanes`; `index` uses
+`per_resource` grouping (1:1 description docs) while `memory`/`skill` use
+`adaptive` grouping (LLM-grouped, summarized docs).
 
 At runtime, `MemoryService` orchestrates ingestion, retrieval, and manual CRUD over these layers.
 
@@ -23,10 +39,9 @@ flowchart TD
   B --> C["Workflow Pipelines"]
   C --> D["LLM Clients"]
   C --> E["Database Repositories"]
-  E --> F["Resources"]
-  E --> G["Memory Items"]
-  E --> H["Memory Categories"]
-  E --> I["Category Relations"]
+  E --> F["Resources (lane: source/index/memory/skill)"]
+  E --> G["Entries (lane: index/memory/skill)"]
+  E --> I["ResourceEntry edges"]
 ```
 
 ## Core runtime components
@@ -121,20 +136,23 @@ Key behavior:
 
 ### Repository contracts
 
-Storage is abstracted through a `Database` protocol with four repositories:
+Storage is abstracted through a `Database` protocol with three repositories:
+
+- `ResourceRepo` (incl. `get_or_create_doc`, `update_resource`, lane-filtered
+  `vector_search_resources`)
+- `EntryRepo` (incl. `vector_search_entries`, similarity/salience), lane-filtered
+- `ResourceEntryRepo` (membership edges)
 
-- `ResourceRepo` (incl. `vector_search_resources`)
-- `MemoryItemRepo` (incl. `vector_search_items`, similarity/salience)
-- `MemoryCategoryRepo`
-- `CategoryItemRepo`
+All three repos take an optional `lane` filter so a single physical table per
+record type backs every lane (the `lane` column is the discriminator).
 
 Vector ranking over **stored** embeddings is a repository responsibility:
-`vector_search_items` and `vector_search_resources` keep the retrieval layer from
-reaching into any concrete backend. The pure cosine/salience math lives in the
-storage-neutral `memu.vector` module (not under any backend), so the app layer
-and every backend depend on it instead of on each other. (Category recall still
-ranks freshly re-embedded summaries at query time in the retrieval layer, since
-that is query-time policy rather than search over stored vectors.)
+`vector_search_entries` and `vector_search_resources` keep the retrieval layer
+from reaching into any concrete backend. The pure cosine/salience math lives in
+the storage-neutral `memu.vector` module (not under any backend), so the app
+layer and every backend depend on it instead of on each other. (Memory-doc
+recall still ranks freshly re-embedded summaries at query time in the retrieval
+layer, since that is query-time policy rather than search over stored vectors.)
 
 ### Backends
 
@@ -148,7 +166,7 @@ For Postgres, startup runs migration bootstrap and attempts `CREATE EXTENSION IF
 
 ### Scope model propagation
 
-`UserConfig.model` is merged into record/table models so scope fields (for example `user_id`) become first-class columns/attributes across resources, items, categories, and relations.
+`UserConfig.model` is merged into record/table models so scope fields (for example `user_id`) become first-class columns/attributes across resources, entries, and resource-entry edges.
 
 This is why `where` filters and `user_data` writes are consistently available across APIs.
 
@@ -235,36 +253,25 @@ payload directory:
 <output_dir>/
 ├── INDEX.md                     ← index of the raw files under resource/
 ├── MEMORY.md                    ← overview + index of memory/
-├── SKILL.md                     ← index of the skills under skill/
 ├── resource/
 │   └── <file_name>              ← one copied raw source file (verbatim bytes)
-├── memory/
-│   └── <slug>.md                ← one MemoryCategory (description + summary)
-└── skill/
-    └── <skill_name>/SKILL.md    ← one synthesized skill per folder
+└── memory/
+    └── <slug>.md                ← one memory-lane Resource (description + summary)
 ```
 
 - `resource/` holds the raw source files copied verbatim out of the blob store
   (`Resource.local_path`); `INDEX.md` indexes them (name, modality, description,
   link), so an agent knows which raw resources exist.
-- `memory/<slug>.md` is the living memory split one file per `MemoryCategory`
-  (its description + summary); `MEMORY.md` is an overview that links to each one.
-- `skill/<name>/SKILL.md` is a reusable skill synthesized from the descriptions
-  (a sibling of `MEMORY.md`, never derived from extracted skill-type memory
-  items); the root `SKILL.md` indexes the tree.
+- `memory/<slug>.md` is the living memory split one file per memory-lane
+  `Resource` (its description + summary); `MEMORY.md` is an overview that links to each one.
 
 ### Synthesis mode
 
-The `skill/` tree is always synthesized from the per-source descriptions by an LLM
-(`memu.memory_fs.MemorySynthesizer`, prompts in `memu.prompts.memory_fs`) — one
-pass extracts skills as a JSON array of `{name, body}` objects, each written as its
-own `skill/<name>/SKILL.md` doc. It is never derived from extracted skill-type
-memory items.
-
 `MEMORY.md` is rendered deterministically by default: an overview that links to the
 per-category `memory/<slug>.md` files (themselves deterministic from category
 description + summary). When `memory_files_config.synthesize=True`, the `MEMORY.md`
-body is instead synthesized from all descriptions in one LLM pass.
+body is instead synthesized from all descriptions in one LLM pass
+(`memu.memory_fs.MemorySynthesizer`, prompts in `memu.prompts.memory_fs`).
 
 `INDEX.md`, the `resource/` copies, and `memory/<slug>.md` stay deterministic in
 both modes. Synthesis uses the `synthesis_llm_profile` profile and leaves the
@@ -278,13 +285,10 @@ model. `MemoryFilesBuilder.build(database, where, changed=...)` (delegated to fr
 
 - **Initialization** (no prior tree on disk, or `changed is None`): scan all
   in-scope sources, turn each into its multimodal description, and synthesize the
-  `skill/` tree (and, when `synthesize=True`, the `MEMORY.md` body) from scratch
-  (`MemorySynthesizer.synthesize` / `synthesize_skills`).
+  `MEMORY.md` body from scratch (`MemorySynthesizer.synthesize`).
 - **Incremental update** (a tree already exists and a changed set is supplied):
-  read the existing skill bodies (and `MEMORY.md` body) back off disk and merge
-  only the changed sources' descriptions into them (`MemorySynthesizer.update` /
-  `update_skills`, prompts `MEMORY_UPDATE_PROMPT` / `SKILL_UPDATE_PROMPT`). Skills
-  are upserted by slug, so untouched skills survive.
+  read the existing `MEMORY.md` body back off disk and merge only the changed
+  sources' descriptions into it.
 
 `INDEX.md`, `resource/`, and `memory/` are always recomputed from the current
 store, so they need no LLM merge. `export_memory_files(user=...)` always takes the
@@ -298,7 +302,7 @@ does **not** drive the exporter; it is left entirely untouched.
 The exporter is read-only against the database and disabled by default
 (`memory_files_config.enabled`). Diff detection is handled by a sidecar manifest
 (`.memufs_manifest.json`) that stores per-file content hashes, so each export
-only rewrites artifacts whose rendered content changed (and prunes stale skill
+only rewrites artifacts whose rendered content changed (and prunes stale
 files/dirs) — no database schema change is required. Rendered content avoids
 volatile values so an unchanged store re-exports as a no-op. Exports are
 serialized through a per-service lock.
@@ -315,3 +319,4 @@ serialized through a per-service lock.
 - `docs/adr/0001-workflow-pipeline-architecture.md`
 - `docs/adr/0002-pluggable-storage-and-vector-strategy.md`
 - `docs/adr/0003-user-scope-in-data-model.md`
+- `docs/adr/0006-unified-resource-entry-lane-backbone.md`

docs/tutorials/getting_started.md

@@ -113,9 +113,9 @@ async def main() -> None:
         memory_content = "The user is a senior Python architect who loves clean code and type hints."
 
         # We use 'create_memory_item' to insert a single memory record.
-        # memory_type='profile' indicates this is an attribute of the user.
+        # entry_type='profile' indicates this is an attribute of the user.
         result = await service.create_memory_item(
-            memory_type="profile",
+            entry_type="profile",
             memory_content=memory_content,
             memory_categories=["User Facts"],
         )
@@ -135,7 +135,7 @@ async def main() -> None:
         if items:
             print(f"[OK] Found {len(items)} relevant memory item(s):")
             for idx, item in enumerate(items, 1):
-                print(f"   {idx}. {item.get('summary')} (Type: {item.get('memory_type')})")
+                print(f"   {idx}. {item.get('text')} (Type: {item.get('entry_type')})")
         else:
             print("[!] No relevant memories found.")
 
@@ -153,7 +153,7 @@ if __name__ == "__main__":
 ### Understanding the Code
 
 1.  **Initialization**: We configure `MemoryService` with specific `llm_profiles`. This tells MemU which model to use. We also define a `memorize_config` with a "User Facts" category. Categories help the LLM organize and retrieve information more effectively.
-2.  **Memory Injection**: `create_memory_item` is used to explicitly add a piece of knowledge. We tag it with `memory_type="profile"` to semantically indicate this is a user attribute.
+2.  **Memory Injection**: `create_memory_item` is used to explicitly add a piece of knowledge. We tag it with `entry_type="profile"` to semantically indicate this is a user attribute.
 3.  **Retrieval**: We use `retrieve` with a natural language query. MemU's internal workflow ("RAG" or "LLM" based) will determine the best way to find relevant memories.
 
 ## Troubleshooting