Catalog dedup, curation + Forge Studio (Phase 3)

CHANGELOG.md

@@ -9,6 +9,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Catalog dedup, curation + Forge Studio (Phase 3).** Find near-duplicate
+  episodes from the embeddings and curate a clean, labeled training set.
+
+  ```bash
+  forge catalog dedup -c ./forge-catalog --threshold 0.97
+  forge curate -c ./forge-catalog --where "overall_score > 6" \
+      --dedup 0.97 --dedup-policy keep-higher-quality --label approved
+  forge studio -c ./forge-catalog -o studio.html
+  ```
+
+  - Two new tables (bumps catalog `SCHEMA_VERSION` 2 → 3, additive):
+    `dedup_edges` (near-dup pairs as *facts* — similarity, not verdicts) and
+    `curation_labels` (an append-log of approve/reject/hold decisions,
+    latest-row-wins).
+  - `forge catalog dedup` computes near-dup pairs (cosine over episode
+    embeddings, max over shared cameras; idempotent). `forge curate` applies a
+    WHERE filter + a dedup **policy** (`keep-higher-quality` / `keep-longer` /
+    `keep-first`) and labels survivors approved, dedup losers rejected.
+  - DuckDB views/macros: `v_curation` (latest label per episode) and
+    `v_dup_losers(threshold, policy)`.
+  - **`forge studio`** generates a self-contained, themed HTML app (Overview ·
+    Corpus · Dedup review · Snapshot) from real catalog data and embedded video
+    thumbnails — one shareable file, no server.
+  - Reuses the Phase 2 vectors, the readers for thumbnails, and the catalog
+    writer/commit machinery. The per-dataset `forge dedup` (perceptual-hash) is
+    unchanged; catalog dedup lives under `forge catalog dedup`.
+  - See [forge/catalog/README.md](forge/catalog/README.md).
+
 - **Catalog embeddings + semantic search (Phase 2).** Embed the episodes in a
   catalog and search them by natural language.
 

README.md

@@ -308,6 +308,32 @@ forge search --like <episode_id> -c ./forge-catalog
 
 Vectors are versioned per model (`model_id = siglip-so400m@<ckpt-hash>`) and stored in the same append-only catalog. Brute-force cosine in DuckDB is sub-second at lab scale. See [forge/embed/README.md](forge/embed/README.md) for models, device selection, and reproducibility.
 
+### Dedup & curation
+
+Use the embeddings to find near-duplicate episodes (re-encodes, near-identical retakes) and curate a clean, labeled training set. Near-dup pairs are recorded as **facts** (`dedup_edges`); which episode wins is decided at curation time by **policy**.
+
+```bash
+# Find near-duplicate pairs (cosine over episode embeddings)
+forge catalog dedup -c ./forge-catalog --threshold 0.97
+
+# Approve a high-quality selection, dropping dedup losers by policy
+forge curate -c ./forge-catalog \
+    --where "overall_score > 6 AND task = 'pick_place'" \
+    --dedup 0.97 --dedup-policy keep-higher-quality --label approved
+```
+
+Curation is an append-log (`curation_labels`, latest-row-wins); nothing is ever deleted. Policies: `keep-higher-quality`, `keep-longer`, `keep-first`.
+
+### Forge Studio
+
+Generate a self-contained, themed HTML app to explore the catalog visually — Overview, Corpus (with thumbnails + quality rings), Dedup review (keep/reject pairs), and a Snapshot preview:
+
+```bash
+forge studio -c ./forge-catalog -o studio.html && open studio.html
+```
+
+Everything is embedded (real data + video thumbnails as data URIs) — one shareable file, no server. See [forge/catalog/README.md](forge/catalog/README.md).
+
 ## Dataset Registry
 
 A curated catalog of 23+ prominent robotics datasets — browse, search, and download by name instead of memorizing URIs. **[Browse the registry online](https://arpitg1304.github.io/forge/registry.html)**

forge/catalog/README.md

@@ -108,16 +108,41 @@ for teleop data).
 
 ## Query surface
 
-Views registered on every connection:
+Views/macros registered on every connection:
 
-- `episodes`, `quality_scores` — the raw tables.
+- `episodes`, `quality_scores`, `embeddings`, `dedup_edges`, `curation_labels`
+  — the raw tables.
 - `v_latest_quality` — the quality row for the newest `scorer_version` per
   episode (then most recent `computed_at`).
+- `v_curation` — the latest label per episode (append-log, newest wins).
+- `v_dup_losers(threshold, policy)` — a table macro returning the episodes that
+  *lose* a near-dup pairing under a policy.
 
 ```python
 cat.sql("SELECT task, count(*) FROM episodes GROUP BY task")   # -> pyarrow.Table
 ```
 
+## Dedup, curation & Studio (Phase 3)
+
+- **`dedup_edges`** — near-duplicate pairs as *facts*: `(episode_a, episode_b,
+  similarity, model_id, method)`, canonical order `a < b`. Which episode "wins"
+  is decided at curation, never baked into the edge. `forge catalog dedup`
+  computes them from the Phase 2 embeddings (cosine, max over shared cameras;
+  idempotent). Brute-force is sub-second at lab scale; past ~20k episodes an
+  ANN pass is the scale path (skipped-with-warning until then).
+- **`curation_labels`** — an append-log of decisions (`approved` / `rejected` /
+  `held`); latest row wins per episode, history preserved. `forge curate`
+  selects with a WHERE filter, resolves near-dup losers under a **policy**
+  (`keep-higher-quality` / `keep-longer` / `keep-first`), and labels survivors
+  approved, losers rejected.
+- **`forge studio`** ([studio.py](studio.py)) renders a self-contained, themed
+  HTML app (Overview · Corpus · Dedup review · Snapshot) from real catalog data
+  and embedded video thumbnails — the design system matches the Forge Studio
+  mockup and the `forge visualize` dark theme. One shareable file, no server.
+
+The per-dataset `forge dedup` (perceptual-hash → cleaned dataset) is a separate,
+unchanged command; catalog-level dedup lives under `forge catalog dedup`.
+
 ## Example
 
 [`catalog_example_droid_100/`](catalog_example_droid_100/) is a small, ready-to-query
@@ -130,8 +155,9 @@ commands to reproduce it (from a local MinIO `s3://` source, or straight from
 forge catalog stats -c forge/catalog/catalog_example_droid_100
 ```
 
-## Not in Phase 1
+## Not yet
 
-Embeddings, dedup edges, curation labels, and snapshots (Phases 2–4);
+Snapshots + training-set export (Phase 4: `forge snapshot create/materialize`);
+frame/segment-level embeddings and Lance/ANN for >100k episodes (Phase 5);
 distributed execution, locking, compaction, and gc. The manifest convention is
 in place so gc becomes possible later, but it is not implemented.