Search- and selection-driven curation (curate --ids/--from, search --save) - #34
Merged
Merged
Conversation
…from, search --save)
Close the seam between semantic search / visual review and curation: `forge
curate` was SQL-`--where` only, so you couldn't curate "the top-K of a search"
or a hand-picked set from Studio. Now the selection can also be an explicit
`--ids` list or a `--from <selection.json>` file.
- curate() gains an `ids` selector — composes with `--where` (intersection) and
the dedup policy, and is filtered to episodes that actually exist so a typo
never creates a dangling label.
- `forge search --save sel.json` writes the result ids + provenance
({episode_ids, source}); `forge curate --from sel.json` applies them and
records the source in `labeled_by`. `--ids e1,e2` works too.
- Forge Studio's dedup tab gains a "Download decisions" button that exports the
same selection file (the rejected picks), so keep/reject review can write back
via `forge curate --from` instead of being copy-only.
Tests: curate by ids, id-filtering to existing episodes, ids∩where, and the
search --save → curate --from round-trip with provenance. No existing tests
change. Docs: README data-engine section, forge/catalog/README, CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the seam between semantic search / visual review and curation.
forge curatewas SQL---whereonly, so you couldn't curate "the top-K of a search" or a hand-picked set from Studio — those are selections (ranked lists / picked sets), not predicates. Now the selection can also be an explicit--idslist or a--from <selection.json>file.Changes
curate()gains anidsselector — composes with--where(intersection) and the dedup policy, and is filtered to episodes that actually exist, so a typo/stale id never creates a dangling label.forge search --save sel.jsonwrites the result ids + provenance ({episode_ids, source});forge curate --from sel.jsonapplies them and records thesourceinlabeled_byfor auditability.forge curate --from, instead of being copy-only.Why this shape
The two real requests ("top-K of a search", "what I picked in Studio") are explicit selections, so an id/selection-file path fits them natively and unblocks both in one small change. A SQL similarity predicate (the alternative) is a filter, not a ranker, still needs the model outside SQL for text, and does nothing for the hand-pick case — so it's a bigger lift that solves less. Provenance on the label (
labeled_by= the search/source) keeps curation auditable despite not being a self-documenting SQL rule.Doing this before Phase 4 (snapshots) matters: snapshots filter on
curation_labelsregardless of how they were produced, so search/Studio-curated sets become exportable too, not just SQL-predicate ones.Testing
5 new tests (curate by ids, id-filtering to existing episodes,
ids ∩ where, and thesearch --save → curate --fromround-trip with provenance); Studio template JS re-validated. No regressions.🤖 Generated with Claude Code