Skip to content

Commit 03a6a62

Browse files
committed
skill-eval: split the four-file family by function owner and drop the residual (S01)
Move skill-eval diagnostic, engineering, evaluation, and improvement functions into Benchmark rules, Harness execution/sandbox, and Apps composition. Delete the harness.skill_evaluation to knowledge_operations residual and its gate registration. Public entry signatures and trial-record bytes stay the same under the same fixtures. usage_totals remains the H maintenance implementation. SelectionPolicy thresholds move to Benchmark as a whole; choose stays in H. #338 leftover infra.artifacts call sites outside this family are not taken into S01 (pre-R01 H blank pack). overall isolation_complete stays false. Clean-history candidate: this commit transplants the accepted S01 tree onto current main 57f8e8a without replaying the local H02 / #397 working stack. The resulting tree is identical to 8b4a0cf (ordinary merge of 9218391 and 57f8e8a). Rollback remains at that original SHA. This commit is a local candidate only and is not a rewrite of the published branch. Refs #410 Agent-assisted-by: Claude Code / claude-fable-5-2 Agent-assisted-by: Grok Bot Signed-off-by: GUOGUO <xwdong1998@163.com>
1 parent 57f8e8a commit 03a6a62

52 files changed

Lines changed: 1200 additions & 816 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -242,8 +242,9 @@ See the [source audit and proposed remedies](docs/reviews/source-audit-ac70272.m
242242
for the 34 review points, qualifications and suggested acceptance checks.
243243

244244
Alongside the engine loop, the [offline improvement controller](docs/harness-improvement.md)
245-
composes the existing Skill Eval comparator and independent verifier. Its selection
246-
policy lives in `harness/improvement.py`; the tool adapter freezes snapshots and
245+
composes the existing Skill Eval comparator and independent verifier. Its acceptance
246+
rules live in `benchmark/improvement.py` and its search progression in
247+
`harness/improvement.py`; the tool adapter freezes snapshots and
247248
carries the total budget across probes, rerun and holdout. The engine diagrams above
248249
continue to describe the engine path; this additional lane does not provide engine
249250
performance evidence or an autonomous model-backed optimizer.

‎docs/architecture.md‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,17 @@ src/inferrsi/
152152
benchmark/
153153
registry.py Generic task resource discovery
154154
tasks/ Task-specific definitions, workloads and assessment
155+
skill_evaluation.py Skill trial check validity, verdict and condition statistics
156+
skill_eval_diagnostic.py Synthetic KV admission diagnosis task and grader
157+
skill_eval_engineering.py Serving-parser task, public vectors and vector judgment
158+
improvement.py Improvement-step thresholds and acceptance rules
159+
apps/
160+
api.py Python entry points for assembling and publishing Apps operations
161+
archive.py Publish and read the standardized Apps output in a workspace
162+
result.py Assemble and check the AppsResultV1 output envelope
163+
skill_evaluation.py Composed bundle export, trial run and summary
164+
skill_eval_engineering.py Composed artifact check: task rules around sandbox replay
165+
improvement.py Composed assessment, selection, rerun and holdout acceptance
155166
runspec.py, contracts.py Immutable experiment contract and compatibility export
156167
catalog.py Knowledge search and cited context
157168
skill_catalog.py Packaged/default or custom skill metadata/body/resource resolution
@@ -189,8 +200,10 @@ src/inferrsi/
189200
task_protocol.py Freeze supplied task resources and load their protocol
190201
optimization.py Current-Agent round budget, records and recovery
191202
usage.py Shared observed-token normalization and totals
192-
improvement.py Offline quality-first selection and usage accounting
193-
skill_evaluation.py Skill trial/client interface and independent grading
203+
improvement.py Offline search progression over injected assessments
204+
skill_evaluation.py Skill client adapter, trial execution and retention; injected rules
205+
artifact_sandbox.py Artifact capture and bubblewrap replay; no vectors or verdicts
206+
atomic_json.py Harness-owned atomic JSON evidence writes
194207
runner.py One AB/BA loop; execute/snapshot adapter hooks
195208
evaluation.py Performance/quality/source-integrity adjudication
196209
reporting.py HTML evidence report

‎docs/contracts/current-design.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ Decision: see [`schema-and-semantics.md`](schema-and-semantics.md). Draft 2020-1
153153

154154
## D018 — first-wave gate policy detail
155155

156-
Decision: [`gate-policy-runstore-and-first-prs.md`](gate-policy-runstore-and-first-prs.md) §B: exact seven leaf→module allowances, shape checks, transitive-closure checks, the registered residual `harness/skill_evaluation.py → knowledge_operations → knowledge.*` edge, the staged isolation test, and the rule that a non-empty allowlist keeps `overall isolation_complete=false`.
156+
Decision: [`gate-policy-runstore-and-first-prs.md`](gate-policy-runstore-and-first-prs.md) §B: exact seven leaf→module allowances, shape checks, transitive-closure checks, the registered residual `harness/skill_evaluation.py → knowledge_operations → knowledge.*` edge, the staged isolation test, and the rule that a non-empty allowlist keeps `overall isolation_complete=false`. The skill-evaluation split (#410) later removed the registered residual edge and its exemption.
157157

158158
## D019 — first PRs and numbering
159159

‎docs/contracts/function-owners.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,10 @@ Status: **Proposed**. Author decisions under the user's delegation, based on the
2121
| `catalog.repository_pack`/`REPO_TYPES` | D020 neutral raw-data technical leaf; target interpretation stays H | K does not expose the pack reader publicly; `infra` changes only its same-leaf import; seven K bodies/root entries/tests move together |
2222

2323
Any later split of a mixed function is accepted by real pytest/host/wheel/record comparison in the corresponding PR; none of this delays today's contract or gate-policy start. Existing external public composed entries keep their original semantics and paths per D018/D022, and an owner change cannot be used to delete an old promise.
24+
25+
## Implemented rows
26+
27+
The skill-evaluation split (#410) implements the `skill_evaluation`, `skill_eval_diagnostic`, `skill_eval_engineering` and `improvement` rows. H keeps `harness/skill_evaluation.py`, `harness/artifact_sandbox.py` and `harness/improvement.py`; B holds `benchmark/skill_evaluation.py`, `benchmark/skill_eval_diagnostic.py`, `benchmark/skill_eval_engineering.py` and `benchmark/improvement.py`; Apps composes them in `apps/skill_evaluation.py`, `apps/skill_eval_engineering.py` and `apps/improvement.py`. Two points follow the tree as it stands after #338, not the row text:
28+
29+
- `usage_totals` stays in `harness/usage.py`, because `harness/optimization.py` consumes it for budget accounting. B `assessment` receives the baseline and candidate totals from Apps and does not import H.
30+
- `SelectionPolicy` holds only the two thresholds, so it lives whole in B. H `choose` receives the assessments, the validity block and the composition flag; it holds no threshold and no policy class.

‎docs/contracts/gate-policy-runstore-and-first-prs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ The symbol inventory is AST-derived and does not claim to have found every third
3939
1. The default cross-core prohibition stays. Only the seven leaf→exact-K-module edges above may serve as migration-compatibility exceptions; a module prefix does not authorize adjacent new modules.
4040
2. Shape checks allow only a module docstring, a `future` import, the listed `ImportFrom` lines (alias only as `name as name`), and an `__all__` constant string list. No business definitions, other imports, assignment side effects, or dynamic module hooks.
4141
3. The static graph resolves absolute/relative imports, `from package import name`, and known export edges, and builds the transitive closure. Any target K/B/H core API entry that reaches another core, Apps, or a legacy facade fails. The D020 neutral technical leaf is not a legacy facade and must be independently verified to contain only raw data reading/data constants and no core imports; "only indirectly, so it passes" is not allowed. Target H entry points are the new `harness.api`/`execution_service`/`run_store`; the current mixed `harness` does not automatically count as an isolated core.
42-
4. The existing `harness/skill_evaluation.py → knowledge_operations → knowledge.*` edge is a registered **exact residual legacy dependency**: retained, and it keeps `overall isolation_complete=false`. It is not erased from the report and no new/expanded H bypass is permitted. It is not an exception available to a target H new entry.
42+
4. The existing `harness/skill_evaluation.py → knowledge_operations → knowledge.*` edge is a registered **exact residual legacy dependency**: retained, and it keeps `overall isolation_complete=false`. It is not erased from the report and no new/expanded H bypass is permitted. It is not an exception available to a target H new entry. The skill-evaluation split (#410) later removed this edge and its exemption; [`../module-layout.md`](../module-layout.md#first-wave-compatibility-exceptions-proposed) records the current rule.
4343
5. `test_clean_scan_is_not_isolation` becomes a staged-model test. It does not permanently assert that all three package directories are absent. It verifies that `overall` is still false when any target package, any compat exception, or an old H→model/infra transitive residual is missing. Green is not manufactured by deleting assertions.
4444
6. A non-empty allowlist forces `overall isolation_complete=false`; "new core business closure isolated" is reported as a separate verifiable result and cannot impersonate isolation of the whole historical namespace.
4545
7. New Apps entries call core public APIs only and never pass through old root shells; cores never import Apps. If a dynamic import is genuinely needed later, it must declare exact static edges and get extra review; in the first wave all compatibility shells prohibit it.

‎docs/contracts/isolation-errors-and-cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Only the later resource slice K02 moves Knowledge assets to `knowledge/data`, an
4242

4343
### 4. Gate and sequencing
4444

45-
P0b must accept the policy of **exactly seven leaves + catalog's two-target symbol routes + a business-free technical leaf + infra importing the neutral leaf directly** before K01 is allowed. Apart from root catalog's second, neutral-leaf export, the six leaves export only their same-named K modules; catalog's K export is also limited to the named symbols. No new K entry may reach the legacy facade / H / B / Apps. The new infra path may not reach K / H / B / Apps. The old `harness.skill_evaluation` → root KO → new K path is registered residue; whole-isolation is false, and it is not given to new H to use.
45+
P0b must accept the policy of **exactly seven leaves + catalog's two-target symbol routes + a business-free technical leaf + infra importing the neutral leaf directly** before K01 is allowed. Apart from root catalog's second, neutral-leaf export, the six leaves export only their same-named K modules; catalog's K export is also limited to the named symbols. No new K entry may reach the legacy facade / H / B / Apps. The new infra path may not reach K / H / B / Apps. The old `harness.skill_evaluation` → root KO → new K path is registered residue; whole-isolation is false, and it is not given to new H to use. The skill-evaluation split (#410) later removed that path.
4646

4747
There must be a synthetic graph that does not depend on whether the real target directories already exist: allow the exact leaves; reject adjacent K targets; reject K → old shell → K; reject new H → shell → K; check that Apps → shell does not meet the new Apps rules; reject a hidden core import inside `_repo_pack_data`; infra → root catalog → K must be caught as a negative case, and infra → the neutral leaf must pass. A non-empty allowlist or residue makes overall false, even if a local new-core graph is isolated. The scan scope is explicitly the `src` transfer graph. `tools` / `tests` are checked separately as registered consumers; it must not be said that the existing gate already scans them. Undeclared dynamic edges fail closed.
4848

‎docs/harness-improvement.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,10 @@ flowchart TB
4444
E["Engine RunSpec → AB/BA → quality/performance evidence"]
4545
```
4646

47-
`src/inferrsi/harness/improvement.py` owns selection and usage accounting.
47+
`src/inferrsi/benchmark/improvement.py` owns the thresholds and acceptance rules,
48+
`src/inferrsi/harness/improvement.py` the search progression and
49+
`src/inferrsi/harness/usage.py` usage accounting; `src/inferrsi/apps/improvement.py`
50+
composes them.
4851
`tools/skill_eval_improve.py` owns freezing, declarative snapshot materialization
4952
and orchestration through `tools/skill_eval_compare.py`. There is no second
5053
subprocess runner. The engine lane and its acceptance contract remain separate.

‎docs/local-skill-evaluation.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@ flowchart LR
2020

2121
`tools/skill_eval_compare.py` still owns prepare/execute/report and paired
2222
snapshots. `tools/skill_eval_local.py` adds manual stage/capture, without launching
23-
an agent. `src/inferrsi/harness/skill_eval_engineering.py` owns the one task and
24-
19 public behavior probes for that one task. `skill_evaluation.py` separates the single writable
25-
artifact from frozen inputs. The existing cloud read/submit route is unchanged;
23+
an agent. `src/inferrsi/benchmark/skill_eval_engineering.py` owns the one task, its
24+
19 public behavior probes and their judgment. `src/inferrsi/harness/artifact_sandbox.py`
25+
replays the retained artifact under bubblewrap and separates the single writable
26+
artifact from frozen inputs; `src/inferrsi/apps/skill_eval_engineering.py` composes the two.
27+
The existing cloud read/submit route is unchanged;
2628
it cannot perform this artifact-editing task and is rejected for this adapter.
2729

2830
```mermaid
@@ -56,7 +58,7 @@ Do not add a provider `client`, custom `schema` or `grader`. Generate the case
5658
without silently changing its contract:
5759

5860
```python
59-
from inferrsi.harness.skill_eval_engineering import REQUEST
61+
from inferrsi.benchmark.skill_eval_engineering import REQUEST
6062
from inferrsi.infra.artifacts import write_json
6163

6264
write_json("cases.json", [{

‎docs/module-layout.md‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,9 @@ Several flat modules remain shared seams until their slice starts.
6565
| `harness/optimization_session.py`, `harness/project.py`, `harness/task_protocol.py` | Shared preparation, execution and retained-result assessment | CLI supplies task resources and knowledge access; ordinary project lifecycle does not import Benchmark |
6666
| `harness/optimization.py`, `harness/candidates.py`, `harness/usage.py` | Shared project and packaged-task lifecycle | Injected execution/assessment; declared file/directory candidates, selection direction and exact/up-to rounds |
6767
| `harness/agent_sessions.py` | Native host handoff records | Freezes requests and retains native IDs and exported observations; the initiating Agent calls host tools |
68+
| `harness/skill_evaluation.py`, `harness/artifact_sandbox.py`, `harness/improvement.py`, `harness/atomic_json.py` | Skill-evaluation host adapter, trial execution and retention, sandbox replay, search progression | Verdict rules and task definitions arrive as injected values; no Model, Infra, Knowledge, Benchmark or Apps import |
69+
| `benchmark/skill_evaluation.py`, `benchmark/skill_eval_diagnostic.py`, `benchmark/skill_eval_engineering.py`, `benchmark/improvement.py` | Skill-evaluation criteria, task definitions and acceptance rules | Pure judgments over retained facts; no Harness import and no process start |
70+
| `apps/skill_evaluation.py`, `apps/skill_eval_engineering.py`, `apps/improvement.py` | Composed skill-evaluation entries | Join Knowledge reads, Harness execution and Benchmark rules behind the signatures that `harness/` held before the split |
6871
| `knowledge_operations.py`, `knowledge_store.py`, `catalog.py`, `research.py`, `skill_catalog.py`, `_skill_selection.py`, `_source_snapshot.py` | Static compatibility shells since K01; bodies in `knowledge/` | Interfaces unchanged |
6972
| `harness/` | Mixed legacy tree, not target isolation | Not redefined |
7073
| `evaluation.py`, `model/validators/` | Benchmark-leaning checks | Not moved |
@@ -132,8 +135,7 @@ A clean result means no new cross-core import of those names. It does not mean
132135
the modules are isolated. Isolation stays false while any one of these holds:
133136
`knowledge/`, `benchmark/`, or `apps/` is absent or has no scannable
134137
module; the compatibility allowlist
135-
is non-empty; the registered residual `harness.skill_evaluation` → root
136-
`knowledge_operations` exists; or `harness/**` still imports Model or Infra,
138+
is non-empty; or `harness/**` still imports Model or Infra,
137139
directly or through another module. The three packages may appear in stages.
138140
A clean graph for the new cores is reported separately as "new core business
139141
closure isolated" and never sets overall `isolation_complete` to true.
@@ -191,8 +193,8 @@ The leaf's own import rule and `REPO_TYPES` object identity govern it.
191193
- **Closures.** A Knowledge module reaches nothing outside Knowledge except
192194
the data leaf and `inferrsi._json_v1`, the canonical encoder in the P0a
193195
design. Benchmark modules and every `inferrsi.harness.*` module reach no
194-
other core, no Apps module, and no old root leaf, except the registered
195-
residual below. Apps reaches no old root leaf. `inferrsi.infra`
196+
other core, no Apps module, and no old root leaf. Apps reaches no old root
197+
leaf. `inferrsi.infra`
196198
reaches no Knowledge, Harness, Benchmark, or Apps module, so `infra` → root
197199
`catalog` → Knowledge fails while `infra` → the data leaf passes.
198200
- **New H entries.** The registered entry modules are
@@ -203,14 +205,15 @@ The leaf's own import rule and `REPO_TYPES` object identity govern it.
203205
paths from labels. Any `inferrsi.harness.*` module that exists is still
204206
checked against the closure rule above. The current mixed `harness` tree is
205207
not automatically an isolated core.
206-
- **Residual.** The registered `harness.skill_evaluation` → root
207-
`knowledge_operations` edge stays listed and keeps overall isolation false.
208-
A second direct harness importer of that target fails, and a new H entry
209-
may not use the residual even through `skill_evaluation`. A mixed non-entry
210-
harness module may still reach that target only through the residual
211-
importer; that existing leak is not a new exception. The exemption is
212-
reachability after removing only that exact residual edge: another path
213-
to the same target is still a violation, regardless of traversal order.
208+
- **No residual.** The skill-evaluation split (#410) moved `export_bundle` to
209+
Apps and removed the registered `harness.skill_evaluation` → root
210+
`knowledge_operations` edge together with its exemption. Every harness
211+
module is walked on the full graph. A harness module that reaches an old
212+
root leaf fails, and so does each harness module that reaches the leaf
213+
through it. The report lists every remaining `harness` → Model or Infra
214+
path in `harness_model_infra`; the split modules `harness.skill_evaluation`,
215+
`harness.improvement`, `harness.artifact_sandbox` and `harness.atomic_json`
216+
have none.
214217

215218
The manifest records the base commit, each old leaf's file hash at that
216219
commit, export names, audited definitions, consumer evidence, catalog's two

‎docs/skill-eval-cloud-replay.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,11 @@ The UTF-8 JSON object has exactly these keys (duplicate keys are rejected):
2323
- `task_id`: 1–128 ASCII letters, numbers, underscores, dots or hyphens; an opaque
2424
provenance identifier, not a path or executable task selector.
2525
- `request_sha256`: SHA256 of the trusted `engineering.REQUEST` UTF-8 bytes.
26-
- `grader_sha256`: SHA256 of the trusted `skill_eval_engineering.py` file bytes.
26+
- `grader_sha256`: SHA256 of the trusted
27+
`src/inferrsi/benchmark/skill_eval_engineering.py` file bytes: the task, its
28+
vectors and their judgment. The replay sandbox
29+
(`src/inferrsi/harness/artifact_sandbox.py`) is not part of this digest; the
30+
receipt's qualification records the exact revision that ran it.
2731
- `inputs_sha256`, `protocol_sha256`: lowercase 64-character provenance digests.
2832
These are retained, **not independently authenticated by this receipt**.
2933
- `artifacts`: exactly two objects, ordered baseline then candidate, each with

0 commit comments

Comments
 (0)