Skip to content

Commit d5f6e02

Browse files
committed
benchmark: present the B01 tree as a single commit on current main
History-only rewrite for PR 422. This commit has a single parent (current main) and does not rebase or replay unsquashed #397 / H02 review-stack commits. Method: merge-tree(current main, db924ea); not the raw db924ea tree Official head at preparation: db924ea Current main at preparation: 57f8e8a No file-content change relative to that method's resulting tree. Local candidate only; no push. Refs #414 Agent-assisted-by: Grok Bot (decision record and comment fixes) Agent-assisted-by: Claude Code / claude-fable-5-2 [effort=xhigh] (prior architecture review) Agent-assisted-by: Claude Code / claude-fable-5-2 [effort=xhigh] (architecture review and draft) Agent-assisted-by: Grok Bot (architecture decision integration) Agent-assisted-by: Grok Bot Agent-assisted-by: Codex (author) Agent-assisted-by: Claude Code / claude-fable-5-2 effort=xhigh (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
1 parent 57f8e8a commit d5f6e02

52 files changed

Lines changed: 1335 additions & 112 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,24 @@
22

33
## [Unreleased]
44

5+
### Changed
6+
7+
- The old `run` chain is split by owner. The CLI, its exit codes 0/1/2 and the
8+
`evaluation.json` bytes are unchanged. Root `inferrsi.evaluation`,
9+
`inferrsi.execution` and `inferrsi.reporting` keep working as forwarders.
10+
- Moved import paths, with no forwarder left under `inferrsi.harness`:
11+
- `inferrsi.harness.evaluation` → `inferrsi.apps.legacy.evaluation` (`evaluate` with a
12+
`RunSpec`) and `inferrsi.benchmark.legacy_run_v0` (the gates).
13+
- `inferrsi.harness.reporting` → `inferrsi.apps.legacy.reporting`.
14+
- `inferrsi.harness.scenarios.media` → `inferrsi.benchmark.sampling`.
15+
- `inferrsi.harness.runner.run_experiment` now requires its `execute`, `snapshot`,
16+
`assess` and `report` ports. Use `inferrsi.apps.legacy.run.run_experiment` for the
17+
composed old run with the previous defaults.
18+
- `inferrsi.harness.scenarios.external.execute_trial` takes `metric=` (a complete
19+
validation function) in place of `metric_reader=`. Use
20+
`inferrsi.apps.legacy.run.execute_trial` to keep `metric_reader=` with the common
21+
metric contract applied.
22+
523
## [0.1.0] — Developer Preview
624

725
Pre-release only. v0.1.0 is a planned Developer Preview. The `v0.1.0` Git tag is not published yet, and this note does not publish a tag or a GitHub Release. There is no API stability promise. The package is not on PyPI. After the tag is published, install from that Git tag. Licensed under Apache-2.0 (`LICENSE`).

‎docs/architecture.md‎

Lines changed: 26 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,12 @@ src/inferrsi/
152152
benchmark/
153153
registry.py Generic task resource discovery
154154
tasks/ Task-specific definitions, workloads and assessment
155+
legacy_run_v0.py Old run gates: metric contract, summaries and verdict
156+
sampling.py Warmup seeds, first-response retention, sample/artifact groups
157+
apps/legacy/
158+
run.py Old run composed from Harness, Benchmark and the report
159+
evaluation.py Old evaluate entry: RunSpec to the Benchmark gates
160+
reporting.py HTML evidence report
155161
runspec.py, contracts.py Immutable experiment contract and compatibility export
156162
catalog.py Knowledge search and cited context
157163
skill_catalog.py Packaged/default or custom skill metadata/body/resource resolution
@@ -191,12 +197,10 @@ src/inferrsi/
191197
usage.py Shared observed-token normalization and totals
192198
improvement.py Offline quality-first selection and usage accounting
193199
skill_evaluation.py Skill trial/client interface and independent grading
194-
runner.py One AB/BA loop; execute/snapshot adapter hooks
195-
evaluation.py Performance/quality/source-integrity adjudication
196-
reporting.py HTML evidence report
200+
runner.py One AB/BA loop; execute/snapshot/assess/report ports injected
201+
atomic_json.py Harness-owned atomic JSON evidence writes
197202
scenarios/
198-
external.py External benchmark/validator commands; metric-reader hook
199-
media.py Warmup seeds, first-response retention, sample/artifact groups
203+
external.py External command trials; metric validation injected
200204
model/
201205
protocols/
202206
metrics.py Finite positive dotted-key metric contract
@@ -217,10 +221,10 @@ src/inferrsi/
217221
repo_packs/ Contribution descriptors and versioned tuning knowledge
218222
knowledge_packs/ Dependency and peer knowledge, preserving existing tiers
219223
skill_library/ Complete categorized English skills and resources
220-
execution.py Compatibility entrypoint -> harness.runner
221-
evaluation.py Compatibility exports -> harness.evaluation
224+
execution.py Compatibility entrypoint -> apps.legacy.run
225+
evaluation.py Compatibility exports -> apps.legacy.evaluation
222226
inspection.py Compatibility entrypoint -> infra.inspection
223-
reporting.py Compatibility export -> harness.reporting
227+
reporting.py Compatibility export -> apps.legacy.reporting
224228
examples/ Current Qwen runner, image validator and v4 adapter
225229
tests/fixtures/legacy/ Frozen historical validation dependencies for regressions
226230
```
@@ -241,15 +245,16 @@ synthetic contract fixtures, not accelerator measurements.
241245

242246
### Compose a new scenario
243247

244-
`harness.runner.run_experiment(spec, environment, *, execute, snapshot)` accepts
245-
small callable adapters. For a new external benchmark format, reuse the standard
246-
executor and only replace metric interpretation:
248+
`apps.legacy.run.run_experiment(spec, environment, *, execute, snapshot)` composes
249+
the old run and accepts small callable adapters. It passes them, the Benchmark v0
250+
gates and the report to `harness.runner.run_experiment`, whose four ports
251+
(`execute`, `snapshot`, `assess`, `report`) have no defaults. For a new external
252+
benchmark format, reuse the standard executor and only replace metric interpretation:
247253

248254
```python
249255
from functools import partial
250256

251-
from inferrsi.harness.runner import run_experiment
252-
from inferrsi.harness.scenarios.external import execute_trial
257+
from inferrsi.apps.legacy.run import execute_trial, run_experiment
253258
from inferrsi.model.protocols.metrics import read_metric
254259

255260

@@ -269,10 +274,12 @@ The standard executor retains timeout, cancellation, logs, result paths and chil
269274
cleanup. A custom `execute(command, cwd, trial_dir, deadline, metric_key)` must
270275
preserve that trial record contract. Prefer composing `execute_trial` over
271276
reimplementing process lifetime. Metric readers convert a result document into a
272-
float; `execute_trial` then applies the one finite, positive metric contract
273-
(`model.protocols.metrics.validate_metric`) to whatever the reader returns, so a
274-
reader that yields `0.0`, a `bool` or a non-finite value records a metric error
275-
rather than entering the evaluation.
277+
float; `benchmark.legacy_run_v0.trial_metric` then applies the one finite, positive
278+
metric contract (`model.protocols.metrics.validate_metric`) to whatever the reader
279+
returns, so a reader that yields `0.0`, a `bool` or a non-finite value records a
280+
metric error rather than entering the evaluation. The Harness executor
281+
`harness.scenarios.external.execute_trial` holds no metric rule: a caller that
282+
passes a metric key also passes `metric=`, its validation function.
276283

277284
Run state is written by the termination path, never assumed. `status.json`
278285
reports `completed` only after the final artifacts were written;
@@ -287,7 +294,8 @@ fingerprint relevant source, including changes between individual commands; it
287294
must not claim availability without a trustworthy identity. Git remains the
288295
default. Alternative adapters must define their own exclusion semantics.
289296

290-
Image campaigns can compose `ImageProtocol.request` with `collect_samples`.
297+
Image campaigns can compose `ImageProtocol.request` with
298+
`benchmark.sampling.collect_samples`.
291299
The protocol receives model name, dimensions and sampling parameters from the
292300
campaign. The collector owns warmups, first-response retention, groups and
293301
artifact hashes; it accepts a request callback and file suffix, so it has no

‎docs/assets/modular-directory-map-en.svg‎

Lines changed: 10 additions & 10 deletions
Loading
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# D008 wave-2 ownership decision
2+
3+
Status: **Adopted for the five entries below**. Decision date: **2026-11-14 Asia/Shanghai**.
4+
5+
## Authority and scope
6+
7+
**RSI-master** names the architecture-coordination role delegated by the requesting maintainer, **Dong1017 (xinwen dong)**, to the assistant coordinating these changes. It is not an additional GitHub account, an independent reviewer, an extra approver, or a statement of legal repository ownership. Under that delegation, RSI-master makes and records the architecture choice rather than asking reviewers to select an owner.
8+
9+
This file is the citable record of the five decisions adopted under that delegation for Issues [#414](https://github.com/InferMatrixLab/InferRSI/issues/414) (B01 / PR #422) and [#413](https://github.com/InferMatrixLab/InferRSI/issues/413) (OBS / PR #425). It makes no blanket acceptance claim for the other proposed entries in [function-owners.md](function-owners.md).
10+
11+
Architecture adoption is **not** reviewer `APPROVE`, implementation acceptance, permission to merge, a gate-policy exception, or evidence that tests passed. Reviewers verify the boundary, correctness and evidence against the actual head; their independent PR decision remains separate.
12+
13+
## Benchmark sampling
14+
15+
**Adopted:** `collect_samples` belongs to **Benchmark sampling**. Benchmark owns sampling order, timing records and artifact hashes. Request transport remains the caller-injected `request(item, path)` callback; it does not become a Harness or model dependency of Benchmark.
16+
17+
Issue #414 explicitly assigns this sampling to Benchmark. The implementation is `benchmark/sampling.py`; the artifact hasher equivalence is pinned by `test_sampling_sha256_matches_the_infra_hasher`. Retaining existing protocol output does not confer ownership of UI presentation or of a caller-owned record's lifecycle.
18+
19+
## Snapshot port
20+
21+
**Adopted:** **Harness** owns the required keyword-only `snapshot(repo, exclude) -> dict` port contract; **Apps legacy** owns the binding; **infra/targets** owns the Git identity implementation. Harness has **no concrete default** for this port and does not hold a copy of the target implementation.
22+
23+
This preserves the Issue #414 split: execution facts in H, composition in Apps, implementation behind an injected port. It does not restore an H-to-infra edge. Existing compatibility disclosure for out-of-tree direct H callers remains applicable; this record does not erase it.
24+
25+
## Dashboard workflow
26+
27+
**Adopted:** `command_state`, `run_workflow` and `validation_progress` belong to **Apps views**. They project caller-loaded records into display state/counters. Their output is not a Benchmark verdict or a claim-acceptance decision.
28+
29+
## Profile records and overview
30+
31+
**Adopted:** profile attachment, factual record encoding, bounded single-trace overview generation, fixed-byte integrity checks and digest-based record reads belong to **Harness records**. UI projections and route decoration belong to **Apps views**. In particular, constructing an `events_url` for the live server is not an H responsibility.
32+
33+
This explicitly refines Issue #413's broad wording that dashboards/timelines/templates move to Apps: a persisted factual record and its bounded overview remain H record content, while the presentation and URL are Apps concerns. This is an explicit split, not a claim that the issue wording already made the distinction verbatim. Bounding events/lanes within one trace does not authorize H to rank candidates, runs or claims or to generate verdicts.
34+
35+
The route decoration is to be moved into Apps in PR #425 while preserving stored bytes, digest verification and the external view response. This record does not by itself assert that the implementation or validation has finished.
36+
37+
## Pre-profile searches
38+
39+
**Adopted:** `pre_profile_searches` belongs to **Apps views**. It combines profile facts with an existing search/ledger read model for display. Its annotations are not a new claim-evidence verdict.
40+
41+
## Integration constraints
42+
43+
The four-module dependency direction remains unchanged: Apps assembles the peers; H records do not import K or Apps. Pure knowledge context assembly belongs to K; explicit capture orchestration belongs to Apps; H retains actual delivered facts/bytes and their integrity. The #428 capture/navigation features must be preserved when OBS absorbs the newer main layout, rather than restoring an `observability` forwarding package or dropping retrieval metadata.
44+
45+
Implementation acceptance remains in the governing issues: old-run behavior and byte evidence for #414; fixed-record CLI/offline/live/trace behavior, H closure, resource packaging and disappearance of `observability/` for #413. `overall isolation_complete` remains false. None of those evidence requirements is waived here.

0 commit comments

Comments
 (0)