Skip to content

Commit 2cbfab6

Browse files
authored
Standardize evidence archives and compact duplicate model records (#472)
* Standardize evidence archives and remove redundant scan data Define the core bundle and v2 manifest contract, normalize frozen pilot layouts, and verify external archive bytes, members and replay outputs explicitly. Keep the inventory authoritative through references and generate the pending budget checker input on demand. Replace the full generated history scan with a pinned, hash-checked source reference and retain scanner coverage on synthetic Git history. Agent-assisted-by: Codex (implementation and validation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * Refresh the installed resource count after the knowledge update The current base added the Ascend model-adaptation source notes, increasing the complete Knowledge and Harness resource inventory from 927 to 928. Keep the fixed count check aligned with that tree while preserving all per-file digest comparisons. Agent-assisted-by: Codex (implementation and validation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * Archive duplicate model evidence with reproducible v2 views Preserve both original snapshots and frozen checks while regenerating compact summaries and observation indexes. Agent-assisted-by: Codex (implementation and validation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> --------- Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
1 parent f7c0390 commit 2cbfab6

507 files changed

Lines changed: 3411 additions & 102334 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/contributing.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,11 +127,18 @@ research spikes.
127127
```text
128128
evidence/<campaign-id>/
129129
plan/ Hypothesis, RunSpec, frozen gates and environment
130-
results/ Raw measurements, outputs, profiler data and verdict rationale
130+
results/ Metric summaries, complete trial index, archive references and rationale
131131
patch/ Candidate diff or exact revision pointer
132132
validator/ Frozen validation code, versions and inputs
133133
```
134134

135+
Follow the [archiving contract](evidence-archiving.md) and
136+
[bundle template](../evidence/_template/README.md). Keep large raw measurements,
137+
logs, profiler data and generated inventories in versioned external archives.
138+
The campaign's `results/artifacts.json` owns metadata; index every attempt and
139+
retain executable replay and comparison criteria. Existing frozen evidence is
140+
not rewritten to match the template.
141+
135142
| Verdict | Meaning |
136143
| --- | --- |
137144
| PASS | Pinned comparison, complete evidence and every required gate met |

‎docs/evidence-archiving.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# Experiment evidence and archive verification
2+
3+
Implements the inclusion/template/tooling slice of the
4+
[scoped archiving decision](https://github.com/InferMatrixLab/InferRSI/issues/464#issuecomment-6913744401).
5+
The RFC's completion and campaign acceptance remain separate owner decisions.
6+
7+
## Repository contents
8+
9+
Retain implementation, necessary tests, goals, conclusions, limitations, key
10+
metrics, effective configuration, input/validator identities, a complete trial
11+
index and verification entry points. Start from [the bundle template](../evidence/_template/README.md).
12+
Small test fixtures stay only when their consumers and necessity are clear.
13+
14+
Archive per-request results, per-sample scores, profiler output, logs, media,
15+
complete source inventories and generated process reports outside Git. Classify
16+
JSON by its purpose; changing directories, minifying, splitting or committing
17+
compressed archives does not justify retaining large generated artifacts.
18+
Do not retain temporary outputs with no review or reproduction value.
19+
20+
Generate summaries and trial indexes from the original control/evidence records.
21+
Record all attempts, including failed, regressing, interrupted and repeated
22+
trials. Retain sample/failure counts, repetitions, variability, metric definitions
23+
and incomplete validation. Preserve frozen raw bytes and historical conclusions.
24+
25+
## Archive authority and schema
26+
27+
Each campaign's `results/artifacts.json` owns its archive metadata. The inventory
28+
references it via `schema: archive-sha256`, `field: archive`, and the expected
29+
local artifact path. Do not copy URL/hash/bytes/owner into inventory entries.
30+
The offline inventory rejects such duplicate `external_assets` metadata.
31+
32+
New documents use **schema_version 2**, illustrated by
33+
[artifacts.example.json](../evidence/_template/results/artifacts.example.json).
34+
Replace placeholders with measured values, including integer byte/file counts.
35+
The executable format contract is `normalize_manifest` in
36+
[tools/evidence_archives.py](../tools/evidence_archives.py).
37+
38+
| Field | Contract |
39+
| --- | --- |
40+
| `archive.id`, `file` | Stable version identity and basename |
41+
| `archive.url` | Credential-free HTTPS download URL; no query or fragment |
42+
| `archive.sha256`, `bytes` | Lowercase SHA-256 and positive compressed byte count |
43+
| `archive.format` | `tar+gzip`; regular nonsparse files only |
44+
| `archive.source_revision` | Immutable 40–64 hexadecimal source revision |
45+
| `archive.root` | Safe relative member prefix, or empty for the whole archive |
46+
| `archive.files`, `uncompressed_bytes` | File count and sum of file bytes under root |
47+
| `archive.key_files` | Named archive-relative files or directory prefixes ending `/` |
48+
| `archive.retention_owner` | Named owner responsible while references remain |
49+
| `verification.inputs` | Archive-relative SHA-256 identities with at least control, configuration and validator roles |
50+
| `verification.command` | Reviewed replay argv, recorded but never executed by the generic tool |
51+
| `verification.trials`, `summary` | Paths relative to retained/regenerated bundle roots |
52+
| `verification.absolute_tolerance`, `relative_tolerance` | Finite nonnegative float-comparison tolerances; integers/structure exact |
53+
54+
Both frozen v1 pilots remain readable through the same normalization function:
55+
`tar.gz`/`tar+gzip`, `files`/`original_files`,
56+
`uncompressed_bytes`/`original_bytes`, and the two source-revision/key-file
57+
locations normalize into one view. Conflicting aliases fail validation. H3's
58+
`snapshot.json` is outside its original-file root; it is still inspected and
59+
counted against safety limits. V1 replay remains with each frozen campaign
60+
checker; normalization does not rewrite either pilot's bytes or interpretation.
61+
62+
## Publication and explicit verification
63+
64+
1. Generate the complete raw artifact locally. Review privacy before publication.
65+
2. Generate summaries and the index from those raw/control records; test coverage.
66+
3. Package regular files, publish a versioned Release asset, and record measured
67+
metadata. Controlled originals remain in authorized storage under the
68+
[privacy rules](privacy-security.md).
69+
4. Download again and verify the published bytes and contents with the explicit
70+
tool. Run the reviewed replay command and compare outputs.
71+
5. Submit the compact bundle, verification results and availability limits.
72+
73+
From the repository root (replace angle-bracket placeholders):
74+
75+
```bash
76+
# Offline metadata only. Never downloads or certifies content.
77+
python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json
78+
python tools/validate_evidence.py
79+
80+
# Explicit network operation, bounded bytes/hash/member verification.
81+
python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json --download
82+
83+
# Controlled storage: retrieve through its authorized client, then verify offline.
84+
python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json --archive <local-archive>
85+
86+
# Separately run the reviewed, pinned replay command from the manifest, then:
87+
python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json \
88+
--archive <local-archive> --recomputed <new-output-directory>
89+
```
90+
91+
The common tool authenticates compressed bytes before tar parsing, then streams
92+
members without filesystem extraction or archived code execution. It rejects
93+
absolute/traversing paths, duplicate names, links, devices and sparse members.
94+
Defaults cap compressed data at 256 MiB, expanded tar bytes (including metadata)
95+
at 512 MiB, files at 10,000, and each download/verification operation at 120
96+
seconds. Downloads have a 10-second socket timeout and permit only GitHub
97+
Release hosts and their HTTPS asset redirects. These are tool resource limits,
98+
not experiment acceptance gates. Larger archives need an explicitly reviewed
99+
tool configuration. Split campaigns only for meaningful storage reasons.
100+
101+
Failures distinguish metadata, unavailable/restricted access, bad bytes/hash,
102+
unsafe paths/members/redirects, resource limits, missing files, bad contents,
103+
trial mismatch and summary recomputation mismatch. Output always states which
104+
scope was checked. Metadata success or verified archive bytes alone leaves
105+
`recomputation: not_checked`; successful comparison reports `outputs_match`.
106+
The caller must review the replay's derivation and coverage; comparing two
107+
identical files cannot prove they came from independent recomputation.
108+
109+
Offline CI does not download, unpack or run a replay. A broken link records a
110+
verification gap without changing the historical verdict. Do not use expiring
111+
CI links or local paths as the only archive source. New bytes require a new
112+
version; keep the old digest/version retrievable. Redacted public copies have
113+
their own identity and explicit limits, never overwrite frozen originals.
114+
115+
## R6 reporting and exceptions
116+
117+
Existing R6 thresholds remain report-only: 512 KiB per `evidence/` file and
118+
2 MiB net evidence growth per PR. This change adds no blocking size gate.
119+
Reconsideration remains on or after 2026-12-02 under the existing decision.
120+
121+
[PR #328](https://github.com/InferMatrixLab/InferRSI/pull/328) owns the pending byte
122+
reporter. Its legacy input can be generated without maintaining duplicate
123+
metadata (after that checker is integrated):
124+
125+
```bash
126+
python tools/validate_evidence.py --budget-assets > <temporary-budget-view.json>
127+
python tools/check_evidence_budget.py --base <base-ref> --report-only \
128+
--inventory <temporary-budget-view.json>
129+
```
130+
131+
The generated view is disposable, not committed. #328's Release-only URL
132+
format check does not support controlled-store URLs; use the shared reader and
133+
an authorized local archive for those and report that compatibility gap. The
134+
view does not claim remote availability. Added/deleted/net byte reporting
135+
remains in #328's follow-up scope.
136+
137+
Use [the exception record](../evidence/_template/plan/storage-exception.md) when
138+
a specific oversized artifact must stay in Git. Record purpose, consumers,
139+
externalization obstacle, byte count, maintenance owner and explicit maintainer
140+
agreement. An exception record does not create automatic enforcement.
141+
142+
## Cleanup and acceptance
143+
144+
Before removing redundant generated data, identify its consumers and preserve
145+
an exact, retrievable source with digest/size and a concise summary. Convert
146+
behavioral tests to small synthetic inputs when they do not need a full source
147+
scan. Verify links and affected consumers after removal. This removes the file
148+
from the current tree, not existing Git history.
149+
150+
This workflow does not automatically migrate frozen merged campaign evidence.
151+
An explicitly requested, scoped cleanup may replace the current presentation
152+
only after every original file is preserved in a verified versioned archive;
153+
original frozen bytes, historical conclusions and Git history remain unchanged.
154+
For new/unmerged work, externalize before the required squash merge so large
155+
blobs do not enter main. The two pilot archive/replay checks and their
156+
independent reviews remain campaign-specific acceptance records. Completing
157+
this tool or removing redundant data neither establishes experiment PASS nor
158+
closes the parent RFC.
159+
160+
### Scoped legacy record projections
161+
162+
The [H3 four-device bundle](../evidence/h3-ref2va-910c-4logical-20261101-297/README.md)
163+
and [Qwen3.8 bundle](../evidence/qwen38-h20-fp8-single-request-vllm032/README.md)
164+
use v2 for their explicitly requested cleanup. Their full originals, including
165+
superseded cumulative checkpoints and repeated diagnostic code, remain in the
166+
archives. This is not blanket authorization to migrate other frozen campaigns.
167+
168+
`tools/rebuild_model_evidence.py` regenerates compact views from authenticated
169+
originals. `archive.key_files.snapshot` identifies the original per-file hash map;
170+
`archive.key_files.projection` identifies the declarative source/metric selection.
171+
Both and the replay code are pinned by `verification.inputs`. The replayer uses
172+
the common bounded reader to capture authenticated bytes, checks original frozen
173+
manifests, and requires its local implementation to match the pinned code. It
174+
never executes code from an archive and never overwrites an existing output.
175+
176+
Index coverage is deliberately labelled: every published result JSON document
177+
and named/nested attempt observation, including failures and cancellations.
178+
Identical observations of the same named attempt share one row and retain all
179+
source pointers; changed observations, retries and anonymous records remain
180+
distinct. `configuration.record_sha256` hashes canonical JSON for the source
181+
observation. Entry counts are not independent trial counts. Narrative-only
182+
history and unpublished controller records cannot become fully structured
183+
trial evidence through a storage migration; those limits remain explicit.
184+
185+
Declared sample distributions are recomputed; other frozen reported statistics
186+
and verdicts are labelled as recorded values. Replaying the projection does not
187+
rerun inaccessible quality checks, change gates or complete campaign acceptance.

‎docs/operations.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@ Effect classes used below:
4343
| Amend repository-owned knowledge | Scoped repository PR | — | Repository review and integration authority apply; this is not a runtime store Update |
4444
| Inspect/prepare experiments | CLI `doctor`; `plan` or `--dry-run` | Read-only inspection / preview | `doctor` may invoke bounded probes and does not certify target compatibility; `plan` previews a specification without persisting a campaign or authorizing execution |
4545
| Execute/inspect experiments | CLI `run`; `status`; `report` | Experiment-executing / read-only / report-generating | `run` executes caller-supplied commands and records evidence; `status` reads recorded progress; `report` generates HTML from recorded evidence |
46+
| Inspect/verify archived evidence | `python tools/verify_evidence_archive.py`; [archive contract](evidence-archiving.md) | Read-only by default; explicit network/local scratch with `--download` | Default validates metadata offline; `--archive`/`--download` authenticates and bounds archive reading; `--recomputed` compares separately generated v2 results; never executes archived code or changes verdicts |
47+
| Rebuild scoped legacy evidence views | `python tools/rebuild_model_evidence.py <manifest> --archive <path> --output <new-directory>` | CPU-only; creates a new output directory | Authenticates original bytes and pinned projection/code; no extraction, network, archived-code execution or campaign qualification |
48+
| Inspect evidence inventory / generate budget input | `python tools/validate_evidence.py [--budget-assets]` | Read-only | Offline layout/hash/schema checks; optional disposable #328 metadata view derived from campaign authority; no downloads |
4649
| Execute one ordinary local CPU command with a fixed record | Python `inferrsi.harness.api.Harness.allocate` / `execute` / `read` | Experiment-executing / record-writing | One explicit RunStore root and one trusted local CPU target; ordinary command mode only; a record holds execution facts, so exit 0 is not a verdict; no device, model or network fallback |
4750
| Visualize a recorded run or campaign | CLI `dashboard`; [dashboard](dashboard.md) | Report-generating / local read-only serving | Writes an offline HTML snapshot or serves an optional live view on loopback; does not run an agent or validate a claim |
4851
| Capture a knowledge context | CLI `campaign capture-context`; [offline dashboard](dashboard.md#record-the-optimization-story-campaign-trace) | Local evidence-writing | Runs `collect` with a context budget and keeps the full envelope and resource hashes in a new `knowledge/<id>.json`; proves what was prepared, not what a model read |
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Historical source-scan inventory
2+
3+
The complete generated inventory is retained at its [exact source revision](https://github.com/InferMatrixLab/InferRSI/blob/a590ccb25596396a592f237105166ade4aedeaaa/docs/reviews/vllm-ascend-0271rc1/history-inventory.json).
4+
It is omitted from the current tree under the [archiving contract](../../evidence-archiving.md).
5+
Runtime knowledge readers use the compact reviewed records and screening script;
6+
they do not consume this full scan.
7+
8+
- Original bytes: 1,401,157
9+
- SHA-256: `5df28360760a6d5cb513b7a4627eeb5a365d2d557b0fc2ff3d5c84cff925b72c`
10+
- Upstream repository: https://github.com/vllm-project/vllm-ascend
11+
- Upstream revision: `3b31886237ed65c435e0965b10e8002ff5769766`
12+
- Reachable commits: 5,029
13+
- Keyword/path/diff candidates: 1,381
14+
- Topic counts (overlapping): {"communication": 148, "deepseek-v4-dspark": 275, "fusion": 973, "priority-models": 251}
15+
16+
Before removal, the pinned copy was compared byte-for-byte; unique candidate
17+
count, topic totals and inclusion of the separately reviewed commits were
18+
checked. These are historical screening observations, not whole-commit semantic
19+
review or native validation. The immutable source remains available in history.
20+
This cleanup removes a generated file from the current tree only, not Git history.
21+
22+
For local retrieval from a checkout containing that revision:
23+
24+
```bash
25+
git show a590ccb25596396a592f237105166ade4aedeaaa:docs/reviews/vllm-ascend-0271rc1/history-inventory.json > <local-inventory.json>
26+
```
27+
28+
Check its byte count and SHA-256 above before use. Re-run the published screening
29+
script against the pinned upstream revision for an independent reconstruction;
30+
behavioral regression tests use a small synthetic Git history, not this scan.

0 commit comments

Comments
 (0)