|
| 1 | +# Experiment evidence and archive verification |
| 2 | + |
| 3 | +Implements the inclusion/template/tooling slice of the |
| 4 | +[scoped archiving decision](https://github.com/InferMatrixLab/InferRSI/issues/464#issuecomment-6913744401). |
| 5 | +The RFC's completion and campaign acceptance remain separate owner decisions. |
| 6 | + |
| 7 | +## Repository contents |
| 8 | + |
| 9 | +Retain implementation, necessary tests, goals, conclusions, limitations, key |
| 10 | +metrics, effective configuration, input/validator identities, a complete trial |
| 11 | +index and verification entry points. Start from [the bundle template](../evidence/_template/README.md). |
| 12 | +Small test fixtures stay only when their consumers and necessity are clear. |
| 13 | + |
| 14 | +Archive per-request results, per-sample scores, profiler output, logs, media, |
| 15 | +complete source inventories and generated process reports outside Git. Classify |
| 16 | +JSON by its purpose; changing directories, minifying, splitting or committing |
| 17 | +compressed archives does not justify retaining large generated artifacts. |
| 18 | +Do not retain temporary outputs with no review or reproduction value. |
| 19 | + |
| 20 | +Generate summaries and trial indexes from the original control/evidence records. |
| 21 | +Record all attempts, including failed, regressing, interrupted and repeated |
| 22 | +trials. Retain sample/failure counts, repetitions, variability, metric definitions |
| 23 | +and incomplete validation. Preserve frozen raw bytes and historical conclusions. |
| 24 | + |
| 25 | +## Archive authority and schema |
| 26 | + |
| 27 | +Each campaign's `results/artifacts.json` owns its archive metadata. The inventory |
| 28 | +references it via `schema: archive-sha256`, `field: archive`, and the expected |
| 29 | +local artifact path. Do not copy URL/hash/bytes/owner into inventory entries. |
| 30 | +The offline inventory rejects such duplicate `external_assets` metadata. |
| 31 | + |
| 32 | +New documents use **schema_version 2**, illustrated by |
| 33 | +[artifacts.example.json](../evidence/_template/results/artifacts.example.json). |
| 34 | +Replace placeholders with measured values, including integer byte/file counts. |
| 35 | +The executable format contract is `normalize_manifest` in |
| 36 | +[tools/evidence_archives.py](../tools/evidence_archives.py). |
| 37 | + |
| 38 | +| Field | Contract | |
| 39 | +| --- | --- | |
| 40 | +| `archive.id`, `file` | Stable version identity and basename | |
| 41 | +| `archive.url` | Credential-free HTTPS download URL; no query or fragment | |
| 42 | +| `archive.sha256`, `bytes` | Lowercase SHA-256 and positive compressed byte count | |
| 43 | +| `archive.format` | `tar+gzip`; regular nonsparse files only | |
| 44 | +| `archive.source_revision` | Immutable 40–64 hexadecimal source revision | |
| 45 | +| `archive.root` | Safe relative member prefix, or empty for the whole archive | |
| 46 | +| `archive.files`, `uncompressed_bytes` | File count and sum of file bytes under root | |
| 47 | +| `archive.key_files` | Named archive-relative files or directory prefixes ending `/` | |
| 48 | +| `archive.retention_owner` | Named owner responsible while references remain | |
| 49 | +| `verification.inputs` | Archive-relative SHA-256 identities with at least control, configuration and validator roles | |
| 50 | +| `verification.command` | Reviewed replay argv, recorded but never executed by the generic tool | |
| 51 | +| `verification.trials`, `summary` | Paths relative to retained/regenerated bundle roots | |
| 52 | +| `verification.absolute_tolerance`, `relative_tolerance` | Finite nonnegative float-comparison tolerances; integers/structure exact | |
| 53 | + |
| 54 | +Both frozen v1 pilots remain readable through the same normalization function: |
| 55 | +`tar.gz`/`tar+gzip`, `files`/`original_files`, |
| 56 | +`uncompressed_bytes`/`original_bytes`, and the two source-revision/key-file |
| 57 | +locations normalize into one view. Conflicting aliases fail validation. H3's |
| 58 | +`snapshot.json` is outside its original-file root; it is still inspected and |
| 59 | +counted against safety limits. V1 replay remains with each frozen campaign |
| 60 | +checker; normalization does not rewrite either pilot's bytes or interpretation. |
| 61 | + |
| 62 | +## Publication and explicit verification |
| 63 | + |
| 64 | +1. Generate the complete raw artifact locally. Review privacy before publication. |
| 65 | +2. Generate summaries and the index from those raw/control records; test coverage. |
| 66 | +3. Package regular files, publish a versioned Release asset, and record measured |
| 67 | + metadata. Controlled originals remain in authorized storage under the |
| 68 | + [privacy rules](privacy-security.md). |
| 69 | +4. Download again and verify the published bytes and contents with the explicit |
| 70 | + tool. Run the reviewed replay command and compare outputs. |
| 71 | +5. Submit the compact bundle, verification results and availability limits. |
| 72 | + |
| 73 | +From the repository root (replace angle-bracket placeholders): |
| 74 | + |
| 75 | +```bash |
| 76 | +# Offline metadata only. Never downloads or certifies content. |
| 77 | +python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json |
| 78 | +python tools/validate_evidence.py |
| 79 | + |
| 80 | +# Explicit network operation, bounded bytes/hash/member verification. |
| 81 | +python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json --download |
| 82 | + |
| 83 | +# Controlled storage: retrieve through its authorized client, then verify offline. |
| 84 | +python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json --archive <local-archive> |
| 85 | + |
| 86 | +# Separately run the reviewed, pinned replay command from the manifest, then: |
| 87 | +python tools/verify_evidence_archive.py evidence/<campaign-id>/results/artifacts.json \ |
| 88 | + --archive <local-archive> --recomputed <new-output-directory> |
| 89 | +``` |
| 90 | + |
| 91 | +The common tool authenticates compressed bytes before tar parsing, then streams |
| 92 | +members without filesystem extraction or archived code execution. It rejects |
| 93 | +absolute/traversing paths, duplicate names, links, devices and sparse members. |
| 94 | +Defaults cap compressed data at 256 MiB, expanded tar bytes (including metadata) |
| 95 | +at 512 MiB, files at 10,000, and each download/verification operation at 120 |
| 96 | +seconds. Downloads have a 10-second socket timeout and permit only GitHub |
| 97 | +Release hosts and their HTTPS asset redirects. These are tool resource limits, |
| 98 | +not experiment acceptance gates. Larger archives need an explicitly reviewed |
| 99 | +tool configuration. Split campaigns only for meaningful storage reasons. |
| 100 | + |
| 101 | +Failures distinguish metadata, unavailable/restricted access, bad bytes/hash, |
| 102 | +unsafe paths/members/redirects, resource limits, missing files, bad contents, |
| 103 | +trial mismatch and summary recomputation mismatch. Output always states which |
| 104 | +scope was checked. Metadata success or verified archive bytes alone leaves |
| 105 | +`recomputation: not_checked`; successful comparison reports `outputs_match`. |
| 106 | +The caller must review the replay's derivation and coverage; comparing two |
| 107 | +identical files cannot prove they came from independent recomputation. |
| 108 | + |
| 109 | +Offline CI does not download, unpack or run a replay. A broken link records a |
| 110 | +verification gap without changing the historical verdict. Do not use expiring |
| 111 | +CI links or local paths as the only archive source. New bytes require a new |
| 112 | +version; keep the old digest/version retrievable. Redacted public copies have |
| 113 | +their own identity and explicit limits, never overwrite frozen originals. |
| 114 | + |
| 115 | +## R6 reporting and exceptions |
| 116 | + |
| 117 | +Existing R6 thresholds remain report-only: 512 KiB per `evidence/` file and |
| 118 | +2 MiB net evidence growth per PR. This change adds no blocking size gate. |
| 119 | +Reconsideration remains on or after 2026-12-02 under the existing decision. |
| 120 | + |
| 121 | +[PR #328](https://github.com/InferMatrixLab/InferRSI/pull/328) owns the pending byte |
| 122 | +reporter. Its legacy input can be generated without maintaining duplicate |
| 123 | +metadata (after that checker is integrated): |
| 124 | + |
| 125 | +```bash |
| 126 | +python tools/validate_evidence.py --budget-assets > <temporary-budget-view.json> |
| 127 | +python tools/check_evidence_budget.py --base <base-ref> --report-only \ |
| 128 | + --inventory <temporary-budget-view.json> |
| 129 | +``` |
| 130 | + |
| 131 | +The generated view is disposable, not committed. #328's Release-only URL |
| 132 | +format check does not support controlled-store URLs; use the shared reader and |
| 133 | +an authorized local archive for those and report that compatibility gap. The |
| 134 | +view does not claim remote availability. Added/deleted/net byte reporting |
| 135 | +remains in #328's follow-up scope. |
| 136 | + |
| 137 | +Use [the exception record](../evidence/_template/plan/storage-exception.md) when |
| 138 | +a specific oversized artifact must stay in Git. Record purpose, consumers, |
| 139 | +externalization obstacle, byte count, maintenance owner and explicit maintainer |
| 140 | +agreement. An exception record does not create automatic enforcement. |
| 141 | + |
| 142 | +## Cleanup and acceptance |
| 143 | + |
| 144 | +Before removing redundant generated data, identify its consumers and preserve |
| 145 | +an exact, retrievable source with digest/size and a concise summary. Convert |
| 146 | +behavioral tests to small synthetic inputs when they do not need a full source |
| 147 | +scan. Verify links and affected consumers after removal. This removes the file |
| 148 | +from the current tree, not existing Git history. |
| 149 | + |
| 150 | +This workflow does not automatically migrate frozen merged campaign evidence. |
| 151 | +An explicitly requested, scoped cleanup may replace the current presentation |
| 152 | +only after every original file is preserved in a verified versioned archive; |
| 153 | +original frozen bytes, historical conclusions and Git history remain unchanged. |
| 154 | +For new/unmerged work, externalize before the required squash merge so large |
| 155 | +blobs do not enter main. The two pilot archive/replay checks and their |
| 156 | +independent reviews remain campaign-specific acceptance records. Completing |
| 157 | +this tool or removing redundant data neither establishes experiment PASS nor |
| 158 | +closes the parent RFC. |
| 159 | + |
| 160 | +### Scoped legacy record projections |
| 161 | + |
| 162 | +The [H3 four-device bundle](../evidence/h3-ref2va-910c-4logical-20261101-297/README.md) |
| 163 | +and [Qwen3.8 bundle](../evidence/qwen38-h20-fp8-single-request-vllm032/README.md) |
| 164 | +use v2 for their explicitly requested cleanup. Their full originals, including |
| 165 | +superseded cumulative checkpoints and repeated diagnostic code, remain in the |
| 166 | +archives. This is not blanket authorization to migrate other frozen campaigns. |
| 167 | + |
| 168 | +`tools/rebuild_model_evidence.py` regenerates compact views from authenticated |
| 169 | +originals. `archive.key_files.snapshot` identifies the original per-file hash map; |
| 170 | +`archive.key_files.projection` identifies the declarative source/metric selection. |
| 171 | +Both and the replay code are pinned by `verification.inputs`. The replayer uses |
| 172 | +the common bounded reader to capture authenticated bytes, checks original frozen |
| 173 | +manifests, and requires its local implementation to match the pinned code. It |
| 174 | +never executes code from an archive and never overwrites an existing output. |
| 175 | + |
| 176 | +Index coverage is deliberately labelled: every published result JSON document |
| 177 | +and named/nested attempt observation, including failures and cancellations. |
| 178 | +Identical observations of the same named attempt share one row and retain all |
| 179 | +source pointers; changed observations, retries and anonymous records remain |
| 180 | +distinct. `configuration.record_sha256` hashes canonical JSON for the source |
| 181 | +observation. Entry counts are not independent trial counts. Narrative-only |
| 182 | +history and unpublished controller records cannot become fully structured |
| 183 | +trial evidence through a storage migration; those limits remain explicit. |
| 184 | + |
| 185 | +Declared sample distributions are recomputed; other frozen reported statistics |
| 186 | +and verdicts are labelled as recorded values. Replaying the projection does not |
| 187 | +rerun inaccessible quality checks, change gates or complete campaign acceptance. |
0 commit comments