Skip to content

feat(experiments): validate reproducible performance records - #1238

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
ZacharyZcR:feat/experiment-manifests
Sep 7, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
ZacharyZcR:feat/experiment-manifests

Conversation

@ZacharyZcR

@ZacharyZcR ZacharyZcR commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a dependency-free validator for reproducible experiment manifests
  • require a single changed variable, at least three raw samples, a correct median, and an explicit quality gate
  • bind raw evidence with SHA-256 and retain negative or no-change results in the same format
  • document the workflow in CONTRIBUTING.md and the pull-request checklist

Why

Performance claims are difficult to review when the changed variable, raw samples, aggregation, quality result, or source revision is missing. The manifest makes those fields machine-checkable without adding a runtime dependency.

Validation

  • rebased onto current dev
  • PYTHONPATH=. python3 tests/test_experiment_manifest.py: 5/5 passed
  • python3 experiment_manifest.py ../docs/experiments/manifest.example.json: passed
  • git diff --check: passed

@ZacharyZcR
ZacharyZcR force-pushed the feat/experiment-manifests branch from 98b3910 to 00ecd50 Compare August 31, 2026 07:35
@JustVugg
JustVugg merged commit d18fdf1 into JustVugg:dev Sep 7, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants