Skip to content

Complete the governance instrument family: DPP, IAE, attribution-FAR - #250

Merged
HarperZ9 merged 1 commit into
mainfrom
feat/complete-instrument-family
Sep 14, 2026
Merged

HarperZ9 merged 1 commit into
mainfrom
feat/complete-instrument-family

Conversation

@HarperZ9

Copy link
Copy Markdown
Owner

What

Three re-derivable instruments under flywheel gov, each shipping its does-not-prove line and covered by unit + CLI tests.

  • DPP (dataset_provenance.py, gov dpp) — verify a dataset's declared shard hashes and origins against the shards an evaluation actually observed; flag hash mismatch, undeclared shards, unknown-origin shards, and duplicate content. MATCH / DRIFT / UNVERIFIABLE. Motivated by the "distilled frontier data of unknown origin" gap and the local-first enterprise data-as-asset market.
  • IAE (intervention_ablation.py, gov iae) — the paired-ablation test (an alignment intervention vs its ablated counterpart, misalignment judged per trajectory by a separate check); reports the misalignment-rate effect and a REDUCES / NO_EFFECT / INCREASES verdict, so a backfiring or inert intervention is a finding, not a silence. This is Daniel Tan's experiment as an instrument.
  • attribution-FAR (attribution_far.py, gov far) — check the checker: measure how often an attribution method accepts a known-null control (false accept). RELIABLE / UNRELIABLE over the control corpus. The false-accept-corpus discipline applied to attribution.

Why

These complete the model-level instrument family (EIM, CED, NCEC, ERG, IGAP, SMP, ICP + these three). They keep the shared discipline: deterministic given the input, a verdict lattice, honest does-not-prove bounds, and a false-success control (compliance/misalignment/attribution judged by a separate signal, never self-confirming). attribution-FAR in particular makes the family's own attribution claims checkable — robust to the conflict-of-interest and capture concerns that motivate independent, re-derivable evaluation.

Tests

tests/test_dataset_provenance.py, tests/test_intervention_ablation.py, tests/test_attribution_far.py: verdict paths, computations, both-condition / min-control requirements, malformed inputs, threshold ranges, and the gov dpp|iae|far CLI (exit codes + dispatch through cli_entry). Full slice green; file-size gate clean; writing lint clean.

🤖 Generated with Claude Code

Add three instruments under `flywheel gov`, each re-derivable and shipping its
does-not-prove line:

- DPP (dataset provenance): verify a dataset's declared shard hashes and origins
  against the shards an evaluation observed; flag hash mismatch, undeclared and
  unknown-origin shards, and duplicate content. MATCH / DRIFT / UNVERIFIABLE.
- IAE (intervention ablation): the paired-ablation test of an alignment
  intervention against its ablated counterpart; reports the misalignment-rate
  effect and a REDUCES / NO_EFFECT / INCREASES verdict, so a backfiring or inert
  intervention is a finding, not a silence.
- attribution-FAR: check the checker; measure how often an attribution method
  accepts a known-null control. RELIABLE / UNRELIABLE over the control corpus.

Wire all three into the CLI with verdict-shaped exit codes and cover them with
unit and CLI tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@HarperZ9
HarperZ9 merged commit 48d98af into main Sep 14, 2026
50 checks passed
@HarperZ9
HarperZ9 deleted the feat/complete-instrument-family branch September 14, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant