Skip to content

Remove references to the private experiments repository and its results - #197

Merged
RoldSI merged 2 commits into
mainfrom
chore/remove-experiments-refs
Sep 26, 2026
Merged

RoldSI merged 2 commits into
mainfrom
chore/remove-experiments-refs

Conversation

@RoldSI

@RoldSI RoldSI commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

The private superred-experiments repository holds the authors' analysis. This PR removes every reference to it, and to its study results, from the public modules repo. Two commits:

1. References to the repository (48d74cb)

  • README.md: combinations are now described as wired in a short script handed to the Controller, not as living in superred-experiments/.
  • security_claims/harmbench/README.md: the example imported DirectRequestOptimizer from a file that only exists in the private repo. It now uses the public GoalPassthroughOptimizer, which does the same thing (injects the goal once, verbatim).
  • DTAP_README.md: the section telling readers to cd ../superred-experiments/test_dtap is replaced by a generic description of the end-to-end wiring.
  • Smoke scripts (crescendo, fitd, pair) and live tests (asb, dtap_scaffold) no longer look for the private repo's .env. The smoke scripts use load_dotenv(), and the live tests read environment variables only.
  • A HarmBench test comment no longer names a file in the private repo.

2. Study labels and result counts (bb47be4)

  • Comments, docstrings, READMEs and ASSUMPTIONS.md files (106 files) no longer cite internal study labels (RQ1.x, "first run/sweep/archive", "DTAP indirect sweep") or counts measured in those runs. The technical rationale is kept, reworded without the numbers. Scope labels (s3-s6) stay.
  • Includes one sentence removed from the 55 identical test_no_temperature.py docstrings. The 5 judge_failure.py copies stay byte-identical.
  • No code behavior changes: no identifiers, runtime strings, test assertions or data were modified.

Verification

  • A grep for labels, archive references and private counts returns nothing. Vendored upstream code and public benchmark counts are left unchanged.
  • ruff check passes on all changed .py files.
  • The tests of every touched package pass. Two failures are unrelated to this PR: aegis test_imports_without_heavy_deps (the package is not installed in the local venv) and dtap_scaffold test_judge_runner.py (the tests import from a different checkout).

Not covered

  • Git history still contains the old text.
  • Crescendo test fixtures keep the literal $0.750000 cost-cap string.

Docs no longer point at superred-experiments; the HarmBench example uses the
public goal_passthrough baseline; smoke scripts and live tests read LiteLLM
credentials from the environment (or load_dotenv) instead of the sibling repo.
Comments, docstrings, READMEs and ASSUMPTIONS.md files no longer cite
internal study labels (RQ1.x, first run/sweep/archive, DTAP indirect sweep)
or counts measured in those runs. The technical rationale is kept; scope
labels (s3-s6) stay. No code behavior changes.
@RoldSI
RoldSI merged commit efc0e53 into main Sep 26, 2026
1 check passed
@RoldSI
RoldSI deleted the chore/remove-experiments-refs branch September 26, 2026 05:49
@claude

claude Bot commented Sep 26, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant