Train/val data leakage in CaseOps records — prepare_caseops_data.py default overlaps 80% of val docs with training data

[leon2k2k2k]

Summary

This audit was conducted in collaboration with @andrewbaggio1. We audited all 34 CaseOps-lineage record-track PRs since #1729 (2026-04-18) for train/val data overlap. 15 have a systematic leak caused by the default behaviour of prepare_caseops_data.py. The current claimed frontier (#2118 at 1.04350) is one of them. The true clean frontier is #2014 at 1.05759.

This is posted to the best of our ability. If you are a PR author and believe your record has been misclassified, please reply and we will revisit.

---

The bug

prepare_caseops_data.py ships with:

SHARD_TOKENS = 10_000_000
parser.add_argument("--val-docs", default=10_000)

With the default --val-docs=10000, train shards start at canonical-stream document 10,000. Val is generated from the first 50,000 documents. Documents 10,000–49,999 (40,000 docs, 80% of val) appear in both train and val. The model partially memorizes them during training, then is scored on them as if they were held out.

The canonical HF dataset (romeerp/parameter-golf-caseops-v1) does not have this problem — its manifest has docs_val=50000, docs_train=8,181,945 with a strict disjoint partition by construction.

Every shipped copy of prepare_caseops_data.py across all 34 PRs is byte-identical. No PR ever passes --val-docs=50000.

The gold-standard description of the leak is in PR #2018's DATASET_AUDIT.md, which explicitly documents --val-docs=10000 train shards and independently verifies the first 80 shards byte-for-byte.

---

How we classified each PR

We applied two questions to each PR's reproduce flow (README data-setup section + all shipped shell scripts):

• Q1: Is there an explicit HF download command? (snapshot_download, cached_challenge_fineweb.py, huggingface-cli download — all targeting romeerp/parameter-golf-caseops-v1)
• Q2: Is prepare_caseops_data.py invoked?

Q1	Q2	Verdict
✅	❌	CLEAN
❌	✅	LEAK
✅	✅	check which is the real reproduce step
❌	❌	check train log (train_shards: 39 → CLEAN; train_shards > 1000 → LEAK; triple-nested path → LEAK; otherwise AMBIGUOUS)

---

Results

Bucket	Count	PRs
CLEAN	12	#1729, #1851, #1855, #1868, #1945, #1953, #1967, #2014, #2019, #2027, #2031, #2068, #2123, #2124
LEAK	15	#1736, #1769, #1797, #1923, #2007, #2018, #2060, #2071, #2078, #2100, #2101, #2109, #2118
AMBIGUOUS	6	#1787, #1908, #2041, #2075, #2117, #2121

---

Updated timeline, reconstructed to the best of our ability

Apr 18   #1729   1.06780   CLEAN
Apr 23   #1787   1.06335   AMBIGUOUS lean LEAK
Apr 27   #1851   1.06128   CLEAN
Apr 25   #1855   1.0611     CLEAN
Apr 28   #1908   1.06081   AMBIGUOUS lean CLEAN
Apr 29   #1868   1.06141   CLEAN
Apr 29   #1945   1.05943   CLEAN
Apr 30   #1953   1.05855   CLEAN
Apr 30  #1967    1.05851    CLEAN
Apr 30   #2014   1.05759   CLEAN

---

Recommendation

1. Records using prepare_caseops_data.py without --val-docs=50000 should be flagged pending re-run on clean data.
2. The current merged SOTA (Record: val_bpb = 1.06128 SmearGate BOS Fix + PR #1787 Base + Smear Gate + LQER Asymmetric + Phased TTT (indirect 3 seed mean) #1851/Record: SmearGate BOS Fix 3-Seed Compliance Re-run — val_bpb 1.06141 (3-seed mean) #1868 at 1.06128/1.06141) is clean and stands.
3. The script's default should be updated to --val-docs=50000, or submissions should be required to use the canonical HF dataset.

---

Credit

Thanks to @aquariouseworkman who both identified and attempted to fix this issue (PR #1851, the first clean record post-leak), and who raised it with the community. Full audit with per-PR evidence: caseops-memory-leakage/ on the research branch of our fork.

@cocohearts @valerio-oai

[jamesEmerson112]

@leon2k2k2k Acknowledged. Thanks

[ndokutovich]

@leon2k2k2k @andrewbaggio1 — thanks for the careful audit. Re-checking PR #1967 (which you bucketed LEAK): the measurement runs were on the clean 50K-val partition. The shipped setup.sh does have the documentation bug you describe and I'll patch it. Evidence below.

Measurement coverage — three shipped train logs

All three seed logs in PR #1967 head:

[seed 42]    val_tokens: 47851520    train_shards: 1499    val_doc_fraction: 1.0
[seed 0]     val_tokens: 47851520    train_shards: 1499    val_doc_fraction: 1.0
[seed 1234]  val_tokens: 47851520    train_shards: 1499    val_doc_fraction: 1.0

47,851,520 ≈ 47.85M is the dex PR #1797 reference (50,000 val_docs). The leaky default (--val-docs=10000) yields ~9.66M val_tokens. We're on the disjoint 50K-doc partition, not the 10K-overlapped one.

How that happened — pre-built shards, not local prep

Our actual reproduce orchestrator (not shipped in #1967) does:

# Fast-resume from our own prep upload — pulled before prepare_caseops_data.py is invoked
if [ "$SIDECARS" -lt 1 ] && [ -n "${HF_TOKEN:-}" ]; then
    snapshot_download(repo_id='alienNiko/caseops-s11-data-v3', ...)
fi
# Local prep is only the fallback when HF_TOKEN is missing
if [ "$SIDECARS" -lt 1 ]; then
    python3 prepare_caseops_data.py --docs ... --out ... --sp ...   # leaky default
fi

alienNiko/caseops-s11-data-v3 was prepared on a separate machine using a wrapper that hardcodes --val-docs=50000 and a verifier that hard-fails on val_tokens divergence >1% from dex's 47,851,520 reference. The train_seed*.log proof points show fast-resume hit, fallback never executed.

The shipped-setup bug

The setup.sh checked into PR #1967 doesn't carry the fast-resume path and doesn't pass --val-docs=50000 to prepare_caseops_data.py. A reviewer cloning the PR head and running the shipped reproduce verbatim would end up on the leaky 10K partition. That's a documentation/reproducibility bug — I'll push a fix that either:

(a) adds --val-docs=50000 explicitly, or
(b) adds the snapshot_download('romeerp/parameter-golf-caseops-v1') path Q1 of your classifier expects.

Either is fine; (b) is preferred since it removes the local-prep step entirely.

Classification request

Per your invitation in the OP, asking #1967 be re-bucketed as CLEAN measurement, leaky reproduce script (or AMBIGUOUS pending the setup.sh patch), based on the three shipped seed logs. Happy to provide additional artifacts — preprocessed shard checksums, or a re-run on romeerp/parameter-golf-caseops-v1 directly if you'd like an independent confirmation.

Crediting @aquariouseworkman for the --val-docs=50000 discipline; that wrapper is what kept our measurement clean.

[dexhunter]

Thanks for the audit. For #1736, #1769, #1797: the submitted seed logs show val_tokens: 47,851,520, which corresponds to the canonical 50,000-doc val partition. A --val-docs=10000 prep would produce a different token count, so the submitted measurements were on the canonical val — the same partition used by the CLEAN-bucketed PRs.

The shipped README does invoke prepare_caseops_data.py without overriding the --val-docs=10000 default — that's a reproducibility bug we'll patch (pin --val-docs=50000 in the README invocation). The reported BPB numbers are unaffected since the actual run used the canonical val.

Could you reclassify these three as CLEAN? The measurement evidence (val_tokens) is falsifiable directly from the seed logs.