HYDRA Golf v6.1

Extreme language-model compression under a 10-minute / 16 MB artifact budget

→ project site · solo research fork · Evreu1pro

AI helped with the boring glue. Training ideas / what ships = me.

Train the smallest LM you can. Best verified model wins.

---

Overview

HYDRA Golf = my working fork of OpenAI Parameter Golf.

Goal of the challenge (upstream): smallest LM that fits in 16 MB, lowest BPB on FineWeb val, ~600s train budget.

What I bolted on: runnable entrypoints, safer artifacts, tokenizer-aware BPB, multi-seed checks, and a pile of runs under records/. Some of it is rough. Numbers below are my verified logs — double-check gates before you quote them.

Metric	Value
Global SOTA (May 2026)	1.0565 BPB — openai/parameter-golf
HYDRA best verified	1.06108 BPB (3-seed mean) — ~#5 on historical leaderboard
Gap to SOTA	~0.0046 BPB
Artifact size	~15.9 MB
Improvement vs 1.0810 baseline	−0.01992 BPB
Next target	<1.0540 BPB (research horizon, not claimed)

Results are repository-local verified records. See acceptance gates before treating any number as competitive.

---

Competition participation

I run this fork of OpenAI Parameter Golf under the name HYDRA. Not an OpenAI team — just a public research fork. What lives here:

• Verified records under records/ with logs, seeds, and submission.json
• Top-5 stack — SmearGate + LQER + SparseAttnGate at 1.06108 BPB (Apr 2026)
• HYDRA v6.1 — engineering platform for the next push toward 1.0565 global SOTA

Rank	Run	BPB	Date
#1	Calib32 N-gram + AsymLogit	1.0565	2026-05-01
#2	Progressive Context + TTT	1.0576	2026-04-30
#5	HYDRA stack (this fork)	1.06108	2026-04-27
…	3-Layer Recurrence baseline	1.0810	2026-04-09

Why HYDRA

Parameter Golf is not solved by a single trick. HYDRA treats optimization as a multi-head problem:

• Architecture — 11L GQA GPT, XSA, SmearGate, depth recurrence, U-Net skips
• Quantization — LQER int4, greedy 1–8 bit packing, per-group compression
• Schedule — 600s warmdown, token batching at memory limits
• Verification — 3+ seeds, post-load eval, submission.json with SHA-256

Cut one head — the others keep advancing.

---

Quick start

Smoke test (local, no dataset required)

python -m pip install -r requirements.txt
python -m unittest discover -s tests -v
HYDRA_PROFILE=smoke ITERATIONS=2 MAX_SECONDS=15 python train_gpt.py

Smoke BPB uses synthetic shards — do not compare with the leaderboard.

Challenge-shaped run

export HYDRA_PROFILE=challenge
export DATA_PATH=./data/datasets/fineweb10B_sp8192
export TOKENIZER_PATH=./data/tokenizers/fineweb_8192_bpe.model
export VOCAB_SIZE=8192
export NUM_LAYERS=11
export MODEL_DIM=512
export NUM_HEADS=8
export NUM_KV_HEADS=4
export MLP_MULT=4
export SEQ_LEN=2048
export BATCH_TOKENS=786432
export MAX_SECONDS=600
export TTT_ENABLED=0

torchrun --standalone --nproc_per_node=8 train_gpt.py

---

Outputs

File	Description
model.ptz	Versioned ZIP artifact — code manifest + weights + checksum
submission.json	Round-trip BPB, exact byte size, SHA-256
logs/<run_id>.txt	Master-rank training log

---

Repository layout

hydra_golf/          Safe artifact serialization and BPB metrics
train_gpt.py         Main PyTorch entrypoint (smoke / challenge)
train_gpt_mlx.py     Experimental MLX baseline
records/             Verified experiments with logs and submissions
tests/               Correctness tests
configs/             Environment profiles
data/                Dataset preparation utilities
docs/                Project website (GitHub Pages)

---

HYDRA v6.1 — new engineering wave (Jul 2026)

After competition runs, v6.1 turns the fork into a reproducible research platform:

Phase	Milestone
Mar 2026	Fork + first records, baseline → 1.08x BPB
Apr 2026	Top-5 stack at 1.06108 BPB, 3-seed verified
Jul 2026	v6.1: hydra_golf/, model.ptz, tests, CI, project site
Next	Close 0.0046 BPB gap to global SOTA 1.0565

What changed in v6.1

• Replaced notebook wrapper with executable train_gpt.py
• Fixed forward_logits() batch/sequence/vocabulary axes
• Separated smoke and challenge modes with explicit failure behavior
• Added tokenizer-aware BPB utilities
• Real 1–8 bit packing (not pseudo N-bit labels)
• Safe model.ptz artifacts — ZIP + SHA-256, no pickle execution
• Unit tests and CI
• TTT disabled by default

See CHANGELOG.md for details.

---

Acceptance gates

A competitive result must include:

1. Exact commit and environment specification
2. At least three independent seeds
3. Official tokenizer-aware validation BPB
4. Independent train and validation data
5. Artifact size ≤ 16,000,000 bytes (exact)
6. Evaluation after loading the artifact in a clean process
7. Mean, standard deviation, and per-seed results

---

Experimental status

Mobius recurrence, evolutionary TTT, learned asymmetric logit caps, GEGLU, QAT, and differentiable head gating are research components. Benefits require isolated ablations. This repo does not claim CMA-ES, guaranteed FLOP reduction, or an achieved <1.0540 result.

---

Security

model.ptz loading does not execute embedded code and does not use pickle. Model source is returned as text for inspection — never execute code from untrusted artifacts.

---

Links

• Project site: https://evreu1pro.github.io/parameter-golf/
• Best record: records/track_10min_16mb/2026-04-27_SP8192_LQER_SparseGate_BOSSmearFix_9HpStack_1.0611
• Upstream: https://github.com/openai/parameter-golf

---

License

MIT — see LICENSE and THIRD_PARTY_NOTICES.md.