Record: L_ParallelResiduals_UNetSkips_DepthRecur_Muon_ 1.0901

[Hieuabssy]

Record candidate: 11L Parallel Residuals + U-Net Skip Fix + Depth Recurrence + Muon + GPTQ int6

val_bpb: 1.09012178 (3-seed mean, std 0.00139) | val_loss: 2.50840009 nats (std 0.00319) | 15.99 MB max | 8xH100 80GB HBM3 | 590s effective train cap / ~209s sliding eval

This submission targets the track_10min_16mb challenge with an 11-layer, 512d GPT-style model. The headline score is the final int6+brotli sliding-window evaluation averaged over seeds {42, 1337, 1024}. All three artifacts are under the 16,000,000-byte cap; the maximum observed submission size is 15,991,908 bytes, leaving 8,092 bytes of headroom.

The contribution is not a test-time-training or validation-cache trick. It is a compact architecture/training stack built around parallel late residual lanes, U-Net-style skip injection, delayed depth recurrence, Muon optimization, EMA, GPTQ int6 quantization, and a compiled sliding-window evaluator.

Results

Seed	Steps	Train ms	Pre-quant BPB	Int6 roundtrip BPB	Sliding BPB	Sliding eval s	Artifact bytes
42	5,254	590,094	1.09634716	1.10793344	1.08946190	208.6	15,981,801
1337	5,071	590,082	1.09758610	1.11025822	1.09171602	208.9	15,955,242
1024	5,260	590,052	1.09627485	1.10764332	1.08918743	208.7	15,991,908
Mean	5,195	590,076	1.09673604	1.10861166	1.09012178	208.7	15,976,317

3-seed sample std: 0.00139 BPB / 0.00319 nats.

The run reserves 10 seconds for GPTQ and trains against an effective 590,000 ms cap. Logs show wallclock early stopping at 590.052-590.094s, followed by EMA application, GPTQ calibration, int6+brotli serialization, and final sliding-window scoring.

Statistical profile

The best seed is 1024 at 1.08918743 BPB. The worst seed is 1337 at 1.09171602 BPB. The 3-seed spread is 0.00252859 BPB, with the submitted mean matching submission.json exactly:

Field	Value
val_loss	2.50840009
val_bpb	1.09012178
bytes_total	15,976,317
bytes_code	84,383

No paired PR baseline is included in this record folder, so this note does not claim a significance test versus another PR. It reports the 3-seed aggregate and per-seed logs directly.

What is new in this submission

The folder metadata names the submission as Parallel Residuals + U-Net Skip Fix. The two reviewer-facing changes are:

Lever	Setting / behavior	Mechanism
U-Net skip fix	Skip tensors are gated and injected into the decoder path while coexisting with late parallel residual lanes.	Preserves shallow linguistic features without letting the parallel split drop or double-count skip information.
Compiled sliding eval	eval_val_sliding uses the optimized compiled model path when available.	Keeps the higher-quality stride-64 sliding evaluation practical inside the evaluation budget.

The broader stack also includes delayed recurrence over layers 3,4,5, value embeddings on layers 9,10, Muon for matrix weights, EMA with EMA_DECAY=0.9965, and GPTQ over 64 calibration batches.

Architecture and training stack

Component	Setting
Model	11 layers, 512d, 8 query heads, 4 KV heads, MLP 4x
Parameters	34,401,372
Tokenizer/data	FineWeb SP4096 BPE, fineweb10B_sp4096 shards
Train context	TRAIN_SEQ_LEN=2048, TRAIN_BATCH_TOKENS=786432, 8-way distributed
Eval context	EVAL_SEQ_LEN=2048, EVAL_STRIDE=64, sliding-window scoring
U-Net skips	Encoder/decoder split with learned per-dimension skip_weights and sigmoid skip_gates
Parallel residuals	Late split from PARALLEL_START_LAYER=7; attention and MLP lanes merged by learned lane_merge
Depth recurrence	RECUR_LAYERS=3,4,5, activated at RECUR_START_STEP=3000; virtual layer order becomes [0,1,2,3,4,5,3,4,5,6,7,8,9,10]
Value embeddings	VE_ENABLED=1, VE_LAYERS=9,10, VE_DIM=128
Optimizer	Muon on matrix params (MATRIX_LR=0.022, momentum 0.99, warmup from 0.92 over 1500 steps), AdamW on embeddings/scalars
EMA	EMA_DECAY=0.9965 applied before quantization
Quantization	GPTQ int6 on attention/MLP matrices, 64 calibration batches, full GPTQ on 66 layers
Compression	byte-shuffled brotli, plus selective lowest-error +/-1 pruning to fit 16MB
TTT	Disabled (TTT_ENABLED=False)

Quantization and compression

Each seed collects 66 GPTQ Hessians in about 10.5s and uses full GPTQ for all 66 quantized layers with no clip-search fallback. After quantization, the serializer applies selective pruning only if the unpruned package exceeds the cap:

Seed	Unpruned size	Pruned entries	Excess removed	Final bytes
42	16.01 MB	115,352 / 9,407,077	14,419 B	15,981,801
1337	16.04 MB	286,088 / 9,372,630	35,761 B	15,955,242
1024	16.01 MB	53,824 / 9,411,782	6,728 B	15,991,908

This is a storage-fitting step on the already-quantized representation. The logs show every final artifact under the 16,000,000-byte submission limit.

Compliance

• C1 causality. The model is a causal Transformer-style decoder. The final score comes from eval_val_sliding, which feeds only left context into each window and scores tokens after their available prefix.
• C2 single accounting. Sliding-window evaluation uses EVAL_STRIDE=64 and scores the suffix of each window after the context prefix (s = 0 for the first window, otherwise s = seq_len - stride), so validation tokens are accounted once in the aggregate.
• C3 no TTT dependency. TTT_ENABLED=False in the logged hyperparameters. The submitted BPB is the int6 sliding-window eval, not a score-first TTT pass.
• C4 train budget. GPTQ reserves 10s from the training wallclock (effective=590000ms). All seeds stop by the wallclock cap before serialization and quantization.
• C5 artifact budget. Maximum logged final submission size is 15,991,908 bytes, including code, below the 16,000,000-byte cap.

No n-gram cache, validation-derived logit table, persistent retrieval state, or PPM-style mixture is present in this record. GPTQ calibration is collected from the training loader during the post-training quantization phase.

Lineage and credits

• Submission metadata: Parallel Residuals + U-Net Skip Fix, author trung.
• README author line: Quan group HCMUS.
• Architectural ideas used here: GPT-style decoder, U-Net skip routing, late parallel residual lanes, delayed depth recurrence, value embeddings, Muon optimization, EMA, GPTQ int6 quantization, byte-shuffled brotli compression, and stride-64 sliding-window evaluation.

Related PR and issue links

• PR #1445 - 3-layer depth recurrence + EMA 0.9965 + WD 0.095; immediate local draft baseline.
• PR #1334 - SP4096 + depth recurrence + parallel residuals + MuonEq-R + QK-Gain 5.0 base architecture credited by PR [Record] 3-Layer Depth Recurrence + EMA 0.9965 + WD 0.095 — val_bpb 1.0889 #1445.
• PR #1331 - 3-layer recurrence + WD/LR tuning credited by PR [Record] 3-Layer Depth Recurrence + EMA 0.9965 + WD 0.095 — val_bpb 1.0889 #1445.
• PR #1421 - EMA decay tuning (EMA_DECAY=0.9965) credited by PR [Record] 3-Layer Depth Recurrence + EMA 0.9965 + WD 0.095 — val_bpb 1.0889 #1445.
• Issue #897 - tokenizer byte-accounting issue referenced in the included code comments.
• Base architecture + depth recurrence: PR Record: SP4096 + Depth Recurrence + Parallel Residuals + MuonEq-R + QK-Gain 5.0 — val_bpb 1.0897 (3-seed mean) #1334 by @aryanbhosale
  3-layer recurrence + WD/LR tuning: PR Record: MuonEq-R + 3-Layer Recurrence + WD=0.095 + MLR=0.022 + All-Int6 — val_bpb 1.0900 (3-seed mean) #1331
  EMA decay tuning (0.9965): PR [Record] 11L Depth Recurrence + EMA Tuning (0.9965) — val_bpb 1.0925 #1421 by @X-Abhishek-X (this author)
  The new contribution captured by this record folder is the corrected interaction between U-Net skips and parallel residual lanes, plus the compiled-model path for final sliding-window evaluation, validated over three seeds under the 10-minute / 16MB constraints.

Credit:

University of Science, VNU-HCM. HCMUS-Quan