Add validated 60k DeepSeek V4 Flash LoRA training and serving

[devin-ai-integration]

Summary

Adds one focused DeepSeek-V4-Flash workflow:

• rank-64 LoRA SFT at 60,000 tokens with NeMo AutoModel on 16 x 8 H200 GPUs;
• five real forward/backward/optimizer steps and a pipeline-sharded checkpoint;
• strict conversion to a standard PEFT adapter;
• adapter serving with vLLM 0.25.1 on 4 H200 GPUs using PP=4;
• a base-versus-adapter GSM8K sanity slice and semantic retrieval at exactly 60,000 input tokens.

The training module now exposes only run_id for training, finalization, and validation. Topology, sequence length, LoRA shape, optimizer steps, checkpoint behavior, EFA eligibility, and memory limits are fixed to the validated recipe. Disposable smoke tests, transport probes, experimental backends, and inactive NVSHMEM controls were removed.

The failed Megatron/ms-swift experiment and its large DSv4 boundary rewrite remain excluded. None of the retained patches changes model forward functions, attention math, or autograd.

Validated training result

Dimension	Validated value
Hardware	16 nodes x 8 H200 (128 GPUs)
TP / DP / PP / CP / EP	1 / 2 / 4 / 16 / 32
Sequence length	60,000
Global / local batch	8 / 4
Attention / MoE dispatch	TileLang / UCCL-EP
LoRA	rank 64, alpha 64
Targets	wq_a, wq_b, wkv in all 43 layers
Optimizer steps	5
Host memory	128 MiB scheduler request, 256 GiB cgroup limit

Evidence:

• Modal training app ap-2wSNiwan5bsesrgftETpAb
• Run ID: dsv4-flash-h200-16n-cp16-60k-lora64-5step-20260716-203704
• Every step processed 479,896 supervised label tokens globally, or 59,987 per sample.
• Loss remained finite from 0.0010 to 0.0008; gradient norm remained finite and nonzero from 0.0017 to 0.0014.
• Step times were approximately 413, 28, 23, 22, and 22 seconds. The first includes compilation and distributed initialization.
• All four PP stages wrote their step-4 adapter shards.

The request was EFA-enabled and landed on Mellanox/RoCE capacity. The image contains runtime-selected UCCL extensions for both EFA and Mellanox.

Checkpoint result

The finalizer merges the PP shards, rejects missing, duplicate, unexpected, wrong-shaped, non-finite, or unchanged LoRA tensors, maps projection names to official Transformers names, and performs a PEFT load against a meta-device base model.

Validated artifact:

• Path: /dsv4-flash-h200-16n-cp16-60k-lora64-5step-20260716-203704/epoch_0_step_4/model/adapter_model.safetensors
• SHA-256: 93839c8de196c581bf76f4305906eeaa5d9e22bf7ac5d5fe073ddc3970bf6a4f
• 258 tensors across all 43 transformer layers.
• 129 logical LoRA modules, with every LoRA-B tensor nonzero.
• Zero missing adapter keys and zero unexpected keys through PEFT.

The recipe intentionally omits AdamW state. This is a serving/export adapter, not an exact optimizer-state resume checkpoint.

End-to-end serving and eval

The serving path uses vLLM 0.25.1 and the native FP4/FP8 checkpoint.

• Modal eval app ap-zfkYYZIHgVGwbYp8izzdPP
• The exact adapter SHA above passed preflight and mapped 129 logical modules: 43 each for q_a_proj, kv_proj, and wq_b.
• vLLM loaded all 46 base shards with native FP8/DeepGEMM, FP4/Marlin, FP8 MLA cache, and fused MoE LoRA.
• /v1/models returned both deepseek-ai/DeepSeek-V4-Flash and deepseek-v4-flash-60k-lora.
• On deterministic GSM8K indices 0, 50, ..., 550, base and adapter each scored 11/12 and their numeric predictions agreed on all 12 examples.
• Both model IDs retrieved 7391846250 from near the beginning of a semantic prompt containing exactly 60,000 input tokens. Each request reported prompt_tokens: 60000 and completion_tokens: 32.

This synthetic five-step adapter is not expected to improve quality. The GSM8K result is a regression sanity check that the adapter loads and executes without obvious output collapse.

Serving uses PP=4 rather than TP. DeepSeek-V4 PP support shipped in vLLM #41694, while vLLM #48697 still tracks incorrect native FP4 MoE TP output on NVIDIA GPUs.

vLLM 0.25.1 and current main do not advertise DeepseekV4ForCausalLM as LoRA-capable or provide its packed Q/KV adapter mapping. The retained vLLM patch adds only that interface registration and PEFT-to-vLLM name mapping. A clean official config.json is downloaded outside the shared training cache so stale metadata cannot select an unquantized loader.

Retained patches

Patch	Purpose
automodel_checkpoint_dequant.patch	Pass FP4 checkpoint dequantization through the pinned AutoModel recipe API.
automodel_pp_peft_checkpoint.patch	Write one PEFT shard per PP stage and allow optimizer-state omission.
automodel_composite_backend.patch	Handle the cpu:gloo,cuda:nccl control-plane backend.
automodel_uccl_teardown.patch	Free UCCL before distributed/CUDA teardown.
uccl_ipv6_oob.patch	Add IPv6 OOB support for Modal's inter-node interface.
vllm_deepseek_v4_lora.patch	Register DSv4 LoRA and map PEFT attention targets to fused vLLM modules.

Verification

• Real five-step, 128-H200 training run at 60,000 tokens.
• Real four-H200 base-versus-adapter eval and semantic 60,000-token retrieval.
• The cleaned _recipe_yaml output is byte-for-byte identical to the successful run's exact invocation.
• uv run --frozen ruff check deepseek-v4-flash-sft/automodel_modal_train.py
• uv run --frozen ruff format --check deepseek-v4-flash-sft/automodel_modal_train.py
• uv run --frozen pyright deepseek-v4-flash-sft/automodel_modal_train.py
• uv run --frozen python -m py_compile deepseek-v4-flash-sft/automodel_modal_train.py
• Modal CLI parsing checked for train, finalize, and serving validation; each exposes only required --run-id.
• No secrets or local credentials are committed.

The cleanup head was not subjected to a second 128-GPU training run. Its generated training YAML is byte-identical to the successful configuration, and the orchestration changes remove inactive options rather than changing the model recipe.

Limits

• Training uses synthetic chat data. This proves the systems path, not model quality or convergence.
• The 12-example GSM8K slice is not a model-quality benchmark.
• Serving was validated for correctness at 60k, not benchmarked for throughput or latency.
• The pinned upstream commits and local patches should be replaced as their fixes ship upstream.

Outside contributors

Link to Devin session: https://modal.devinenterprise.com/sessions/27b14e1d226e47f0b14a07b0c108a7ff

Requested by: @pawalt

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

[pawalt]

Validated the new 8-node default end to end on 8x8 H200s (64 GPUs).

• Training topology: TP=1, DP=1, PP=4, CP=16, EP=16; EFA-enabled request landed on Mellanox/RoCE and selected the matching UCCL provider.
• Sequence length: 60,000; global/local batch: 8/4; gradient accumulation: 2.
• Completed all 5 optimizer steps. Final step: loss 0.0008, grad norm 0.0014, trainer memory 25.35 GiB, throughput 15,767.93 tokens/s.
• Finalizer found all 4 PP shards, 258 adapter tensors across all 43 layers, 129 updated LoRA-B tensors, and zero missing/unexpected PEFT keys. Adapter SHA-256: 756747a4f470ca3d783353a12b58a74ebcc02574f9aea209b1d04bac422a6539.
• vLLM 0.25.1 loaded that exact adapter on 4 H200s. Base and adapter each scored 11/12 on the fixed GSM8K smoke slice with 12/12 numeric prediction agreement. Both model IDs retrieved the expected value from an actual 60,000-token prompt (prompt_tokens=60000).

Training app: https://modal.com/apps/modal-labs/peyton-agents/ap-RQAgxZiwey6e8Xg2Zyw7On
Finalizer app: https://modal.com/apps/modal-labs/peyton-agents/ap-YrXs4ImD3jjg8kJ3ghbY3M
Serving/eval app: https://modal.com/apps/modal-labs/peyton-agents/ap-qWTFKVR3yw2W4j0Sw6B4pS

The training app stopped with zero tasks after checkpointing. UCCL emitted its known nonfatal cudaFree failed: invalid argument destroy-time warnings, but the function did not fail and the independently loaded/finalized checkpoint passed all structural and serving checks.

Code/docs for this topology are in e363adb355f325d042e70e79d1ee570b3390e2f0.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🔴 Serving deployment always fails to start because the selected run is never seen by the server

The run to serve is read from a shell environment value (os.environ.get("SERVE_RUN_ID") at deepseek-v4-flash-sft/modal_train.py:46) that only exists on the machine that runs the deploy command, so when the server process starts it finds nothing and refuses to run.
Impact: The documented serving command starts a server that immediately errors out instead of serving the adapter, so serving never comes up.

Why the deploy-time environment value is empty at runtime

SERVE_RUN_ID is evaluated once at module import (deepseek-v4-flash-sft/modal_train.py:46). The README instructs SERVE_RUN_ID=<run-id> uv run --frozen modal deploy ..., which sets the variable only in the local deploy process. Modal re-imports modal_train.py inside the serving container, where SERVE_RUN_ID is not present in the environment (it is not injected via the image .env in deepseek-v4-flash-sft/dsv4_images.py:168-173 nor via a Secret), so it resolves to None. serve() then hits if not SERVE_RUN_ID: raise RuntimeError(...) at deepseek-v4-flash-sft/modal_train.py:375-376 and the web server fails to start.

The train path avoids exactly this pitfall by passing expected_nodes=N_NODES as an explicit function argument (deepseek-v4-flash-sft/modal_train.py:394) and re-validating it remotely (modal_train.py:183-184), rather than relying on the remote container re-reading N_NODES. The same technique was not applied to SERVE_RUN_ID.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[pawalt]

Cleanup is pushed in 777ade9e58463c3b2744ac7768a55da8beb6cc39.

• Renamed the launcher to deepseek-v4-flash-sft/modal_train.py.
• Reduced it from 1,496 to 410 lines by moving image construction to dsv4_images.py, checkpoint finalization to dsv4_checkpoint.py, and the AutoModel config to train_recipe.yaml.
• Removed the GSM8K/60k evaluation harness, server retry/debug machinery, cgroup diagnostics, and the validate_lora_serving entrypoint.
• Retained only the operational train, finalize, and protected serve workflows.
• Moved all six integration patches under deepseek-v4-flash-sft/patches/ without content changes.
• Rewrote the README around the current workflow and removed the Limits, validation-history, and superseded-experiment prose.

Validation:

• Ruff, Ruff format, Pyright, and py_compile pass for all three Python modules.
• Both 8-node and 16-node train CLI entrypoints resolve, as does ::finalize.
• The refactored finalizer ran against the existing 8-node checkpoint and reproduced adapter SHA 756747a4f470ca3d783353a12b58a74ebcc02574f9aea209b1d04bac422a6539, with 258 tensors across 43 layers: https://modal.com/apps/modal-labs/peyton-agents/ap-08yEqmi1E87z9nCaVRINtj
• The current source deployed successfully as example-deepseek-v4-flash-sft: https://modal.com/apps/modal-labs/peyton-agents/deployed/example-deepseek-v4-flash-sft

The endpoint remains protected with requires_proxy_auth=True; no proxy token is stored in this checkout, so the deployment was not invoked over HTTP during this cleanup pass.

[pawalt]

lgtm