From 625c8f2a687ff3e5db3f4a1805529407f3d14d66 Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Wed, 27 May 2026 21:13:48 -0400 Subject: [PATCH 1/7] Add PRD for CVS DTNI suite expansion Architecture-only PRD for the inference + training suite refactor: unified WorkloadAdapter lifecycle, Strategy + Composite topology, RAII resource handles, persistent manifest, typed config schema, and the temporal threshold predicate language. Each section labels what lands now vs what the architecture admits as a future seam. Companion design doc lives separately and is referenced by name in the PRD header. --- docs/prd/cvs-dtni-suite-expansion-prd.md | 533 +++++++++++++++++++++++ 1 file changed, 533 insertions(+) create mode 100644 docs/prd/cvs-dtni-suite-expansion-prd.md diff --git a/docs/prd/cvs-dtni-suite-expansion-prd.md b/docs/prd/cvs-dtni-suite-expansion-prd.md new file mode 100644 index 00000000..f9f13576 --- /dev/null +++ b/docs/prd/cvs-dtni-suite-expansion-prd.md @@ -0,0 +1,533 @@ +# CVS DTNI Suite Expansion — PRD + +> **Status:** Draft for team review. +> **Scope:** Architecture only — no sequencing, no slice plans, no effort estimates. Implementation rollout lives separately. Source of detail: `cvs-dtni-suite-expansion-design.md`. +> **Audience:** CVS contributors who will own, extend, or debug the inference and training suites. + +Each architectural piece below carries two explicit labels: **In scope** — what this redesign commits to — and **Future seams** — what the design admits without warping, but is not building today. + +--- + +## 1. Problem + +The CVS DTNI (data-center training and inference) suites — inference (vllm, sglang, pytorch_xdit, inferencemax) and training (megatron, jax) — have grown by copy-paste-modify. Seven framework variants, ~5000 LOC across `cvs/tests/{inference,training}/`, ~80% duplication. Each new model is a new file. Each new framework is a new lib *and* a new test file. + +Three problems compound: + +1. **Robustness.** Silent-pass on missing thresholds (inferencemax). Pass/fail direction inferred from substring `"ms"` in the metric name (any future `latency_seconds` flips comparison). `fail_test()` does not actually call `pytest.fail` — multi-step tests continue past the first failure. HF tokens logged in plaintext on every multi-node run. `UnboundLocalError` in the HF-token fixture turns a missing-file error into a crash trace. `sudo rm -rf $log_dir` runs unquoted on user-supplied paths. Containers default to `--privileged` + `seccomp=unconfined`. `docker system prune --force` on shared nodes wipes other users' containers. +2. **Coverage.** Frameworks emit per-request JSONL, per-step trajectories, Prometheus metrics. CVS regex-greps the console and discards the rest. No loss-trajectory check, no per-rank jitter, no convergence sanity — a NaN-loss training run passes if the throughput line parsed. +3. **Scale.** Module-level `inf_res_dict = {}` and `inference_dict["_test_output_dir"]` mutation pass data between tests. Re-running `-k "verify"` against an already-run benchmark is impossible — the data is gone when the pytest process exits. + +The dominant gap across all seven frameworks is "framework emits, CVS throws away" — not "framework doesn't emit." + +--- + +## 2. Goals and non-goals + +**Goals.** + +- One typed config per (framework, model). No mega-configs. No stringified-param result dicts. +- One parametrized test per suite (`test_inference.py`, `test_training.py`). New model = new config file, not new test file. +- Trajectory and per-request data are first-class and retained on disk. Verification can be re-run without re-launching workloads. +- Failures classified into five disjoint categories the operator can act on (setup / safety / pattern-matched / liveness / verification). +- Every external resource (container, GPU, secret, ssh fan-out) is RAII-managed. Cleanup is guaranteed. +- Robustness over speed-to-first-deliverable. + +**Non-goals.** + +- Unifying CVS broadly. Scoped to DTNI; cross-suite changes (e.g. `globals.error_list` removal) ride along only as preconditions. +- Quality-metric implementations (CLIP / FID / LPIPS for xDiT; lm-eval-harness, GSM8K for LLM serving). The architecture exposes the seam; the implementations are follow-ups. +- Remote ingest deployment (Prometheus remote-write, OpenSearch, S3). The on-disk event schema is ingestion-ready; deployment is a follow-up. +- MLPerf official submission path (LoadGen, SPEC PTD power, TEST01/04/05, submission packaging). The Composite pattern and `accuracy_harness` seam keep this future-feasible; nothing lands today. + +--- + +## 3. Core architecture + +### 3.1 Lifecycle — Template Method + +A single `Job` driver runs the same six steps in order for every workload, training or inference. No mode branching at the driver level. + +```python +def run(adapter, ctx): + adapter.prepare(ctx) + run = adapter.launch(ctx) + adapter.await_completion(run) # polls progress_predicate + result = adapter.parse(run, manifest) + verdict = adapter.verify(result, thresholds) + adapter.teardown(run) # always runs (RAII) +``` + +The protocol has a **closed set** of lifecycle methods (`prepare`, `launch`, `await_completion`, `parse`, `verify`, `teardown`) plus `progress_predicate`. The protocol does not gain new steps to accommodate one mode's needs. This is the guardrail that keeps a single spine safe across both training and inference. + +**In scope.** + +- One `WorkloadAdapter` protocol (Python `Protocol`), six methods + `progress_predicate`. +- One `Job` driver. No `if mode == "training"` anywhere in the driver. +- Cross-cutting behavior goes through hooks (§3.5), handles (§3.3), or adapter-internal helpers — never into the protocol. + +**Future seams.** + +- If a future requirement genuinely needs a method only one mode needs, the answer is to split into `InferenceAdapter` and `TrainingAdapter` — a mechanical refactor in known places. The protocol does not drift into a `**kwargs` puddle. + +### 3.2 Adapter topology — Strategy + Composite + +Two patterns, chosen by a factory based on workload shape. + +- **Strategy** for single-role workloads (vllm, inferencemax, xdit, single-node training). +- **Composite** for multi-role workloads (sglang disagg = prefill + decode + router + bench-client; distributed training = N symmetric workers). A `CompositeAdapter` *is itself* a `WorkloadAdapter`. The `Job` driver does not know which it has. + +Strategy adapters are the unit of reuse. `SglangServerAdapter` is used twice inside the disagg Composite (once for prefill, once for decode, differing by a CLI flag). Today's `SglangDisaggPD` is monolithic — 1200 LOC at `cvs/lib/sglang_disagg_lib.py` that cannot be partially reused. + +```python +@dataclass +class CompositeAdapter(WorkloadAdapter): + sub_adapters: list[WorkloadAdapter] + sub_role_names: list[str] + launch_model: Literal["barrier", "dag"] + barrier_cohorts: list[list[int]] # parallel within cohort; sequential between cohorts + + def progress_predicate(self, run): + for sub in self.sub_adapters: # conjunction over sub-roles + s = sub.progress_predicate(run) + if not s.ok: return s + return Ok() +``` + +Two launch models: + +- `"barrier"` — symmetric distributed training (torchrun, jax.distributed). All sub-adapters in a cohort start simultaneously to avoid racing collective bootstrap timeouts. +- `"dag"` — genuinely sequential pipelines (sglang disagg: servers → router → bench). Cohorts run in topological order; within a cohort, sub-adapters still launch in parallel. + +**In scope.** + +- `VllmAdapter`, `PytorchXditAdapter`, `InferenceMaxAdapter`, single-node training as Strategy. +- `SglangServerAdapter` (×2 in disagg), `SglangRouterAdapter`, `SglangBenchAdapter` inside a `dag`-mode `CompositeAdapter`. +- `MegatronWorkerAdapter`, `JaxWorkerAdapter` inside a `barrier`-mode `CompositeAdapter` (or as a cohort-of-1 Strategy when N=1). +- Factory per framework. The `Job` driver calls `REGISTRY[cfg.framework](...)` and gets back a `WorkloadAdapter` — never branches on topology. + +**Future seams.** + +- **MLPerf LoadGen** drops in as an additional sub-role under the existing Composite pattern — no protocol change. +- **TEST01/04/05 compliance scripts** drop in as decorators over an inner adapter — no protocol change. +- **TensorRT-LLM `trtllm-bench`** drops in as an alternative bench-client sub-role under the sglang/vllm Composite — same shape. + +### 3.3 Resource handles — RAII + +Four context-managed handles. Adapters use them; nothing else touches the underlying tools. + +| Handle | Owns | Replaces | +|---|---|---| +| `Pssh` | All node communication. Localhost is opt-in via `cluster.localhost: bool`, not heuristic. Single log seam routes through the redactor. | Inline `subprocess.run`, duplicated `_is_local_target` and `LocalPssh` across xdit files | +| `GpuPlatform` | GPU vendor / family / device_count / device_paths / visible_env. `GpuPlatform.detect(phdl)` cross-checks PCI device ID + `rocm-smi`/`amd-smi` + `rocminfo` GFX version; disagreement fails config load. `SystemFingerprint.capture(pssh)` gathers CPU / RAM / kernel / NIC / container image digest for the manifest. | Inline `rocm-smi` parsing in 4 inference test files; `"mi355"` vs `"mi355x"` literal mismatch; no automated capture of "what changed between two runs?" forensic data | +| `SecretValue` | Token storage with type-level redaction. `repr`/`str` return ``. `.reveal()` only at `--env-file` write time. | Plaintext `HF_TOKEN` strings in command lines and logs | +| `ContainerHandle` | `docker run -d` with `run_id` label; readiness probe blocks `__enter__`; `__exit__` always captures logs + dmesg + `rocm-smi` and `docker rm` by label (not `system prune`). Non-privileged default. | `docker_lib.py` helpers, ad-hoc `try/finally` cleanup blocks, `docker system prune --force`, `--privileged` default | + +**In scope.** + +- All four handles; redactor registered at the Pssh log seam. +- `GpuPlatform.detect()` as the single source of truth for GPU identity. +- `SystemFingerprint.capture(pssh)` called once at `prepare.start`. +- Sentinel-leak guard: run with `HF_TOKEN=hf_LEAK_SENTINEL_xyz`; grep full session log + in-container `/tmp/*.sh` for the sentinel — must not find. + +**Future seams.** + +- **Wall-plug power capture** (SPEC PTD 1.11.1) lands as a new handle. Adapters use it; the protocol does not change. +- **K8s / Slurm orchestration backends** slot behind the `Pssh` interface. Adapters are unchanged. +- **New GPU SKUs** add a row to `PCI_ID_TO_FAMILY` in `handles.py`. No code changes. + +### 3.4 Persistent manifest + +Per-run `//manifest.json` + append-only `events.jsonl`. Manifest is the index; events are the time-ordered stream. State lives on disk, not in module globals. + +```jsonc +// manifest.json (abbreviated) +{ + "run_id": "20260115-221400-a7f3", + "test_id": "test_inference[vllm-gpt-oss-120b-mi355x]", + "config_hash": "sha256:...", + "status": "verification_failure", + "system_desc": { + "hosts": [{ + "node": "mi355x-node-01", + "cpu": "AMD EPYC 9654 96-Core", "ram_gb": 1536, + "kernel": "6.5.0-25-generic", "os": "Ubuntu 22.04.3 LTS", + "gpus": [{ "vendor": "AMD", "family": "MI355X", "count": 8, + "vbios": "113-MI355X-...", "driver": "rocm-7.0.0" }], + "nics": [{ "model": "Mellanox CX-7", "link_gbps": 400, "count": 8 }], + "container": { "image": "rocm/vllm:7.0.0", "digest": "sha256:a91c..." } + }] + }, + "dataset_checksums": { "/datasets/gsm8k/test.jsonl": "sha256:e4f1b2..." }, + "failure": { + "category": "verification_failure", + "thresholds_failed": [ + { "kind": "Percentile", "metric": "ttft_ms", "value_seen": 78.2, + "value_required": 50.0, "op": "<=" } + ] + }, + "result": { + "samples": { "ttft_ms": [...], "itl_ms": [...] }, + "trajectory": {}, + "scalars": { "ttft_p99_ms": 78.2, "throughput_tokens_per_sec": 1184.0 }, + "status": { "qps_sweep_complete": "yes" } + } +} +``` + +`events.jsonl` is small and closed: + +```jsonc +{ "ts": "...", "run_id": "...", "test_id": "...", "node": "n3", "role": "server", + "event": "step", "source": "prometheus_poll", + "fields": { "iter": 1024, "throughput": 4823.1, "loss": 2.34 } } +``` + +Event vocabulary: `prepare.{start,done}`, `launch.{container_up,role_ready}`, `seed.logged`, `arrival.{start,end}`, `accuracy.{start,end}`, `step`, `request`, `safety.violated`, `pattern.matched`, `parse.done`, `verify.{passed,failed}`, `teardown.{start,done}`. + +**In scope.** + +- `manifest.json` and `events.jsonl` per test, on devbox disk. +- `system_desc` (via `SystemFingerprint`) and `dataset_checksums` captured at `prepare.start` — a run is reconstructible from `//` alone. +- Re-runnability: `cvs run -k "parse and verify"` re-reads manifest + events without re-launching the workload. Re-verify against relaxed thresholds against the same samples is one command. +- Single closed event vocabulary above. Adding a new event name is a schema change, not a silent producer. + +**Future seams.** + +- **Remote ingest** — Prometheus remote-write, OpenSearch index, S3 archive. The event schema is unchanged; only the sink swaps. +- **MLPerf submission archive readiness** — `system_desc` + run reconstructibility is the precondition; submission packaging is the remaining work. +- **LoadGen event slot** — `loadgen.*` is intentionally not pre-allocated; added with LoadGen integration if/when that lands. + +### 3.5 Hooks at the seam + +Pytest hooks (`conftest.py`, `pytest_runtest_makereport`, `pytest_terminal_summary`) and the Pssh log seam are the single places where cross-cutting behavior lives. + +- Failure-artifact capture (wrapping `ContainerHandle.__exit__` on the exception path). +- Cross-test results aggregation — replaces what `globals.error_list` did, via end-of-session iteration over per-test manifests. +- Secret redaction at the Pssh log seam — substring-replace any active `SecretValue.raw` before the log line is emitted. +- `FailurePatternScanner` (§4.4) registers here. + +**In scope.** + +- `pytest_runtest_makereport` writes failure context to manifest (best-effort, never propagates). +- `pytest_terminal_summary` reads every per-test manifest in the run directory and emits the cross-test summary. +- Redactor at `pssh.py:256` (the single `self.log.info(f'cmd = {full_cmd}')` line). +- `FailurePatternScanner` registered as a hook that scans during-run logs against `failure_patterns.yaml`. + +**Future seams.** + +- New cross-cutting behaviors (additional summary writers, alternative artifact destinations, in-process metric exporters) land here. They never land in the adapter protocol. + +### 3.6 Typed config schema + +Pydantic schemas, `extra = "forbid"`. Inference and training have separate registries so configs cannot route through the wrong mode. Typos fail at load, not after 20 minutes of compute. + +```python +class BaseTestConfig(BaseModel): + model_config = ConfigDict(extra="forbid") + schema_version: Literal["2"] + target_gpu: str # asserted against GpuPlatform.detect() + cluster_ref: Path + secrets: SecretsBlock + seed: SeedBlock = SeedBlock() + thresholds: list[Threshold] # see §4.2 + +class InferenceTestConfig(BaseTestConfig): + framework: Literal["vllm", "sglang_disagg", "inferencemax", "pytorch_xdit"] + params: VllmParams | SglangDisaggParams | InferenceMaxParams | PytorchXditParams + scenario: Literal["server", "offline", "single_stream", "multistream"] = "server" + arrival: ArrivalConfig | None = None + accuracy_harness: AccuracyHarnessConfig | None = None + mlperf_inference_compliance: Literal["v6.0"] | None = None + +class TrainingTestConfig(BaseTestConfig): + framework: Literal["megatron", "jax"] + params: MegatronParams | JaxParams + mlperf_training_compliance: Literal["v6.0"] | None = None +``` + +`scenario` (MLPerf-style traffic shape) and `framework` are orthogonal axes: any inference framework can serve under any scenario. `arrival` and `accuracy_harness` are inference-only optional sub-phase configs that the `Job` driver hands to `launch()` (see §4.3). `mlperf_*_compliance` is declarative only today — it documents which MLPerf round's SLO presets the config gates against; no enforcement validators. + +**Intentionally absent.** `division` (CVS lives in MLPerf-Open methodology by default), `pair_hash` (single-config paired runs make pairing structural, not by-hash), `mode` enum (presence/absence of `accuracy_harness` is the discriminator). + +**Rollout.** New schema is the only schema. A `cvs migrate-config` helper runs once per config: splits mega-configs (vllm 4-models, sglang 2-models) into one file per (framework, model), converts `result_dict[ISL=...,OSL=...,TP=...,CONC=...]` lookups into typed `Threshold` predicates (§4.2), forbids `` sentinels, stamps `schema_version: "2"`. Worked example (inferencemax): + +```jsonc +// Before +"result_dict": { + "ISL=1024,OSL=1024,TP=8,CONC=64": { "throughput_tokens_per_sec": 1200, "ttft_ms": 50, "itl_ms": 25 } +} + +// After +"thresholds": [ + { "kind": "Rate", "metric": "throughput", "per_unit": "sec", "op": ">=", "min_rate": 1200 }, + { "kind": "Percentile", "metric": "ttft_ms", "percentile": 99.0, "op": "<=", "value": 50 }, + { "kind": "Percentile", "metric": "itl_ms", "percentile": 99.0, "op": "<=", "value": 25 } +] +``` + +**In scope.** + +- `BaseTestConfig`, `InferenceTestConfig`, `TrainingTestConfig` with separate per-mode framework registries. +- `extra = "forbid"` everywhere — typo fails at `model_validate()`. +- `Threshold` union (§4.2). `target_gpu` asserted against detected GPU at config load. +- `seed`, `scenario`, `arrival`, `accuracy_harness`, `mlperf_*_compliance` fields land in the base schema even if their evaluators ship empty. +- `cvs migrate-config` helper covers every current config under `cvs/input/config_file/{training,inference}/`. + +**Future seams.** + +- **`mlperf_*_compliance` validator** — today declarative; future Pydantic validators can enforce HP-table / closed-division rules. +- **`threshold_table` syntactic sugar** for mega-configs — collapses a (ISL, OSL, concurrency) grid into a single block; expands to predicates at load. Pure sugar, no new evaluator. +- **`accuracy_harness.driver` extension** — `lm_eval_harness` is the first driver; MLPerf accuracy harnesses land alongside. + +--- + +## 4. Result and verification model + +### 4.1 Temporal result types + +Results carry explicit temporal structure rather than flattening to scalars. Without this, "verify throughput ≥ X" and "verify loss decreased monotonically over the last 1000 steps" would have to be encoded the same way — which means encoding one of them badly. + +```python +@dataclass +class WorkloadResult: + samples: dict[str, list[float]] # iid: per-request latency, per-image FPS + trajectory: dict[str, list[tuple[int, float]]] # time-ordered: (step|ts, value) + scalars: dict[str, float] # derived: P99, median, slope, final + status: dict[str, str] # categorical: completion flags, validity +``` + +- **Inference** populates `samples` (per-request latencies) and `scalars` (derived P50/P99/mean). +- **Training** populates `trajectory` (per-iter loss, throughput, step time, grad norm) and `scalars` (median throughput, final loss). +- **`status`** carries categorical fields that don't coerce to floats — completion flags (`qps_sweep_complete: "yes"`), accuracy-mode markers, and forward-compatibility slots. Empty by default; populated by `parse()`. +- **Composite** nests by role name: `{role: WorkloadResult}`. Sub-role rows are *not* flattened into the top-level container (role attribution would be lost). + +**In scope.** + +- `WorkloadResult` with all four carriers. +- Per-rank trajectories for training adapters (today's `tail -15 last-node/training.log` loses per-rank signal; redesign reads from every rank). +- Composite nesting rule: sub-role results in `per_role`; only cross-role joined data (e.g. prefill→decode handoff) at the top level. + +**Future seams.** + +- **`artifacts`** field for side-channel data (paths to generated images / video frames) — read by quality-evaluator predicates without touching the temporal carriers. + +### 4.2 Threshold predicates + +A `Threshold` is a typed predicate, not a `(metric, op, value)` triple. Six kinds, each with its own temporal semantics. + +```python +class PercentileThreshold: # over samples + percentile: float; op: Op; value: float + # MLPerf-shaped default is p99 (Server/Multistream); p90 only for SingleStream. + +class MonotonicityThreshold: # over trajectory + window: tuple[int, int] | Literal["last_quarter"] + direction: Literal["non_increasing", "non_decreasing"] + tolerance: float + +class ConvergenceThreshold: # over trajectory + target: float; epsilon: float + by_step: int | None = None # one of by_step xor by_wallclock_sec + by_wallclock_sec: float | None = None # MLPerf training's primary metric + +class StabilityThreshold: # rolling-variance bound over samples or trajectory + window_size: int; max_variance: float + +class RateThreshold: # derived from samples or trajectory + per_unit: Literal["sec", "step"]; op: Op; min_rate: float + +class GoodputThreshold: # filtered rate over samples (the MLPerf headline metric) + metric_pair: dict[Literal["ttft", "tpot"], str] + ttft_max_ms: float; tpot_max_ms: float + op: Op; min_qps: float + # Evaluates: requests/sec where (ttft <= ttft_max_ms) AND (tpot <= tpot_max_ms). + # Rate + Percentile cannot compose to express this filtered rate. +``` + +Worked example for inference (vllm Llama-2-70B, MLPerf v6.0 Interactive SLO): + +```jsonc +"thresholds": [ + { "kind": "Goodput", "metric_pair": { "ttft": "ttft_ms", "tpot": "tpot_ms" }, + "ttft_max_ms": 450.0, "tpot_max_ms": 40.0, "op": ">=", "min_qps": 600.0 }, + { "kind": "Percentile", "metric": "ttft_ms", "percentile": 99.0, "op": "<=", "value": 450.0 }, + { "kind": "Percentile", "metric": "tpot_ms", "percentile": 99.0, "op": "<=", "value": 40.0 }, + { "kind": "Stability", "metric": "ttft_ms", "window_size": 500, "max_variance": 2500.0 } +] +``` + +Worked example for training (megatron Llama 70B): + +```jsonc +"thresholds": [ + { "kind": "Rate", "metric": "throughput", "per_unit": "step", "op": ">=", "min_rate": 4800 }, + { "kind": "Monotonicity", "metric": "loss", "window": "last_quarter", "direction": "non_increasing", "tolerance": 0.02 }, + { "kind": "Convergence", "metric": "loss", "target": 2.1, "epsilon": 0.1, "by_wallclock_sec": 14400.0 }, + { "kind": "Stability", "metric": "step_time_ms", "window_size": 50, "max_variance": 25.0 } +] +``` + +Both modes can use most predicates: inference may want `Stability` to detect warmup jitter; training may want `Percentile` on step time to bound worst-case per-iter slowness. `Goodput` is inference-only. + +**In scope.** + +- All six predicates above with explicit `op` (no substring-based direction inference). +- Named metric registry — unknown metric strings fail config load. +- Per-framework "already-emitted but discarded" data populated into `WorkloadResult`: + + | Framework | Lands now | + |---|---| + | vllm | Per-request JSONL → `samples`; Prometheus queue depth + memory pressure → `trajectory` | + | sglang | Router queue trajectory; per-role startup times; decode KV-cache utilization | + | xdit-Flux | Per-step latency trajectory; per-image generation time | + | xdit-WAN | Per-frame latency; bitrate stats | + | inferencemax | Per-batch JSON (already present but ignored) | + | megatron | Full trajectory (loss, throughput, step time, grad norm); per-rank logs | + | jax | Per-step JSONL trajectory; per-host metrics from the coordinator role | + +**Future seams.** + +- **`QualityThreshold(metric, evaluator)`** — a one-screen class. The evaluator is a callable that takes `WorkloadResult` plus optional side-channel data and returns a verdict. Implementations follow: + - **CLIP / FID / LPIPS / GenEval** for xDiT-Flux image quality + - **PSNR / SSIM / VBench** for xDiT-WAN video quality + - **lm-eval-harness** (MMLU, GSM8K, HumanEval, IFEval) for LLM accuracy via the `accuracy_harness` sub-phase + - **RULER @32K / @128K** for long-context accuracy + - **HELM Safety / VHELM** for bias/toxicity +- **Memory headroom / collective comm fraction / MFU** for training — all inputs are in `WorkloadResult.trajectory`; the predicates are mechanical. Not landing today. +- **Tokens/sec/W** (`rocm-smi` package power) — indicative cross-arch perf/W; wall-plug power is separately a `Future seam` under §3.3. + +### 4.3 Progress predicates + +Small, code-defined invariants — *what it means for the test to be meaningfully running.* The `Job` driver polls these during `await_completion`; a violation becomes `safety_violation` in the manifest. + +| Adapter | `progress_predicate` | +|---|---| +| `MegatronAdapter` / `JaxAdapter` | `step_counter_increasing ∧ loss_is_finite` | +| `VllmAdapter` / `SglangServerAdapter` / `InferenceMaxAdapter` | `server_responds_to_health_probe` | +| `PytorchXditAdapter` | `process_alive ∧ no_stuck_step` | +| `CompositeAdapter` | `∧` over sub-roles' predicates | + +**Sub-phases inside `launch()`.** Optional `arrival` and `accuracy_harness` configs (§3.6) execute as *sequential sub-phases inside the adapter's `launch()`* against the same long-running SUT. They are **not** new lifecycle methods, **not** new progress predicates, and **not** separate `WorkloadAdapter` instances. The SUT's existing predicate (`server_responds_to_health_probe`) covers both phases; a sub-phase failure raises and the driver classifies as `setup_failure`. This preserves the §3.1 protocol-stability invariant: paired perf + accuracy is a single test with two clients hitting one server, not a `run_accuracy_pass()` method. + +**In scope.** + +- Per-adapter named predicates as above. +- `CompositeAdapter` conjunction over sub-roles. +- Sub-phase invocation inside `launch()` for `arrival` (Poisson sweep) and `accuracy_harness`. + +**Future seams.** + +- **Composite-level predicates** (e.g. `PerRankStepSyncWithin`, `PerRankThroughputSkewBelow`) extend the same predicate language without protocol changes — straggler-rank detection is a future addition under the existing seam. +- **Router-balance predicates** for sglang disagg (`RouterBalancePredicate(max_imbalance_pct=0.30)`) similarly extend without protocol changes. + +### 4.4 Failure pattern catalog + +Unbounded set of "bad things in log streams" (OOM, RCCL timeout, HBM ECC, PCIe AER, thermal throttle). YAML, not class hierarchy — adding a new pattern is a YAML edit, no code changes. + +```yaml +- id: oom_kill + source: dmesg + pattern: "Out of memory: Killed process" + severity: fatal + hint: "Container OOM-killed; check workload memory footprint." + +- id: rccl_timeout + source: framework_log + pattern: "NCCL.*Async operation timeout|RCCL.*timed out" + severity: fatal + hint: "RCCL/NCCL collective timed out; check network/topology and ulimit." +``` + +**In scope.** + +- `failure_patterns.yaml` seeded with ~8 high-confidence patterns (OOM, RCCL timeout, HBM ECC, PCIe AER, thermal throttle, OOM-killer dmesg, kernel panic, NCCL watchdog warning). +- `FailurePatternScanner` registered as a hook (§3.5); scans `tail -n 200` of each declared `source_path` per node, per run, periodically. +- Matches recorded as `pattern.matched` events with `{id, severity, line, node}`. + +**Future seams.** + +- Pattern additions are pure data — no code review beyond the YAML diff. +- **Per-pattern auto-remediation** (e.g. `dmesg -C` + retry on transient ECC) lands as new fields on the pattern row; the scanner already has the hook. + +### 4.5 Five-category failure taxonomy + +Each comes from a different mechanism above and they are prioritized: setup > safety > pattern > liveness > verification. + +| Category | Trigger | +|---|---| +| `setup_failure` | `prepare` or `launch` raised before workload started | +| `safety_violation` | Progress predicate broke mid-run | +| `failure_pattern_matched` | Catalog pattern hit | +| `liveness_failure` | `await_completion` timed out without progress predicate breaking | +| `verification_failure` | Threshold evaluated to `False` at end-of-test | + +**In scope.** + +- All five categories surface in `manifest.failure.category`. +- Single non-zero exit code today; category lives in the manifest. + +**Future seams.** + +- **Category-specific exit codes (2–6)** plus pytest-html badge for CI gating — small change once the manifest writes are stable. + +--- + +## 5. MLPerf positioning + +CVS adopts MLPerf **methodology** and explicitly does not adopt MLPerf **workflow choreography**. The two are separate decisions. + +| Methodology (adopted now) | Workflow (not adopted) | +|---|---| +| Poisson arrival at target QPS via `bench_serving --request-rate ` | LoadGen as the canonical traffic generator | +| p99 tail-latency gating (Server / Multistream) | MT19937-specifically seeded traces | +| Goodput-at-SLO as the headline customer metric | Submission packaging (`results/`, `measurements/`, `system_desc_id.json`) | +| Paired (perf, accuracy) verdict against the same SUT | Two separate runs sharing a `pair_hash` — pairing is **structural** (one config, sub-phases in `launch()`), not adversarial | +| Per-model SLO presets from published MLPerf rounds (Conversational + Interactive) | Strict QSL sample counts; benchmark-specific MLPerf accuracy harness | +| Time-to-target quality in wallclock for training | Reference Convergence Point (RCP) envelopes | +| `rocm-smi` package-power telemetry for relative perf/W | MLPerf Power via SPEC PTD 1.11.1 (wall-plug, audit-grade) | +| Logged-seed convergence sanity | `mlperf_logging.mllog` as a runtime dependency | + +CVS lives in the **MLPerf-Open** division by default (any framework, any model, any precision). MLPerf-Closed enforcement (HP verifier table, reference-impl framework gating, ECC/networking JSON) is out of scope — CVS users do not submit and are not adversarial. + +**Paired runs are one test, not two.** A single config with both `arrival` and `accuracy_harness` blocks produces a single `Job` lifecycle: vllm starts, `bench_serving --request-rate ` Poisson sweep runs, `lm_eval --model local-completions` runs against the same vllm, both write into one `WorkloadResult`, all thresholds (perf + accuracy) evaluate against one verdict. The MLPerf "same code for perf and accuracy" rule is satisfied by *construction*, not by *enforcement*. + +**In scope (methodology).** Every row in the left column above is reachable through the architecture in §3–§4. The vllm Llama-2-70B example in §4.2 evaluates the MLPerf v6.0 Interactive SLO directly. Per-model SLO presets (Conversational + Interactive) for the model families in production (Llama-3.1-8B/70B/405B, DeepSeek-R1, GPT-OSS-120B) are named gate presets selectable in config. + +**Future seams (workflow).** Every row in the right column drops in without warping the design: + +- **LoadGen** lands as a future sub-role under the existing Composite pattern (§3.2) +- **MLPerf accuracy harnesses** land alongside `lm_eval_harness` as additional `AccuracyHarnessConfig.driver` values (§3.6 / §4.3) +- **TEST01/04/05 compliance scripts** land as decorators over an inner adapter (§3.2) +- **Submission packaging** consumes `manifest.json` + `events.jsonl` + `system_desc` directly (§3.4) +- **SPEC PTD wall-plug power** lands as a new `PowerHandle` (§3.3) + +**Open decision for this review: wrap vs absorb.** + +- **Wrap.** `MLPerfTrainingAdapter` and `MLPerfInferenceAdapter` run the official MLPerf binaries as workloads. Two new adapters; coupling to MLPerf upstream is high; comparability with submitted numbers is exact. +- **Absorb.** Lift MLPerf workload definitions (model, dataset, target throughput, target accuracy) into typed CVS configs and run through existing per-framework adapters. New code is config + per-adapter knobs; coupling is low; comparability is approximate. + +Both paths drop in cleanly. The first move sets the mental model contributors reach for. **If the intent is MLPerf submission / direct comparability with submitted numbers → wrap. If the intent is MLPerf workloads as a coverage matrix for harness-level validation → absorb.** Decision required at this review. + +--- + +## 6. In scope now vs future seams — consolidated view + +| Section | In scope now | Future seams | +|---|---|---| +| §3.1 Lifecycle | 6-step `WorkloadAdapter` protocol; `Job` driver; no mode branching | Split into `InferenceAdapter` / `TrainingAdapter` only if a method ever needs to diverge | +| §3.2 Adapter topology | Strategy + Composite; barrier-launch for training; DAG-launch for sglang disagg | MLPerf LoadGen sub-role; TEST01/04/05 decorators; `trtllm-bench` alternative client | +| §3.3 Resource handles | `Pssh` + redactor; `GpuPlatform` + `SystemFingerprint`; `SecretValue`; `ContainerHandle` | Wall-plug `PowerHandle` (SPEC PTD); K8s/Slurm backends behind `Pssh`; new SKUs via data-only edit | +| §3.4 Manifest | `manifest.json` + `events.jsonl` on disk; `system_desc` + `dataset_checksums`; re-runnable verify | Remote ingest (Prometheus remote-write, OpenSearch, S3); MLPerf submission archive; LoadGen event slot | +| §3.5 Hooks | `pytest_runtest_makereport`, `pytest_terminal_summary`, redactor seam, `FailurePatternScanner` | Additional summary writers / artifact destinations land here, not in the protocol | +| §3.6 Typed config | Discriminated union, `extra="forbid"`, `cvs migrate-config`, `scenario` / `arrival` / `accuracy_harness` / `seed` / `mlperf_*_compliance` fields | `mlperf_*_compliance` validators; `threshold_table` syntactic sugar; new `accuracy_harness.driver` values | +| §4.1 Result types | `samples` / `trajectory` / `scalars` / `status`; per-rank trajectories; composite nesting | `artifacts` field for side-channel data | +| §4.2 Thresholds | Six predicates (Percentile, Monotonicity, Convergence, Stability, Rate, Goodput); named metric registry; framework-emitted data populated | `QualityThreshold` for CLIP/FID/LPIPS/PSNR/SSIM/VBench; lm-eval-harness, RULER, HELM via `accuracy_harness`; memory headroom / collective comm fraction / MFU thresholds; tokens/sec/W | +| §4.3 Progress predicates | Per-adapter named predicates; Composite conjunction; `arrival` and `accuracy_harness` sub-phases inside `launch()` | Composite-level predicates: `PerRankStepSyncWithin`, `PerRankThroughputSkewBelow`, `RouterBalancePredicate` | +| §4.4 Failure patterns | `failure_patterns.yaml` with ~8 seed patterns; scanner as hook | Pattern additions are pure data; per-pattern auto-remediation hooks | +| §4.5 Failure taxonomy | Five categories, prioritized; single non-zero exit code | Category-specific exit codes (2–6) + pytest-html badge | +| §5 MLPerf | Methodology adopted (Poisson, p99, Goodput, paired verdict, SLO presets, wallclock convergence, package power, seed) | Workflow adoption — LoadGen, MLPerf accuracy harnesses, TEST01/04/05, SPEC PTD power, submission packaging. **Wrap vs absorb: decision required at this review.** | + +--- + +*End of PRD.* From f6c84bac0c0bd9f8d05fe1edf09c6ca2215647e6 Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Wed, 27 May 2026 21:23:01 -0400 Subject: [PATCH 2/7] Add worked sglang disagg PD Composite example to PRD MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Concretizes the Strategy + Composite section (§3.2) with the sglang_disagg_factory: SglangServerAdapter reused twice (prefill, decode), router and bench wired into a dag-launch CompositeAdapter, composite-level RouterBalancePredicate, and PdHandoffJoinParser for the cross-role samples carrier. Replaces the 1200-LOC sglang_disagg_lib.py monolith. --- docs/prd/cvs-dtni-suite-expansion-prd.md | 57 ++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/docs/prd/cvs-dtni-suite-expansion-prd.md b/docs/prd/cvs-dtni-suite-expansion-prd.md index f9f13576..06919c45 100644 --- a/docs/prd/cvs-dtni-suite-expansion-prd.md +++ b/docs/prd/cvs-dtni-suite-expansion-prd.md @@ -99,6 +99,63 @@ Two launch models: - `"barrier"` — symmetric distributed training (torchrun, jax.distributed). All sub-adapters in a cohort start simultaneously to avoid racing collective bootstrap timeouts. - `"dag"` — genuinely sequential pipelines (sglang disagg: servers → router → bench). Cohorts run in topological order; within a cohort, sub-adapters still launch in parallel. +**Worked example: sglang disaggregated PD as a Composite.** + +The current `cvs/lib/sglang_disagg_lib.py` (1200 LOC) is replaced by a factory that builds four Strategy classes — `SglangServerAdapter` (reused for both prefill and decode roles, differing only by `mode`), `SglangRouterAdapter`, `SglangBenchAdapter` — and wires them into a single `CompositeAdapter`. The `Job` driver never sees the topology. + +```python +def sglang_disagg_factory(cfg, cluster, gpu, secrets) -> CompositeAdapter: + # 1. Per-role Strategy instances. SglangServerAdapter is reused twice. + prefills = [SglangServerAdapter(node=n, mode="prefill", + peer_addrs=PeerAddrs(decode_addrs=[d.addr for d in cluster.decode_nodes])) + for n in cluster.prefill_nodes] + decodes = [SglangServerAdapter(node=n, mode="decode", + peer_addrs=PeerAddrs(prefill_addrs=[p.addr for p in cluster.prefill_nodes])) + for n in cluster.decode_nodes] + router = SglangRouterAdapter(node=cluster.router_node, + prefill_addrs=[p.addr for p in cluster.prefill_nodes], + decode_addrs =[d.addr for d in cluster.decode_nodes]) + bench = SglangBenchAdapter(node=cluster.bench_node, + router_addr=cluster.router_node.addr, + workload=cfg.workload) + + # 2. Index every sub-adapter and lay out the DAG: servers -> router -> bench. + sub_adapters = [*prefills, *decodes, router, bench] + sub_role_names = ([f"prefill-{i}" for i in range(len(prefills))] + + [f"decode-{i}" for i in range(len(decodes))] + + ["router", "bench"]) + server_idxs = list(range(len(prefills) + len(decodes))) + router_idx = len(server_idxs) + bench_idx = router_idx + 1 + + return CompositeAdapter( + sub_adapters=sub_adapters, + sub_role_names=sub_role_names, + launch_model="dag", + barrier_cohorts=[ + server_idxs, # cohort 1: every prefill + decode in parallel + [router_idx], # cohort 2: router (after servers ready) + [bench_idx], # cohort 3: bench-client (after router ready) + ], + depends_on={router_idx: server_idxs, bench_idx: [router_idx]}, + composite_progress_predicates=[ + RouterBalancePredicate(max_imbalance_pct=0.30), # catches 90/10 routing + ], + composite_parsers=[ + PdHandoffJoinParser(), # joins prefill/decode rows on request_id + RouterTrajectoryParser(), # router queue depth trajectory + ], + ) +``` + +What this buys: + +- **Reuse.** `SglangServerAdapter` is ~200 LOC and ships once; the disagg Composite invokes it twice (`mode="prefill"`, `mode="decode"`). Today's 1200-LOC monolith cannot be partially reused. +- **Correct launch ordering.** `barrier_cohorts` enforces that all servers come up before the router, and the router before the bench client — but every prefill and every decode launches *in parallel* within cohort 1. Today's sequential `test_launch_prefill_servers` → `test_launch_decode_servers` is unnecessarily serial. +- **Composite-level signal.** `RouterBalancePredicate` lifts a Composite-only invariant (no single server taking >70% of requests) into a `safety_violation` failure category. A stuck router with empty decode queues — silently passing today — surfaces mid-run. +- **Cross-role parsing.** `PdHandoffJoinParser` joins prefill-role rows and decode-role rows on `request_id`, producing the `cross_role_samples_rows` carrier (§4.1) with `{request_id, prefill_done_ns, decode_start_ns, kv_transfer_ms, e2e_ms}`. This is the only shape that can express prefill→decode handoff latency. +- **Driver indifference.** From the `Job` driver's perspective this returns a `WorkloadAdapter`. Same six lifecycle calls. No `if framework == "sglang_disagg"` branching anywhere. + **In scope.** - `VllmAdapter`, `PytorchXditAdapter`, `InferenceMaxAdapter`, single-node training as Strategy. From 0d0dd1a9d7644ee0d9463202a161a942e7597a81 Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Wed, 27 May 2026 21:37:43 -0400 Subject: [PATCH 3/7] Add WorkloadAdapter protocol and Job driver snippets to PRD MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Concretizes the Lifecycle / Template Method section (§3.1): - Full typed WorkloadAdapter Protocol with type signatures for each of the six lifecycle methods plus progress_predicate. - Job driver with explicit failure-category classification at each raise boundary (setup / safety / liveness / pattern-matched), RAII teardown in finally, and the _await_with_progress polling loop that distinguishes safety_violation from liveness_failure. - Three-line driver-level dispatch showing the registry lookup. --- docs/prd/cvs-dtni-suite-expansion-prd.md | 91 +++++++++++++++++++++--- 1 file changed, 83 insertions(+), 8 deletions(-) diff --git a/docs/prd/cvs-dtni-suite-expansion-prd.md b/docs/prd/cvs-dtni-suite-expansion-prd.md index 06919c45..50a7b6b7 100644 --- a/docs/prd/cvs-dtni-suite-expansion-prd.md +++ b/docs/prd/cvs-dtni-suite-expansion-prd.md @@ -48,17 +48,92 @@ The dominant gap across all seven frameworks is "framework emits, CVS throws awa A single `Job` driver runs the same six steps in order for every workload, training or inference. No mode branching at the driver level. +**The `WorkloadAdapter` protocol.** Every adapter — Strategy or Composite, inference or training — satisfies this protocol. It is the closed contract between the `Job` driver and adapter implementations. + +```python +# cvs/lib/adapter_protocol.py +class WorkloadAdapter(Protocol): + def prepare(self, ctx: Context) -> None: ... + def launch(self, ctx: Context) -> AdapterRun: ... + def await_completion(self, run: AdapterRun) -> None: ... # polls progress_predicate + def progress_predicate(self, run: AdapterRun) -> ProgressStatus: ... + def parse(self, run: AdapterRun, manifest: Manifest) -> WorkloadResult: ... + def verify(self, result: WorkloadResult, thresholds: list[Threshold]) -> list[Verdict]: ... + def teardown(self, run: AdapterRun) -> None: ... +``` + +The protocol has a **closed set** of methods. It does not gain new steps to accommodate one mode's needs. Cross-cutting behavior lives in hooks (§3.5), handles (§3.3), or adapter-internal helpers — never in the protocol. This is the guardrail that keeps a single spine safe across both training and inference. + +**The `Job` driver.** Same six-step body for every workload. Failures are classified into the §4.5 five-category taxonomy at the boundary where they originate, not by post-hoc inspection of a stack trace. Teardown always runs. + +```python +# cvs/lib/job.py +class Job: + def __init__(self, adapter: WorkloadAdapter, cfg: BaseTestConfig, + cluster: ClusterConfig, gpu: GpuPlatform, secrets: Secrets): + self.adapter = adapter + self.ctx = Context(cfg=cfg, cluster=cluster, gpu=gpu, secrets=secrets, + pssh=Pssh(cluster), run_id=mint_run_id()) + self.manifest = Manifest.create(self.ctx.test_id, cfg, cluster) + + def run(self) -> Manifest: + run: AdapterRun | None = None + try: + self.manifest.append_event("prepare.start", source="job") + self.adapter.prepare(self.ctx) + + run = self.adapter.launch(self.ctx) # raises -> setup_failure + self.manifest.append_event("launch.role_ready", source="job") + + self._await_with_progress(run) # may raise safety_violation + # or liveness_failure + result = self.adapter.parse(run, self.manifest) + self.manifest.record_result(result) + + verdicts = self.adapter.verify(result, self.ctx.cfg.thresholds) + self.manifest.record_verdicts(verdicts) # status = pass | verification_failure + + except SetupFailure as e: + self.manifest.record_failure("setup_failure", evidence=e.evidence) + except SafetyViolation as e: + self.manifest.record_failure("safety_violation", + predicate_name=e.predicate, evidence=e.evidence) + except LivenessFailure as e: + self.manifest.record_failure("liveness_failure", evidence=e.evidence) + except FailurePatternMatched as e: + self.manifest.record_failure("failure_pattern_matched", + pattern_id=e.pattern_id, evidence=e.line) + finally: + if run is not None: + self.adapter.teardown(run) # RAII: always runs + self.manifest.append_event("teardown.done", source="job") + self.manifest.flush() + + return self.manifest + + def _await_with_progress(self, run: AdapterRun) -> None: + deadline = time.monotonic() + self.ctx.cfg.await_timeout_sec + while time.monotonic() < deadline: + status = self.adapter.progress_predicate(run) + if not status.ok: + raise SafetyViolation(predicate=status.predicate_name, + evidence=status.evidence) + if self.adapter.await_completion_check(run): # workload finished cleanly + return + time.sleep(self.ctx.cfg.poll_interval_sec) + raise LivenessFailure(evidence=f"timeout after {self.ctx.cfg.await_timeout_sec}s") +``` + +Driver-level dispatch — the only place that picks an adapter by framework — is three lines. The driver itself is mode-blind: + ```python -def run(adapter, ctx): - adapter.prepare(ctx) - run = adapter.launch(ctx) - adapter.await_completion(run) # polls progress_predicate - result = adapter.parse(run, manifest) - verdict = adapter.verify(result, thresholds) - adapter.teardown(run) # always runs (RAII) +def run_workload(cfg, cluster, gpu, secrets) -> Manifest: + registry = INFERENCE_REGISTRY if isinstance(cfg, InferenceTestConfig) else TRAINING_REGISTRY + adapter = registry[cfg.framework](cfg, cluster, gpu, secrets) + return Job(adapter, cfg, cluster, gpu, secrets).run() ``` -The protocol has a **closed set** of lifecycle methods (`prepare`, `launch`, `await_completion`, `parse`, `verify`, `teardown`) plus `progress_predicate`. The protocol does not gain new steps to accommodate one mode's needs. This is the guardrail that keeps a single spine safe across both training and inference. +Whether `adapter` is a `VllmAdapter` (Strategy), a four-role sglang `CompositeAdapter`, or a sixteen-rank megatron `CompositeAdapter`, `Job.run()` is identical. **In scope.** From 10f090110325d31ea6b9e8179bb15b4edbaeb043 Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Fri, 29 May 2026 01:30:34 -0400 Subject: [PATCH 4/7] Replace DTNI PRD with v1 implementation spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The original architecture PRD (cvs-dtni-suite-expansion-prd.md) was discussed in review and superseded by a scoped v1 implementation spec: - docs/prd/cvs-dtni-v1-spec.md — eight workstreams that replace today's DTNI stack (lifecycle + adapters + typed configs + manifest + binder + pytest layer + security/correctness fixes + tooling). The original PRD is preserved in git history (commits 625c8f2, f6c84ba, 0d0dd1a on this branch). Anything from it that is still load-bearing for v1 readers has been absorbed into the appropriate workstream description; the spec does not reference the deleted PRD. --- docs/prd/cvs-dtni-suite-expansion-prd.md | 665 ----------------------- docs/prd/cvs-dtni-v1-spec.md | 262 +++++++++ 2 files changed, 262 insertions(+), 665 deletions(-) delete mode 100644 docs/prd/cvs-dtni-suite-expansion-prd.md create mode 100644 docs/prd/cvs-dtni-v1-spec.md diff --git a/docs/prd/cvs-dtni-suite-expansion-prd.md b/docs/prd/cvs-dtni-suite-expansion-prd.md deleted file mode 100644 index 50a7b6b7..00000000 --- a/docs/prd/cvs-dtni-suite-expansion-prd.md +++ /dev/null @@ -1,665 +0,0 @@ -# CVS DTNI Suite Expansion — PRD - -> **Status:** Draft for team review. -> **Scope:** Architecture only — no sequencing, no slice plans, no effort estimates. Implementation rollout lives separately. Source of detail: `cvs-dtni-suite-expansion-design.md`. -> **Audience:** CVS contributors who will own, extend, or debug the inference and training suites. - -Each architectural piece below carries two explicit labels: **In scope** — what this redesign commits to — and **Future seams** — what the design admits without warping, but is not building today. - ---- - -## 1. Problem - -The CVS DTNI (data-center training and inference) suites — inference (vllm, sglang, pytorch_xdit, inferencemax) and training (megatron, jax) — have grown by copy-paste-modify. Seven framework variants, ~5000 LOC across `cvs/tests/{inference,training}/`, ~80% duplication. Each new model is a new file. Each new framework is a new lib *and* a new test file. - -Three problems compound: - -1. **Robustness.** Silent-pass on missing thresholds (inferencemax). Pass/fail direction inferred from substring `"ms"` in the metric name (any future `latency_seconds` flips comparison). `fail_test()` does not actually call `pytest.fail` — multi-step tests continue past the first failure. HF tokens logged in plaintext on every multi-node run. `UnboundLocalError` in the HF-token fixture turns a missing-file error into a crash trace. `sudo rm -rf $log_dir` runs unquoted on user-supplied paths. Containers default to `--privileged` + `seccomp=unconfined`. `docker system prune --force` on shared nodes wipes other users' containers. -2. **Coverage.** Frameworks emit per-request JSONL, per-step trajectories, Prometheus metrics. CVS regex-greps the console and discards the rest. No loss-trajectory check, no per-rank jitter, no convergence sanity — a NaN-loss training run passes if the throughput line parsed. -3. **Scale.** Module-level `inf_res_dict = {}` and `inference_dict["_test_output_dir"]` mutation pass data between tests. Re-running `-k "verify"` against an already-run benchmark is impossible — the data is gone when the pytest process exits. - -The dominant gap across all seven frameworks is "framework emits, CVS throws away" — not "framework doesn't emit." - ---- - -## 2. Goals and non-goals - -**Goals.** - -- One typed config per (framework, model). No mega-configs. No stringified-param result dicts. -- One parametrized test per suite (`test_inference.py`, `test_training.py`). New model = new config file, not new test file. -- Trajectory and per-request data are first-class and retained on disk. Verification can be re-run without re-launching workloads. -- Failures classified into five disjoint categories the operator can act on (setup / safety / pattern-matched / liveness / verification). -- Every external resource (container, GPU, secret, ssh fan-out) is RAII-managed. Cleanup is guaranteed. -- Robustness over speed-to-first-deliverable. - -**Non-goals.** - -- Unifying CVS broadly. Scoped to DTNI; cross-suite changes (e.g. `globals.error_list` removal) ride along only as preconditions. -- Quality-metric implementations (CLIP / FID / LPIPS for xDiT; lm-eval-harness, GSM8K for LLM serving). The architecture exposes the seam; the implementations are follow-ups. -- Remote ingest deployment (Prometheus remote-write, OpenSearch, S3). The on-disk event schema is ingestion-ready; deployment is a follow-up. -- MLPerf official submission path (LoadGen, SPEC PTD power, TEST01/04/05, submission packaging). The Composite pattern and `accuracy_harness` seam keep this future-feasible; nothing lands today. - ---- - -## 3. Core architecture - -### 3.1 Lifecycle — Template Method - -A single `Job` driver runs the same six steps in order for every workload, training or inference. No mode branching at the driver level. - -**The `WorkloadAdapter` protocol.** Every adapter — Strategy or Composite, inference or training — satisfies this protocol. It is the closed contract between the `Job` driver and adapter implementations. - -```python -# cvs/lib/adapter_protocol.py -class WorkloadAdapter(Protocol): - def prepare(self, ctx: Context) -> None: ... - def launch(self, ctx: Context) -> AdapterRun: ... - def await_completion(self, run: AdapterRun) -> None: ... # polls progress_predicate - def progress_predicate(self, run: AdapterRun) -> ProgressStatus: ... - def parse(self, run: AdapterRun, manifest: Manifest) -> WorkloadResult: ... - def verify(self, result: WorkloadResult, thresholds: list[Threshold]) -> list[Verdict]: ... - def teardown(self, run: AdapterRun) -> None: ... -``` - -The protocol has a **closed set** of methods. It does not gain new steps to accommodate one mode's needs. Cross-cutting behavior lives in hooks (§3.5), handles (§3.3), or adapter-internal helpers — never in the protocol. This is the guardrail that keeps a single spine safe across both training and inference. - -**The `Job` driver.** Same six-step body for every workload. Failures are classified into the §4.5 five-category taxonomy at the boundary where they originate, not by post-hoc inspection of a stack trace. Teardown always runs. - -```python -# cvs/lib/job.py -class Job: - def __init__(self, adapter: WorkloadAdapter, cfg: BaseTestConfig, - cluster: ClusterConfig, gpu: GpuPlatform, secrets: Secrets): - self.adapter = adapter - self.ctx = Context(cfg=cfg, cluster=cluster, gpu=gpu, secrets=secrets, - pssh=Pssh(cluster), run_id=mint_run_id()) - self.manifest = Manifest.create(self.ctx.test_id, cfg, cluster) - - def run(self) -> Manifest: - run: AdapterRun | None = None - try: - self.manifest.append_event("prepare.start", source="job") - self.adapter.prepare(self.ctx) - - run = self.adapter.launch(self.ctx) # raises -> setup_failure - self.manifest.append_event("launch.role_ready", source="job") - - self._await_with_progress(run) # may raise safety_violation - # or liveness_failure - result = self.adapter.parse(run, self.manifest) - self.manifest.record_result(result) - - verdicts = self.adapter.verify(result, self.ctx.cfg.thresholds) - self.manifest.record_verdicts(verdicts) # status = pass | verification_failure - - except SetupFailure as e: - self.manifest.record_failure("setup_failure", evidence=e.evidence) - except SafetyViolation as e: - self.manifest.record_failure("safety_violation", - predicate_name=e.predicate, evidence=e.evidence) - except LivenessFailure as e: - self.manifest.record_failure("liveness_failure", evidence=e.evidence) - except FailurePatternMatched as e: - self.manifest.record_failure("failure_pattern_matched", - pattern_id=e.pattern_id, evidence=e.line) - finally: - if run is not None: - self.adapter.teardown(run) # RAII: always runs - self.manifest.append_event("teardown.done", source="job") - self.manifest.flush() - - return self.manifest - - def _await_with_progress(self, run: AdapterRun) -> None: - deadline = time.monotonic() + self.ctx.cfg.await_timeout_sec - while time.monotonic() < deadline: - status = self.adapter.progress_predicate(run) - if not status.ok: - raise SafetyViolation(predicate=status.predicate_name, - evidence=status.evidence) - if self.adapter.await_completion_check(run): # workload finished cleanly - return - time.sleep(self.ctx.cfg.poll_interval_sec) - raise LivenessFailure(evidence=f"timeout after {self.ctx.cfg.await_timeout_sec}s") -``` - -Driver-level dispatch — the only place that picks an adapter by framework — is three lines. The driver itself is mode-blind: - -```python -def run_workload(cfg, cluster, gpu, secrets) -> Manifest: - registry = INFERENCE_REGISTRY if isinstance(cfg, InferenceTestConfig) else TRAINING_REGISTRY - adapter = registry[cfg.framework](cfg, cluster, gpu, secrets) - return Job(adapter, cfg, cluster, gpu, secrets).run() -``` - -Whether `adapter` is a `VllmAdapter` (Strategy), a four-role sglang `CompositeAdapter`, or a sixteen-rank megatron `CompositeAdapter`, `Job.run()` is identical. - -**In scope.** - -- One `WorkloadAdapter` protocol (Python `Protocol`), six methods + `progress_predicate`. -- One `Job` driver. No `if mode == "training"` anywhere in the driver. -- Cross-cutting behavior goes through hooks (§3.5), handles (§3.3), or adapter-internal helpers — never into the protocol. - -**Future seams.** - -- If a future requirement genuinely needs a method only one mode needs, the answer is to split into `InferenceAdapter` and `TrainingAdapter` — a mechanical refactor in known places. The protocol does not drift into a `**kwargs` puddle. - -### 3.2 Adapter topology — Strategy + Composite - -Two patterns, chosen by a factory based on workload shape. - -- **Strategy** for single-role workloads (vllm, inferencemax, xdit, single-node training). -- **Composite** for multi-role workloads (sglang disagg = prefill + decode + router + bench-client; distributed training = N symmetric workers). A `CompositeAdapter` *is itself* a `WorkloadAdapter`. The `Job` driver does not know which it has. - -Strategy adapters are the unit of reuse. `SglangServerAdapter` is used twice inside the disagg Composite (once for prefill, once for decode, differing by a CLI flag). Today's `SglangDisaggPD` is monolithic — 1200 LOC at `cvs/lib/sglang_disagg_lib.py` that cannot be partially reused. - -```python -@dataclass -class CompositeAdapter(WorkloadAdapter): - sub_adapters: list[WorkloadAdapter] - sub_role_names: list[str] - launch_model: Literal["barrier", "dag"] - barrier_cohorts: list[list[int]] # parallel within cohort; sequential between cohorts - - def progress_predicate(self, run): - for sub in self.sub_adapters: # conjunction over sub-roles - s = sub.progress_predicate(run) - if not s.ok: return s - return Ok() -``` - -Two launch models: - -- `"barrier"` — symmetric distributed training (torchrun, jax.distributed). All sub-adapters in a cohort start simultaneously to avoid racing collective bootstrap timeouts. -- `"dag"` — genuinely sequential pipelines (sglang disagg: servers → router → bench). Cohorts run in topological order; within a cohort, sub-adapters still launch in parallel. - -**Worked example: sglang disaggregated PD as a Composite.** - -The current `cvs/lib/sglang_disagg_lib.py` (1200 LOC) is replaced by a factory that builds four Strategy classes — `SglangServerAdapter` (reused for both prefill and decode roles, differing only by `mode`), `SglangRouterAdapter`, `SglangBenchAdapter` — and wires them into a single `CompositeAdapter`. The `Job` driver never sees the topology. - -```python -def sglang_disagg_factory(cfg, cluster, gpu, secrets) -> CompositeAdapter: - # 1. Per-role Strategy instances. SglangServerAdapter is reused twice. - prefills = [SglangServerAdapter(node=n, mode="prefill", - peer_addrs=PeerAddrs(decode_addrs=[d.addr for d in cluster.decode_nodes])) - for n in cluster.prefill_nodes] - decodes = [SglangServerAdapter(node=n, mode="decode", - peer_addrs=PeerAddrs(prefill_addrs=[p.addr for p in cluster.prefill_nodes])) - for n in cluster.decode_nodes] - router = SglangRouterAdapter(node=cluster.router_node, - prefill_addrs=[p.addr for p in cluster.prefill_nodes], - decode_addrs =[d.addr for d in cluster.decode_nodes]) - bench = SglangBenchAdapter(node=cluster.bench_node, - router_addr=cluster.router_node.addr, - workload=cfg.workload) - - # 2. Index every sub-adapter and lay out the DAG: servers -> router -> bench. - sub_adapters = [*prefills, *decodes, router, bench] - sub_role_names = ([f"prefill-{i}" for i in range(len(prefills))] + - [f"decode-{i}" for i in range(len(decodes))] + - ["router", "bench"]) - server_idxs = list(range(len(prefills) + len(decodes))) - router_idx = len(server_idxs) - bench_idx = router_idx + 1 - - return CompositeAdapter( - sub_adapters=sub_adapters, - sub_role_names=sub_role_names, - launch_model="dag", - barrier_cohorts=[ - server_idxs, # cohort 1: every prefill + decode in parallel - [router_idx], # cohort 2: router (after servers ready) - [bench_idx], # cohort 3: bench-client (after router ready) - ], - depends_on={router_idx: server_idxs, bench_idx: [router_idx]}, - composite_progress_predicates=[ - RouterBalancePredicate(max_imbalance_pct=0.30), # catches 90/10 routing - ], - composite_parsers=[ - PdHandoffJoinParser(), # joins prefill/decode rows on request_id - RouterTrajectoryParser(), # router queue depth trajectory - ], - ) -``` - -What this buys: - -- **Reuse.** `SglangServerAdapter` is ~200 LOC and ships once; the disagg Composite invokes it twice (`mode="prefill"`, `mode="decode"`). Today's 1200-LOC monolith cannot be partially reused. -- **Correct launch ordering.** `barrier_cohorts` enforces that all servers come up before the router, and the router before the bench client — but every prefill and every decode launches *in parallel* within cohort 1. Today's sequential `test_launch_prefill_servers` → `test_launch_decode_servers` is unnecessarily serial. -- **Composite-level signal.** `RouterBalancePredicate` lifts a Composite-only invariant (no single server taking >70% of requests) into a `safety_violation` failure category. A stuck router with empty decode queues — silently passing today — surfaces mid-run. -- **Cross-role parsing.** `PdHandoffJoinParser` joins prefill-role rows and decode-role rows on `request_id`, producing the `cross_role_samples_rows` carrier (§4.1) with `{request_id, prefill_done_ns, decode_start_ns, kv_transfer_ms, e2e_ms}`. This is the only shape that can express prefill→decode handoff latency. -- **Driver indifference.** From the `Job` driver's perspective this returns a `WorkloadAdapter`. Same six lifecycle calls. No `if framework == "sglang_disagg"` branching anywhere. - -**In scope.** - -- `VllmAdapter`, `PytorchXditAdapter`, `InferenceMaxAdapter`, single-node training as Strategy. -- `SglangServerAdapter` (×2 in disagg), `SglangRouterAdapter`, `SglangBenchAdapter` inside a `dag`-mode `CompositeAdapter`. -- `MegatronWorkerAdapter`, `JaxWorkerAdapter` inside a `barrier`-mode `CompositeAdapter` (or as a cohort-of-1 Strategy when N=1). -- Factory per framework. The `Job` driver calls `REGISTRY[cfg.framework](...)` and gets back a `WorkloadAdapter` — never branches on topology. - -**Future seams.** - -- **MLPerf LoadGen** drops in as an additional sub-role under the existing Composite pattern — no protocol change. -- **TEST01/04/05 compliance scripts** drop in as decorators over an inner adapter — no protocol change. -- **TensorRT-LLM `trtllm-bench`** drops in as an alternative bench-client sub-role under the sglang/vllm Composite — same shape. - -### 3.3 Resource handles — RAII - -Four context-managed handles. Adapters use them; nothing else touches the underlying tools. - -| Handle | Owns | Replaces | -|---|---|---| -| `Pssh` | All node communication. Localhost is opt-in via `cluster.localhost: bool`, not heuristic. Single log seam routes through the redactor. | Inline `subprocess.run`, duplicated `_is_local_target` and `LocalPssh` across xdit files | -| `GpuPlatform` | GPU vendor / family / device_count / device_paths / visible_env. `GpuPlatform.detect(phdl)` cross-checks PCI device ID + `rocm-smi`/`amd-smi` + `rocminfo` GFX version; disagreement fails config load. `SystemFingerprint.capture(pssh)` gathers CPU / RAM / kernel / NIC / container image digest for the manifest. | Inline `rocm-smi` parsing in 4 inference test files; `"mi355"` vs `"mi355x"` literal mismatch; no automated capture of "what changed between two runs?" forensic data | -| `SecretValue` | Token storage with type-level redaction. `repr`/`str` return ``. `.reveal()` only at `--env-file` write time. | Plaintext `HF_TOKEN` strings in command lines and logs | -| `ContainerHandle` | `docker run -d` with `run_id` label; readiness probe blocks `__enter__`; `__exit__` always captures logs + dmesg + `rocm-smi` and `docker rm` by label (not `system prune`). Non-privileged default. | `docker_lib.py` helpers, ad-hoc `try/finally` cleanup blocks, `docker system prune --force`, `--privileged` default | - -**In scope.** - -- All four handles; redactor registered at the Pssh log seam. -- `GpuPlatform.detect()` as the single source of truth for GPU identity. -- `SystemFingerprint.capture(pssh)` called once at `prepare.start`. -- Sentinel-leak guard: run with `HF_TOKEN=hf_LEAK_SENTINEL_xyz`; grep full session log + in-container `/tmp/*.sh` for the sentinel — must not find. - -**Future seams.** - -- **Wall-plug power capture** (SPEC PTD 1.11.1) lands as a new handle. Adapters use it; the protocol does not change. -- **K8s / Slurm orchestration backends** slot behind the `Pssh` interface. Adapters are unchanged. -- **New GPU SKUs** add a row to `PCI_ID_TO_FAMILY` in `handles.py`. No code changes. - -### 3.4 Persistent manifest - -Per-run `//manifest.json` + append-only `events.jsonl`. Manifest is the index; events are the time-ordered stream. State lives on disk, not in module globals. - -```jsonc -// manifest.json (abbreviated) -{ - "run_id": "20260115-221400-a7f3", - "test_id": "test_inference[vllm-gpt-oss-120b-mi355x]", - "config_hash": "sha256:...", - "status": "verification_failure", - "system_desc": { - "hosts": [{ - "node": "mi355x-node-01", - "cpu": "AMD EPYC 9654 96-Core", "ram_gb": 1536, - "kernel": "6.5.0-25-generic", "os": "Ubuntu 22.04.3 LTS", - "gpus": [{ "vendor": "AMD", "family": "MI355X", "count": 8, - "vbios": "113-MI355X-...", "driver": "rocm-7.0.0" }], - "nics": [{ "model": "Mellanox CX-7", "link_gbps": 400, "count": 8 }], - "container": { "image": "rocm/vllm:7.0.0", "digest": "sha256:a91c..." } - }] - }, - "dataset_checksums": { "/datasets/gsm8k/test.jsonl": "sha256:e4f1b2..." }, - "failure": { - "category": "verification_failure", - "thresholds_failed": [ - { "kind": "Percentile", "metric": "ttft_ms", "value_seen": 78.2, - "value_required": 50.0, "op": "<=" } - ] - }, - "result": { - "samples": { "ttft_ms": [...], "itl_ms": [...] }, - "trajectory": {}, - "scalars": { "ttft_p99_ms": 78.2, "throughput_tokens_per_sec": 1184.0 }, - "status": { "qps_sweep_complete": "yes" } - } -} -``` - -`events.jsonl` is small and closed: - -```jsonc -{ "ts": "...", "run_id": "...", "test_id": "...", "node": "n3", "role": "server", - "event": "step", "source": "prometheus_poll", - "fields": { "iter": 1024, "throughput": 4823.1, "loss": 2.34 } } -``` - -Event vocabulary: `prepare.{start,done}`, `launch.{container_up,role_ready}`, `seed.logged`, `arrival.{start,end}`, `accuracy.{start,end}`, `step`, `request`, `safety.violated`, `pattern.matched`, `parse.done`, `verify.{passed,failed}`, `teardown.{start,done}`. - -**In scope.** - -- `manifest.json` and `events.jsonl` per test, on devbox disk. -- `system_desc` (via `SystemFingerprint`) and `dataset_checksums` captured at `prepare.start` — a run is reconstructible from `//` alone. -- Re-runnability: `cvs run -k "parse and verify"` re-reads manifest + events without re-launching the workload. Re-verify against relaxed thresholds against the same samples is one command. -- Single closed event vocabulary above. Adding a new event name is a schema change, not a silent producer. - -**Future seams.** - -- **Remote ingest** — Prometheus remote-write, OpenSearch index, S3 archive. The event schema is unchanged; only the sink swaps. -- **MLPerf submission archive readiness** — `system_desc` + run reconstructibility is the precondition; submission packaging is the remaining work. -- **LoadGen event slot** — `loadgen.*` is intentionally not pre-allocated; added with LoadGen integration if/when that lands. - -### 3.5 Hooks at the seam - -Pytest hooks (`conftest.py`, `pytest_runtest_makereport`, `pytest_terminal_summary`) and the Pssh log seam are the single places where cross-cutting behavior lives. - -- Failure-artifact capture (wrapping `ContainerHandle.__exit__` on the exception path). -- Cross-test results aggregation — replaces what `globals.error_list` did, via end-of-session iteration over per-test manifests. -- Secret redaction at the Pssh log seam — substring-replace any active `SecretValue.raw` before the log line is emitted. -- `FailurePatternScanner` (§4.4) registers here. - -**In scope.** - -- `pytest_runtest_makereport` writes failure context to manifest (best-effort, never propagates). -- `pytest_terminal_summary` reads every per-test manifest in the run directory and emits the cross-test summary. -- Redactor at `pssh.py:256` (the single `self.log.info(f'cmd = {full_cmd}')` line). -- `FailurePatternScanner` registered as a hook that scans during-run logs against `failure_patterns.yaml`. - -**Future seams.** - -- New cross-cutting behaviors (additional summary writers, alternative artifact destinations, in-process metric exporters) land here. They never land in the adapter protocol. - -### 3.6 Typed config schema - -Pydantic schemas, `extra = "forbid"`. Inference and training have separate registries so configs cannot route through the wrong mode. Typos fail at load, not after 20 minutes of compute. - -```python -class BaseTestConfig(BaseModel): - model_config = ConfigDict(extra="forbid") - schema_version: Literal["2"] - target_gpu: str # asserted against GpuPlatform.detect() - cluster_ref: Path - secrets: SecretsBlock - seed: SeedBlock = SeedBlock() - thresholds: list[Threshold] # see §4.2 - -class InferenceTestConfig(BaseTestConfig): - framework: Literal["vllm", "sglang_disagg", "inferencemax", "pytorch_xdit"] - params: VllmParams | SglangDisaggParams | InferenceMaxParams | PytorchXditParams - scenario: Literal["server", "offline", "single_stream", "multistream"] = "server" - arrival: ArrivalConfig | None = None - accuracy_harness: AccuracyHarnessConfig | None = None - mlperf_inference_compliance: Literal["v6.0"] | None = None - -class TrainingTestConfig(BaseTestConfig): - framework: Literal["megatron", "jax"] - params: MegatronParams | JaxParams - mlperf_training_compliance: Literal["v6.0"] | None = None -``` - -`scenario` (MLPerf-style traffic shape) and `framework` are orthogonal axes: any inference framework can serve under any scenario. `arrival` and `accuracy_harness` are inference-only optional sub-phase configs that the `Job` driver hands to `launch()` (see §4.3). `mlperf_*_compliance` is declarative only today — it documents which MLPerf round's SLO presets the config gates against; no enforcement validators. - -**Intentionally absent.** `division` (CVS lives in MLPerf-Open methodology by default), `pair_hash` (single-config paired runs make pairing structural, not by-hash), `mode` enum (presence/absence of `accuracy_harness` is the discriminator). - -**Rollout.** New schema is the only schema. A `cvs migrate-config` helper runs once per config: splits mega-configs (vllm 4-models, sglang 2-models) into one file per (framework, model), converts `result_dict[ISL=...,OSL=...,TP=...,CONC=...]` lookups into typed `Threshold` predicates (§4.2), forbids `` sentinels, stamps `schema_version: "2"`. Worked example (inferencemax): - -```jsonc -// Before -"result_dict": { - "ISL=1024,OSL=1024,TP=8,CONC=64": { "throughput_tokens_per_sec": 1200, "ttft_ms": 50, "itl_ms": 25 } -} - -// After -"thresholds": [ - { "kind": "Rate", "metric": "throughput", "per_unit": "sec", "op": ">=", "min_rate": 1200 }, - { "kind": "Percentile", "metric": "ttft_ms", "percentile": 99.0, "op": "<=", "value": 50 }, - { "kind": "Percentile", "metric": "itl_ms", "percentile": 99.0, "op": "<=", "value": 25 } -] -``` - -**In scope.** - -- `BaseTestConfig`, `InferenceTestConfig`, `TrainingTestConfig` with separate per-mode framework registries. -- `extra = "forbid"` everywhere — typo fails at `model_validate()`. -- `Threshold` union (§4.2). `target_gpu` asserted against detected GPU at config load. -- `seed`, `scenario`, `arrival`, `accuracy_harness`, `mlperf_*_compliance` fields land in the base schema even if their evaluators ship empty. -- `cvs migrate-config` helper covers every current config under `cvs/input/config_file/{training,inference}/`. - -**Future seams.** - -- **`mlperf_*_compliance` validator** — today declarative; future Pydantic validators can enforce HP-table / closed-division rules. -- **`threshold_table` syntactic sugar** for mega-configs — collapses a (ISL, OSL, concurrency) grid into a single block; expands to predicates at load. Pure sugar, no new evaluator. -- **`accuracy_harness.driver` extension** — `lm_eval_harness` is the first driver; MLPerf accuracy harnesses land alongside. - ---- - -## 4. Result and verification model - -### 4.1 Temporal result types - -Results carry explicit temporal structure rather than flattening to scalars. Without this, "verify throughput ≥ X" and "verify loss decreased monotonically over the last 1000 steps" would have to be encoded the same way — which means encoding one of them badly. - -```python -@dataclass -class WorkloadResult: - samples: dict[str, list[float]] # iid: per-request latency, per-image FPS - trajectory: dict[str, list[tuple[int, float]]] # time-ordered: (step|ts, value) - scalars: dict[str, float] # derived: P99, median, slope, final - status: dict[str, str] # categorical: completion flags, validity -``` - -- **Inference** populates `samples` (per-request latencies) and `scalars` (derived P50/P99/mean). -- **Training** populates `trajectory` (per-iter loss, throughput, step time, grad norm) and `scalars` (median throughput, final loss). -- **`status`** carries categorical fields that don't coerce to floats — completion flags (`qps_sweep_complete: "yes"`), accuracy-mode markers, and forward-compatibility slots. Empty by default; populated by `parse()`. -- **Composite** nests by role name: `{role: WorkloadResult}`. Sub-role rows are *not* flattened into the top-level container (role attribution would be lost). - -**In scope.** - -- `WorkloadResult` with all four carriers. -- Per-rank trajectories for training adapters (today's `tail -15 last-node/training.log` loses per-rank signal; redesign reads from every rank). -- Composite nesting rule: sub-role results in `per_role`; only cross-role joined data (e.g. prefill→decode handoff) at the top level. - -**Future seams.** - -- **`artifacts`** field for side-channel data (paths to generated images / video frames) — read by quality-evaluator predicates without touching the temporal carriers. - -### 4.2 Threshold predicates - -A `Threshold` is a typed predicate, not a `(metric, op, value)` triple. Six kinds, each with its own temporal semantics. - -```python -class PercentileThreshold: # over samples - percentile: float; op: Op; value: float - # MLPerf-shaped default is p99 (Server/Multistream); p90 only for SingleStream. - -class MonotonicityThreshold: # over trajectory - window: tuple[int, int] | Literal["last_quarter"] - direction: Literal["non_increasing", "non_decreasing"] - tolerance: float - -class ConvergenceThreshold: # over trajectory - target: float; epsilon: float - by_step: int | None = None # one of by_step xor by_wallclock_sec - by_wallclock_sec: float | None = None # MLPerf training's primary metric - -class StabilityThreshold: # rolling-variance bound over samples or trajectory - window_size: int; max_variance: float - -class RateThreshold: # derived from samples or trajectory - per_unit: Literal["sec", "step"]; op: Op; min_rate: float - -class GoodputThreshold: # filtered rate over samples (the MLPerf headline metric) - metric_pair: dict[Literal["ttft", "tpot"], str] - ttft_max_ms: float; tpot_max_ms: float - op: Op; min_qps: float - # Evaluates: requests/sec where (ttft <= ttft_max_ms) AND (tpot <= tpot_max_ms). - # Rate + Percentile cannot compose to express this filtered rate. -``` - -Worked example for inference (vllm Llama-2-70B, MLPerf v6.0 Interactive SLO): - -```jsonc -"thresholds": [ - { "kind": "Goodput", "metric_pair": { "ttft": "ttft_ms", "tpot": "tpot_ms" }, - "ttft_max_ms": 450.0, "tpot_max_ms": 40.0, "op": ">=", "min_qps": 600.0 }, - { "kind": "Percentile", "metric": "ttft_ms", "percentile": 99.0, "op": "<=", "value": 450.0 }, - { "kind": "Percentile", "metric": "tpot_ms", "percentile": 99.0, "op": "<=", "value": 40.0 }, - { "kind": "Stability", "metric": "ttft_ms", "window_size": 500, "max_variance": 2500.0 } -] -``` - -Worked example for training (megatron Llama 70B): - -```jsonc -"thresholds": [ - { "kind": "Rate", "metric": "throughput", "per_unit": "step", "op": ">=", "min_rate": 4800 }, - { "kind": "Monotonicity", "metric": "loss", "window": "last_quarter", "direction": "non_increasing", "tolerance": 0.02 }, - { "kind": "Convergence", "metric": "loss", "target": 2.1, "epsilon": 0.1, "by_wallclock_sec": 14400.0 }, - { "kind": "Stability", "metric": "step_time_ms", "window_size": 50, "max_variance": 25.0 } -] -``` - -Both modes can use most predicates: inference may want `Stability` to detect warmup jitter; training may want `Percentile` on step time to bound worst-case per-iter slowness. `Goodput` is inference-only. - -**In scope.** - -- All six predicates above with explicit `op` (no substring-based direction inference). -- Named metric registry — unknown metric strings fail config load. -- Per-framework "already-emitted but discarded" data populated into `WorkloadResult`: - - | Framework | Lands now | - |---|---| - | vllm | Per-request JSONL → `samples`; Prometheus queue depth + memory pressure → `trajectory` | - | sglang | Router queue trajectory; per-role startup times; decode KV-cache utilization | - | xdit-Flux | Per-step latency trajectory; per-image generation time | - | xdit-WAN | Per-frame latency; bitrate stats | - | inferencemax | Per-batch JSON (already present but ignored) | - | megatron | Full trajectory (loss, throughput, step time, grad norm); per-rank logs | - | jax | Per-step JSONL trajectory; per-host metrics from the coordinator role | - -**Future seams.** - -- **`QualityThreshold(metric, evaluator)`** — a one-screen class. The evaluator is a callable that takes `WorkloadResult` plus optional side-channel data and returns a verdict. Implementations follow: - - **CLIP / FID / LPIPS / GenEval** for xDiT-Flux image quality - - **PSNR / SSIM / VBench** for xDiT-WAN video quality - - **lm-eval-harness** (MMLU, GSM8K, HumanEval, IFEval) for LLM accuracy via the `accuracy_harness` sub-phase - - **RULER @32K / @128K** for long-context accuracy - - **HELM Safety / VHELM** for bias/toxicity -- **Memory headroom / collective comm fraction / MFU** for training — all inputs are in `WorkloadResult.trajectory`; the predicates are mechanical. Not landing today. -- **Tokens/sec/W** (`rocm-smi` package power) — indicative cross-arch perf/W; wall-plug power is separately a `Future seam` under §3.3. - -### 4.3 Progress predicates - -Small, code-defined invariants — *what it means for the test to be meaningfully running.* The `Job` driver polls these during `await_completion`; a violation becomes `safety_violation` in the manifest. - -| Adapter | `progress_predicate` | -|---|---| -| `MegatronAdapter` / `JaxAdapter` | `step_counter_increasing ∧ loss_is_finite` | -| `VllmAdapter` / `SglangServerAdapter` / `InferenceMaxAdapter` | `server_responds_to_health_probe` | -| `PytorchXditAdapter` | `process_alive ∧ no_stuck_step` | -| `CompositeAdapter` | `∧` over sub-roles' predicates | - -**Sub-phases inside `launch()`.** Optional `arrival` and `accuracy_harness` configs (§3.6) execute as *sequential sub-phases inside the adapter's `launch()`* against the same long-running SUT. They are **not** new lifecycle methods, **not** new progress predicates, and **not** separate `WorkloadAdapter` instances. The SUT's existing predicate (`server_responds_to_health_probe`) covers both phases; a sub-phase failure raises and the driver classifies as `setup_failure`. This preserves the §3.1 protocol-stability invariant: paired perf + accuracy is a single test with two clients hitting one server, not a `run_accuracy_pass()` method. - -**In scope.** - -- Per-adapter named predicates as above. -- `CompositeAdapter` conjunction over sub-roles. -- Sub-phase invocation inside `launch()` for `arrival` (Poisson sweep) and `accuracy_harness`. - -**Future seams.** - -- **Composite-level predicates** (e.g. `PerRankStepSyncWithin`, `PerRankThroughputSkewBelow`) extend the same predicate language without protocol changes — straggler-rank detection is a future addition under the existing seam. -- **Router-balance predicates** for sglang disagg (`RouterBalancePredicate(max_imbalance_pct=0.30)`) similarly extend without protocol changes. - -### 4.4 Failure pattern catalog - -Unbounded set of "bad things in log streams" (OOM, RCCL timeout, HBM ECC, PCIe AER, thermal throttle). YAML, not class hierarchy — adding a new pattern is a YAML edit, no code changes. - -```yaml -- id: oom_kill - source: dmesg - pattern: "Out of memory: Killed process" - severity: fatal - hint: "Container OOM-killed; check workload memory footprint." - -- id: rccl_timeout - source: framework_log - pattern: "NCCL.*Async operation timeout|RCCL.*timed out" - severity: fatal - hint: "RCCL/NCCL collective timed out; check network/topology and ulimit." -``` - -**In scope.** - -- `failure_patterns.yaml` seeded with ~8 high-confidence patterns (OOM, RCCL timeout, HBM ECC, PCIe AER, thermal throttle, OOM-killer dmesg, kernel panic, NCCL watchdog warning). -- `FailurePatternScanner` registered as a hook (§3.5); scans `tail -n 200` of each declared `source_path` per node, per run, periodically. -- Matches recorded as `pattern.matched` events with `{id, severity, line, node}`. - -**Future seams.** - -- Pattern additions are pure data — no code review beyond the YAML diff. -- **Per-pattern auto-remediation** (e.g. `dmesg -C` + retry on transient ECC) lands as new fields on the pattern row; the scanner already has the hook. - -### 4.5 Five-category failure taxonomy - -Each comes from a different mechanism above and they are prioritized: setup > safety > pattern > liveness > verification. - -| Category | Trigger | -|---|---| -| `setup_failure` | `prepare` or `launch` raised before workload started | -| `safety_violation` | Progress predicate broke mid-run | -| `failure_pattern_matched` | Catalog pattern hit | -| `liveness_failure` | `await_completion` timed out without progress predicate breaking | -| `verification_failure` | Threshold evaluated to `False` at end-of-test | - -**In scope.** - -- All five categories surface in `manifest.failure.category`. -- Single non-zero exit code today; category lives in the manifest. - -**Future seams.** - -- **Category-specific exit codes (2–6)** plus pytest-html badge for CI gating — small change once the manifest writes are stable. - ---- - -## 5. MLPerf positioning - -CVS adopts MLPerf **methodology** and explicitly does not adopt MLPerf **workflow choreography**. The two are separate decisions. - -| Methodology (adopted now) | Workflow (not adopted) | -|---|---| -| Poisson arrival at target QPS via `bench_serving --request-rate ` | LoadGen as the canonical traffic generator | -| p99 tail-latency gating (Server / Multistream) | MT19937-specifically seeded traces | -| Goodput-at-SLO as the headline customer metric | Submission packaging (`results/`, `measurements/`, `system_desc_id.json`) | -| Paired (perf, accuracy) verdict against the same SUT | Two separate runs sharing a `pair_hash` — pairing is **structural** (one config, sub-phases in `launch()`), not adversarial | -| Per-model SLO presets from published MLPerf rounds (Conversational + Interactive) | Strict QSL sample counts; benchmark-specific MLPerf accuracy harness | -| Time-to-target quality in wallclock for training | Reference Convergence Point (RCP) envelopes | -| `rocm-smi` package-power telemetry for relative perf/W | MLPerf Power via SPEC PTD 1.11.1 (wall-plug, audit-grade) | -| Logged-seed convergence sanity | `mlperf_logging.mllog` as a runtime dependency | - -CVS lives in the **MLPerf-Open** division by default (any framework, any model, any precision). MLPerf-Closed enforcement (HP verifier table, reference-impl framework gating, ECC/networking JSON) is out of scope — CVS users do not submit and are not adversarial. - -**Paired runs are one test, not two.** A single config with both `arrival` and `accuracy_harness` blocks produces a single `Job` lifecycle: vllm starts, `bench_serving --request-rate ` Poisson sweep runs, `lm_eval --model local-completions` runs against the same vllm, both write into one `WorkloadResult`, all thresholds (perf + accuracy) evaluate against one verdict. The MLPerf "same code for perf and accuracy" rule is satisfied by *construction*, not by *enforcement*. - -**In scope (methodology).** Every row in the left column above is reachable through the architecture in §3–§4. The vllm Llama-2-70B example in §4.2 evaluates the MLPerf v6.0 Interactive SLO directly. Per-model SLO presets (Conversational + Interactive) for the model families in production (Llama-3.1-8B/70B/405B, DeepSeek-R1, GPT-OSS-120B) are named gate presets selectable in config. - -**Future seams (workflow).** Every row in the right column drops in without warping the design: - -- **LoadGen** lands as a future sub-role under the existing Composite pattern (§3.2) -- **MLPerf accuracy harnesses** land alongside `lm_eval_harness` as additional `AccuracyHarnessConfig.driver` values (§3.6 / §4.3) -- **TEST01/04/05 compliance scripts** land as decorators over an inner adapter (§3.2) -- **Submission packaging** consumes `manifest.json` + `events.jsonl` + `system_desc` directly (§3.4) -- **SPEC PTD wall-plug power** lands as a new `PowerHandle` (§3.3) - -**Open decision for this review: wrap vs absorb.** - -- **Wrap.** `MLPerfTrainingAdapter` and `MLPerfInferenceAdapter` run the official MLPerf binaries as workloads. Two new adapters; coupling to MLPerf upstream is high; comparability with submitted numbers is exact. -- **Absorb.** Lift MLPerf workload definitions (model, dataset, target throughput, target accuracy) into typed CVS configs and run through existing per-framework adapters. New code is config + per-adapter knobs; coupling is low; comparability is approximate. - -Both paths drop in cleanly. The first move sets the mental model contributors reach for. **If the intent is MLPerf submission / direct comparability with submitted numbers → wrap. If the intent is MLPerf workloads as a coverage matrix for harness-level validation → absorb.** Decision required at this review. - ---- - -## 6. In scope now vs future seams — consolidated view - -| Section | In scope now | Future seams | -|---|---|---| -| §3.1 Lifecycle | 6-step `WorkloadAdapter` protocol; `Job` driver; no mode branching | Split into `InferenceAdapter` / `TrainingAdapter` only if a method ever needs to diverge | -| §3.2 Adapter topology | Strategy + Composite; barrier-launch for training; DAG-launch for sglang disagg | MLPerf LoadGen sub-role; TEST01/04/05 decorators; `trtllm-bench` alternative client | -| §3.3 Resource handles | `Pssh` + redactor; `GpuPlatform` + `SystemFingerprint`; `SecretValue`; `ContainerHandle` | Wall-plug `PowerHandle` (SPEC PTD); K8s/Slurm backends behind `Pssh`; new SKUs via data-only edit | -| §3.4 Manifest | `manifest.json` + `events.jsonl` on disk; `system_desc` + `dataset_checksums`; re-runnable verify | Remote ingest (Prometheus remote-write, OpenSearch, S3); MLPerf submission archive; LoadGen event slot | -| §3.5 Hooks | `pytest_runtest_makereport`, `pytest_terminal_summary`, redactor seam, `FailurePatternScanner` | Additional summary writers / artifact destinations land here, not in the protocol | -| §3.6 Typed config | Discriminated union, `extra="forbid"`, `cvs migrate-config`, `scenario` / `arrival` / `accuracy_harness` / `seed` / `mlperf_*_compliance` fields | `mlperf_*_compliance` validators; `threshold_table` syntactic sugar; new `accuracy_harness.driver` values | -| §4.1 Result types | `samples` / `trajectory` / `scalars` / `status`; per-rank trajectories; composite nesting | `artifacts` field for side-channel data | -| §4.2 Thresholds | Six predicates (Percentile, Monotonicity, Convergence, Stability, Rate, Goodput); named metric registry; framework-emitted data populated | `QualityThreshold` for CLIP/FID/LPIPS/PSNR/SSIM/VBench; lm-eval-harness, RULER, HELM via `accuracy_harness`; memory headroom / collective comm fraction / MFU thresholds; tokens/sec/W | -| §4.3 Progress predicates | Per-adapter named predicates; Composite conjunction; `arrival` and `accuracy_harness` sub-phases inside `launch()` | Composite-level predicates: `PerRankStepSyncWithin`, `PerRankThroughputSkewBelow`, `RouterBalancePredicate` | -| §4.4 Failure patterns | `failure_patterns.yaml` with ~8 seed patterns; scanner as hook | Pattern additions are pure data; per-pattern auto-remediation hooks | -| §4.5 Failure taxonomy | Five categories, prioritized; single non-zero exit code | Category-specific exit codes (2–6) + pytest-html badge | -| §5 MLPerf | Methodology adopted (Poisson, p99, Goodput, paired verdict, SLO presets, wallclock convergence, package power, seed) | Workflow adoption — LoadGen, MLPerf accuracy harnesses, TEST01/04/05, SPEC PTD power, submission packaging. **Wrap vs absorb: decision required at this review.** | - ---- - -*End of PRD.* diff --git a/docs/prd/cvs-dtni-v1-spec.md b/docs/prd/cvs-dtni-v1-spec.md new file mode 100644 index 00000000..c4196121 --- /dev/null +++ b/docs/prd/cvs-dtni-v1-spec.md @@ -0,0 +1,262 @@ +# CVS DTNI v1 Specification + +**Status:** Draft for team review. + +## Goal + +Replace the current DTNI (data-center training and inference) stack — `cvs/lib/{megatron_training_lib,jax_training_lib,sglang_disagg_lib,inference_lib}.py` and `cvs/lib/inference/*.py`, plus the seven duplicated test wrappers under `cvs/tests/{training,inference}/` (~5000 LOC, ~80% duplication) — with a single lifecycle-driven framework: typed configs, on-disk manifests, deterministic cluster binder, and pytest-as-first-class test taxonomy. Land the security and correctness fixes that are currently shipping as latent bugs. Establish the foundation that lets v2 features bolt on without retrofit. + +--- + +## W1. Core lifecycle and adapter framework + +Centralize the prepare → launch → await → parse → verify → teardown lifecycle that every workload already executes in copy-pasted form today. One contract, one driver, no mode branching. + +**Deliverables** + +- `cvs/lib/adapter_protocol.py` — `WorkloadAdapter` Protocol with seven methods: `prepare`, `launch`, `await_completion`, `progress_predicate`, `parse`, `verify`, `teardown`. The Protocol shape is v1's commitment; it is not claimed to be a closed contract that handles every conceivable future workload. +- `cvs/lib/base_adapter.py` — `BaseWorkloadAdapter` (`abc.ABC`) providing concrete defaults that most adapters inherit: + - `teardown` always captures container logs, dmesg snapshots, and GPU state, then removes containers by `run_id` label. + - `await_completion` polls `progress_predicate` at a configurable interval and raises on timeout. + - `prepare` is a no-op by default. +- `cvs/lib/job.py` — `Job` driver running the six-step lifecycle. Mode-blind body: no `if mode == "training"` anywhere in the driver. Failures are classified at the boundary where they originate, not by post-hoc inspection of a stack trace. `teardown` always runs in `finally`. +- `cvs/lib/failure_taxonomy.py` — five disjoint failure categories, evaluated in priority order: + - `setup_failure` — `prepare` or `launch` raised before the workload started. + - `safety_violation` — `progress_predicate` broke mid-run. + - `failure_pattern_matched` — a pattern from `failure_patterns.yaml` (W8) hit a log stream. + - `liveness_failure` — `await_completion` timed out without the predicate breaking. + - `verification_failure` — a `Threshold` (W3) evaluated to False at end-of-test. +- `cvs/lib/registry.py` — `INFERENCE_REGISTRY` and `TRAINING_REGISTRY` keyed on the `framework` Literal from the typed config. Driver dispatch is the only place that picks an adapter by name. + +**Replaces** + +- The per-wrapper `try / finally` blocks scattered across `cvs/tests/{training,inference}/**/*.py` for cleanup. +- The module-level `globals.error_list` aggregation pattern (full removal under W6). +- Implicit failure classification ("it printed an error, so it failed") in favor of explicit category assignment at the raise site. + +**Dependencies:** none upstream. + +--- + +## W2. Six concrete adapters + +Port every existing DTNI workload onto the W1 lifecycle as a concrete adapter. Single-role workloads are direct `BaseWorkloadAdapter` subclasses. The one multi-role workload today (sglang disagg) ships as a single adapter with internal orchestration; the `CompositeAdapter` abstraction is explicitly deferred to v2.B. + +**Deliverables** + +- `VllmAdapter` — replaces `cvs/lib/inference/base.py` (the vLLM-flavored informal ABC) plus `cvs/lib/inference/vllm.py` plus four wrappers under `cvs/tests/inference/vllm/`. Lifts vLLM-specific env vars (`VLLM_USE_AITER_UNIFIED_ATTENTION`, `VLLM_ROCM_USE_AITER_MHA`, `VLLM_ROCM_USE_AITER_FUSED_MOE_A16W4`) out of the shared base into the adapter where they belong. Parse step captures per-request JSONL from `bench_serving` plus Prometheus queue depth and memory pressure (today emitted by the framework, today discarded by CVS). +- `InferenceMaxAdapter` — replaces `cvs/lib/inference/inference_max.py` plus `cvs/tests/inference/inferencemax/inferencemax_gpt_oss_120b_single.py`. Captures per-batch JSON (already emitted, currently discarded). Fixes the silent-pass-on-missing-threshold defect. +- `SglangDisaggAdapter` — replaces the 1200-LOC monolith at `cvs/lib/sglang_disagg_lib.py` plus the two near-identical wrappers `cvs/tests/inference/sglang/sglang_llama_70b_distributed.py` and `sglang_deepseek_r1_671b_distributed.py` (which differ by one line). **Single adapter, not Composite.** Internal orchestration manages the prefill / decode / router / bench roles. Parse step captures router queue trajectory, per-role startup times, decode KV-cache utilization. Refactoring into a Composite of sub-adapters is v2.B work, gated on a second multi-role workload landing. +- `PytorchXditAdapter` — replaces the two standalone wrappers `cvs/tests/inference/pytorch_xdit/{pytorch_xdit_flux1_dev_single,pytorch_xdit_wan22_14b_single}.py` (no shared library today; docker-run logic inlined per file). Eliminates the duplicated `LocalPssh`, `_is_local_target`, and `_redact_secrets` blocks across the two files. Parse step captures per-step latency trajectory for Flux and per-frame latency plus bitrate stats for WAN. +- `MegatronAdapter` — replaces `cvs/lib/megatron_training_lib.py` plus four wrappers under `cvs/tests/training/megatron/`. Single and distributed via the same adapter; `distributed: bool` is a config field, not a separate test file. Parse step captures full trajectory (loss, throughput, step time, grad norm) and per-rank logs (today only the last node's log is read, masking rank-0 failures). +- `JaxAdapter` — replaces `cvs/lib/jax_training_lib.py` plus three wrappers under `cvs/tests/training/jax/`. Parse step captures the per-step JSONL trajectory plus per-host metrics from the coordinator role. +- Delete `cvs/lib/inference_lib.py` — orphan `InferenceJobFactory` whose import target `cvs.lib.inference_max_lib` does not exist on disk; zero callers; would `ImportError` on first use. + +**Replaces** + +- The "framework emits, CVS throws away" defect for every adapter: each adapter's `parse` populates the manifest's `samples` and `trajectory` carriers (W4) with framework-native telemetry rather than tail-and-regex on console output. + +**Dependencies:** W1 (Protocol, Job, base class), W3 (typed configs for adapter constructors), W4 (manifest carriers for parse output). + +--- + +## W3. Typed config schema + +One typed config per (framework, model). Mega-configs with multiple models per file are split. Stringified-key result dicts are replaced by typed threshold predicates. Typos fail at config load, not after a 20-minute workload run. + +**Deliverables** + +- `cvs/lib/config/base.py` — `BaseTestConfig` (Pydantic, `extra="forbid"`), `schema_version: "2"`, common fields: `target_gpu`, `cluster_ref`, `secrets`, `seed`, `thresholds`. +- `InferenceTestConfig` and `TrainingTestConfig` — discriminated unions on `framework`. Configs cannot route through the wrong mode; a typo like `framework: vlm` for inference is a `model_validate()` error at load, not a silent miss after the workload runs. +- First-class `knobs` field — `attention`, `quant`, `backend`, `fused_moe`, and similar. Lives at the top level, not buried inside per-framework `params`. These become pytest markers (W6) so users can slice with queries like `pytest -m "knob_aiter and quant_fp4"`. +- Per-framework `params` classes — `VllmParams`, `SglangDisaggParams`, `InferenceMaxParams`, `PytorchXditParams`, `MegatronParams`, `JaxParams`. Framework-specific knobs live here. +- Per-framework `SweepParams` classes — `VllmSweepParams`, `SglangDisaggSweepParams`, `MegatronSweepParams`, and so on. Sweep axes are typed and validated per framework: `concurrency` is valid for inference and rejected for training; `parallelism_combos` is valid for training and rejected for inference. Cross-axis constraints are expressed as Pydantic validators (for example, `product(parallelism) == total_gpus`). +- Six typed `Threshold` predicates, each carrying an explicit `op` field so direction is never inferred from the metric name: + - `PercentileThreshold` — over `samples`; e.g. P99 TTFT ≤ 50 ms. + - `MonotonicityThreshold` — over `trajectory`; e.g. loss non-increasing in the last quarter. + - `ConvergenceThreshold` — over `trajectory`; e.g. loss reaches target ± epsilon by step N or wallclock T. + - `StabilityThreshold` — rolling-variance bound over samples or trajectory. + - `RateThreshold` — derived rate; e.g. throughput ≥ 1200 tokens/sec. + - `GoodputThreshold` — filtered rate over samples (the MLPerf-shaped headline metric); requests/sec where TTFT ≤ X and TPOT ≤ Y. +- `cvs migrate-config` one-shot tool — rewrites every existing config under `cvs/input/config_file/{training,inference}/` from the current JSON schema to v2 YAML. Splits mega-configs (the vLLM 4-model file, the sglang 2-model file). Converts `result_dict["ISL=...,OSL=...,TP=...,CONC=..."]` lookups into typed `Threshold` predicates. Forbids `` sentinels. Stamps `schema_version: "2"`. + +**Replaces** + +- The untyped `json.load` + `dict.setdefault` defaulting that currently lets typos like `percentiles_metrics` (vs `percentile_metrics`) silently fall through in `cvs/input/config_file/inference/inferencemax/*.json`. +- The substring-`"ms"` heuristic in `cvs/lib/inference/base.py` and `cvs/lib/sglang_disagg_lib.py` that infers comparison direction from the metric name. +- Stringified-param result lookup keys like `"ISL=1024,OSL=1024,TP=8,CONC=64"`. + +**Dependencies:** none upstream; W2, W4, W5, W6 depend on it. + +--- + +## W4. Manifest, sidecars, and cross-run export + +State on disk, not in module globals. Per-run manifest as the small hierarchical index; bulky numeric arrays as Parquet sidecars; events as append-only JSONL; raw logs preserved. The schema is designed for cross-run regression analysis from day one. + +**Deliverables** + +- Per-run directory layout: `/////`. Content-addressable; supports v2.A reuse-manifests without retrofit. +- `manifest.json` — small (5–50 KB) hierarchical metadata, verdicts, scalars, and pointers to sidecars. Pydantic-modeled; `cat`-able; `jq`-friendly. Eight content categories: + - **Identity and provenance** — `run_id`, `test_id`, `cell_id`, `config_hash`, `workload_hash`, `verification_hash`, `cvs_version`, `cvs_git_sha`, `framework_image_digest`, `framework_versions`, timestamps, invoker. + - **System fingerprint** — per host: CPU, memory, kernel, OS, GPUs, NICs, container runtime; topology hash for the cluster. + - **Configuration and inputs** — pointer to `config.resolved.yaml`, dataset descriptors with shas, model descriptor with weight sha, redacted env, redacted command lines, seed. + - **Phase timing** — start, end, duration, and status per lifecycle phase. + - **Verdicts and result** — overall status, failure category, per-threshold verdicts with actual + expected + margin, pattern matches, derived scalars, categorical status flags. + - **Resource summary** — per-host GPU utilization, HBM usage, power, network, OOM flag (lightweight aggregate; per-step data lives in the trajectory sidecar). + - **Sidecar pointers** — paths to samples, trajectory, events, logs, dmesg snapshots, GPU state snapshots. + - **Schema metadata** — `schema_version` for the manifest itself, versioned independently of the config schema. + + Both `workload_hash` (workload-defining inputs) and `verification_hash` (threshold and failure-pattern inputs) are recorded from day one even though v1 does not ship `--reuse-manifests`. This makes v2.A a pure tack-on with no historical migration. +- `events.jsonl` — append-only event stream with a **closed vocabulary**: `prepare.{start,done}`, `launch.{container_up,role_ready}`, `seed.logged`, `arrival.{start,end}`, `accuracy.{start,end}`, `step`, `request`, `safety.violated`, `pattern.matched`, `parse.done`, `verify.{passed,failed}`, `teardown.{start,done}`. Adding a new event name is a schema change reviewed in PR, not a free-for-all `log.info` call. +- `samples.parquet` — long-format, one row per request or sample (for example `{request_id, ts, ttft_ms, tpot_ms, itl_ms, e2el_ms, output_tokens, role, host}`). New metrics do not change schema. +- `trajectory.parquet` — long-format, one row per (step, metric, role, host). Same property. +- `config.resolved.yaml` alongside the manifest — full resolved config (post-override, post-substitution) for reproducibility. Hash for quick identity, resolved file for forensics. +- `logs/` — preserved raw artifacts per host and role: container stdout / stderr, dmesg pre and post snapshots, `rocm-smi` / `amd-smi` pre and post snapshots. Enables v2.C reparse-from-logs. +- `cvs export` — walks N run directories, joins manifest scalars with sidecar rows, writes one Parquet fact table partitioned by `experiment_id` / `cvs_git_sha` / `timestamp`. Pandas, Polars, DuckDB read it directly; no service required. + +**Replaces** + +- Module-level `inf_res_dict = {}` and `inference_dict["_test_output_dir"]` mutation that pass data between tests today. +- Tail-and-regex parsing as the only source of result data. +- The impossibility of `cvs run -k "verify"` against an already-completed benchmark (today, results die with the pytest process). + +**Dependencies:** W3 (typed configs feed the manifest); W6, W8 depend on it. + +--- + +## W5. Cluster pool, deterministic binder, sweep expansion + +Decouple "what hardware do I have" (cluster) from "what workload do I want to run, parameterized how" (config). The binder maps role requirements to physical nodes at run time, deterministically, so two runs of the same config on the same cluster pick the same hosts. + +**Deliverables** + +- Cluster file schema — a pool of nodes only: `{nodes: {hostname: {ip, user, ssh_key, gpus, labels}}}`. No role assignments live here. +- Config-side `topology.roles` — `{role_name: {count, gpus_per_node, selector}}`. The selector is a label query against `cluster.nodes[].labels`. Examples: `decode: {count: 2, gpus_per_node: 8, selector: "mi355x"}`, `router: {count: 1, gpus_per_node: 0}`. +- Deterministic binder — first-fit by cluster-file declaration order. Same cluster plus same config always yields the same bindings. This determinism is load-bearing for v2.A reuse-manifests. +- Sweep expansion semantics: + - Scalar lists default to cartesian product across axes. + - Paired axes use a single list-of-objects (no cross); each entry is one cell. Optional `name:` field on each entry becomes the cell's pytest parametrize ID. + - Multiple axis groups stack: cartesian between groups, paired within groups. Matches `pytest.mark.parametrize` semantics so the sweep block lowers cleanly to `pytest_generate_tests`. + - Topology-changing axes (P/D split, node count) carry a per-cell `topology` block; the binder re-evaluates per cell. +- Per-cell skip with reason — when a cell's role requirements cannot be satisfied by the cluster pool, the cell is marked `status: skipped` in its manifest with a concrete reason (for example `insufficient_nodes (need 4, have 3)`). Other cells in the sweep proceed normally. A user with a small dev cluster gets useful partial coverage instead of an error. + +**Replaces** + +- Role-list fields baked into per-test configs today (`prefill_node_list`, `decode_node_list`, `proxy_router_node`, `benchmark_serv_node` in `cvs/input/config_file/inference/sglang/*.json`) that tie configs to specific hostnames. +- Ad-hoc per-file fixture logic for parametrization (the `pytest_generate_tests` block duplicated across four vLLM wrappers). + +**Dependencies:** W3 (typed sweep params); W6 and W8 depend on it. + +--- + +## W6. Pytest layer and test taxonomy + +Pytest tests are first-class user surface: every claim about a workload is a named, sliceable test function. One workload run produces many independent pytest assertions by reading the manifest. Tier structure equals directory tree. + +**Deliverables** + +- Directory tree under `cvs/tests/`: + - `logistics/` — claims that every config exercises (image pullable, container up, role ready, no orphan containers, dmesg clean). + - `training/` — training-kind-only claims (loss finite, trajectory monotonic in expected window); subdir `training/test_distributed.py` for distributed-only claims (per-rank step sync, no straggler, collective health). + - `inference/` — inference-kind-only claims (server health, request success rate, no 5xx burst); subdirs for `disagg` (router balance, P→D handoff, KV-cache utilization) and `distributed`. + - `frameworks/` — one file per framework (`test_vllm.py`, `test_sglang.py`, `test_inferencemax.py`, `test_pytorch_xdit.py`, `test_megatron.py`, `test_jax.py`) for framework-specific knob assertions (AITER flags applied, XLA flags applied, attention backend matches the knob). + - `benchmarks/` — one file per claim family (`test_throughput.py`, `test_latency.py`, `test_accuracy.py`, `test_convergence.py`). Opt-in via the config's `benchmarks: [...]` list. + - `models/` — model-family edge cases (rarely populated; documents known quirks). +- Session-scoped `workload_run` fixture in `cvs/tests/conftest.py` — runs the `Job(adapter, cfg).run()` lifecycle once per (config, sweep cell), yields the manifest. One workload run feeds many test functions. +- Auto-applied pytest markers derived from config fields. Naming convention: `tier_1` through `tier_6`, `framework_`, `workload_`, `topology_`, `model_`, `knob__`, `benchmark_`, `gpu_`. Registered via `pytest_configure` so `-m` queries do not warn about unknown markers. +- `collect-skip` hook in `pytest_collection_modifyitems` — items whose tier predicate does not match the cell's config are **deselected**, not skipped. Keeps `--collect-only` output sane: a vLLM cell does not show fifty deselected megatron / jax / sglang tests. +- `requires_benchmark("name")` decorator — tier-5 tests opt in via the config's `benchmarks` list. A vLLM config that declares `benchmarks: [throughput, ttft_p99]` runs the matching tier-5 functions and nothing else from tier 5. +- `pytest_terminal_summary` hook iterating per-test manifests at session end. Replaces module-level `globals.error_list` aggregation. +- One parametrized test pattern per suite consuming the typed config; adding a new model means adding a new config file, not a new test file. + +**Replaces** + +- The ~1120 LOC of byte-identical fixture boilerplate duplicated across seven training wrappers (the `cluster_file`, `training_config_file`, `cluster_dict`, `training_dict`, `model_params_dict`, `hf_token`, `phdl` fixture block). +- The pattern of one test file per (framework, model, single/distributed) tuple — collapsed into one parametrized test per suite. +- `globals.error_list = []` plus `fail_test()` plus `update_test_result()` across `cvs/lib/*_lib.py`. + +**Dependencies:** W1 (Job and adapters), W3 (typed configs for marker derivation), W4 (manifest as fixture output), W5 (sweep expansion lowering to `pytest_generate_tests`). + +--- + +## W7. Security and correctness fixes + +Latent defects in today's code that are independent of the broader redesign. Ship as a standalone PR series before the rest of v1 if scheduling permits — none of these block on the architecture work. + +**Deliverables** + +- `SecretValue` class — token storage with type-level redaction. `repr` and `str` return ``; `.reveal()` is only called at `--env-file` write time. Removes plaintext HF tokens from logs (currently logged on every multi-node run via `phdl.exec` debug printing). +- `ContainerHandle` context manager — `docker run -d` with a `run_id` label; readiness probe blocks `__enter__`; `__exit__` always captures logs plus dmesg plus GPU state, then removes containers by label. Non-privileged default. Removes the `--privileged + seccomp=unconfined` default and the `docker system prune --force` call from `cvs/lib/docker_lib.py` (the prune currently wipes other users' containers on shared nodes). +- `fail_test()` actually calls `pytest.fail` — the current implementation appends to `globals.error_list` and returns; multi-step tests march past the first failure. +- Explicit `op:` field in `Threshold` (delivered as part of W3 but listed here for completeness) — removes the substring-`"ms"` direction inference that would flip comparison for any future metric name containing those letters (for example `latency_seconds`). +- `inferencemax` adapter (W2) raises on missing threshold instead of silently passing. +- HF-token fixture refactor — the current `UnboundLocalError` on a missing token file becomes a clean `pytest.skip` with a clear reason. +- `mi355` / `mi355x` literal normalization via a single GPU-family detection path. Today four files have inconsistent literals. +- Quote `$log_dir` in every `sudo rm -rf` call (today: `cvs/lib/docker_lib.py` runs unquoted on user-supplied paths). +- Sentinel-leak CI test — run a representative test with `HF_TOKEN=hf_LEAK_SENTINEL_xyz`; grep the full session log plus every in-container `/tmp/*.sh`. Fails if the sentinel appears anywhere. + +**Replaces** + +- The relevant defect lines in `cvs/lib/{sglang_disagg_lib.py, inference/base.py, docker_lib.py}` and the per-test HF-token fixture code paths. + +**Dependencies:** none. Can land first. + +--- + +## W8. Tooling and documentation + +User-facing CLI commands and reference documentation that make the v1 surface usable without code-diving. + +**Deliverables** + +- `cvs plan` — dry-run binder. Parses config and cluster, expands the sweep, runs the binder, applies pytest collection rules, prints the planned matrix (cells × bindings × selected tests × skip reasons). Exits without launching any adapter methods. Same code path as `cvs run` minus execution. +- `cvs export` (delivered under W4; listed here for the user-facing surface) — flattens N run directories into one Parquet fact table for ad-hoc analysis. +- `cvs migrate-config` (delivered under W3; listed here for the user-facing surface) — one-shot rewriter for existing JSON configs to v2 YAML. +- `failure_patterns.yaml` — seed catalog of approximately eight entries for the failure-pattern scanner registered as a pytest hook. Each row carries `id`, `source` (`dmesg` or `framework_log`), `pattern` (regex), `severity` (`fatal` or `warn`), and `hint`. Seed coverage: OOM-killer, RCCL / NCCL collective timeout, HBM ECC, PCIe AER, thermal throttle, kernel panic, NCCL watchdog warning, container OOM. Adding a pattern is a YAML edit; no code changes. +- `FailurePatternScanner` — runs as a pytest hook (W6 seam); tails declared sources during the run; matches recorded as `pattern.matched` events with `{id, severity, line, node}`. +- User-facing documentation under `docs/`: + - Config schema reference (per-framework `params` and `SweepParams`, `Threshold` predicates, `topology.roles`). + - Pytest tier reference (what each directory covers, how to add a tier-5 benchmark, how a benchmark is opted in via config). + - Marker reference (full list with derivation rules). + - Sweep semantics (cartesian default, paired via list-of-objects, stacked axes, topology-changing cells). + - "How to add a new framework" walkthrough (new adapter, registry entry, params class, sweep params class, optional framework-specific test file). + +**Dependencies:** W3 (`cvs migrate-config`), W4 (`cvs export`), W5 (`cvs plan`), W6 (pattern scanner hook seam). + +--- + +## Migration story + +- A single one-shot run of `cvs migrate-config` rewrites every JSON config under `cvs/input/config_file/{training,inference}/` to v2 YAML. No backwards-compatible reader path. The new schema is the only schema. +- All seven existing test wrappers under `cvs/tests/{training,inference}/` are deleted. The new parametrized tests subsume them. +- The `cvs run` CLI surface is unchanged — same flags (`--cluster_file`, `--config_file`), same invocation pattern. Existing automation continues to work. +- New per-framework markers and the tier directory layout are additive on the pytest CLI side. Queries like `pytest -m "framework_vllm"` become possible; nothing previously possible breaks. + +--- + +## Verification + +Per workstream: + +- **W1** — adapter Protocol conformance tests using fake adapters; `Job` driver behavior under each failure-class injection (raise from `prepare` → setup; broken predicate mid-run → safety; pattern match → pattern; timeout → liveness; failing threshold → verification). +- **W2** — per-adapter unit tests with fake `Pssh`; per-adapter end-to-end run of the smallest workload (for example vLLM gpt-oss-120b at concurrency 16) producing a complete manifest with `status: complete`. +- **W3** — `model_validate` against every migrated config; reject every malformed example fixture; `cvs migrate-config` round-trips every shipped config without semantic loss. +- **W4** — Pydantic round-trip for the manifest; Parquet schema test for samples and trajectory; closure test for the events vocabulary; one real run produces manifest plus sidecars that `pd.read_parquet(samples)` consumes with expected columns. +- **W5** — deterministic binding asserted across 100 randomized configs against a fixed cluster (same binding twice); per-cell skip-with-reason for under-resourced cells; `cvs plan` output matches `cvs run` actual bindings on a real cluster. +- **W6** — marker application verified against synthetic configs; `collect-skip` correctness for each tier predicate; `pytest --collect-only` for the example sweep prints the expected ID set. +- **W7** — sentinel-leak CI test; `--privileged` flag absent from every docker invocation in the codebase (grep-asserted); shared-host test confirms other users' containers are untouched. +- **W8** — `cvs plan` golden-file tests; `cvs export` schema test; a nightly run produces a Parquet that opens in a notebook. + +Suite-level: a representative subset of today's configs (one of each shape — vLLM gpt-oss-120b, sglang Llama-70B disagg, megatron 8B distributed, jax 70B single) runs to `status: complete` on v1 CVS, with manifests inspected manually for correctness. + +--- + +## Workstream dependencies + +- W7 has no upstream dependencies — ship first as a standalone PR series. +- W1 enables W2 (adapters subclass the base) and W6 (the fixture runs `Job(adapter)`). +- W3 enables W2 (adapter constructors take typed configs), W6 (markers derived from config fields), and W5 (sweep params per framework). +- W4 enables W6 (the fixture yields the manifest) and W8 (`cvs export` reads the manifest tree). +- W5 enables W6 (sweep block lowers to `pytest_generate_tests`) and W8 (`cvs plan` is the binder's dry-run mode). +- W8 depends on W3, W4, W5, and W6 — ships last. From 6478ab56f46b9a56d58a7915199e375791be9f60 Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Fri, 29 May 2026 01:50:40 -0400 Subject: [PATCH 5/7] Add digestible pr-body.md for PR #189 The v1 spec (cvs-dtni-v1-spec.md) is dense prose; reviewer feedback asked for a more digestible PR-body version with snippets, mermaid diagrams, file outlines, and concrete walkthroughs. This new file becomes the PR body content; the spec stays as the long-form prose reference. pr-body.md adds: - end-to-end data flow mermaid diagram - sglang before/after with real LOC numbers - one config -> N pytest IDs walkthrough (cvs plan + pytest --collect-only) - lifecycle + failure-classification mermaid diagram with Protocol code and Job.run() body - before/after lib directory tree with LOC counts; pytest tier tree; class hierarchy mermaid - marker derivation table - cluster + binder walkthrough with the insufficient-nodes skip case - sample manifest.json (~50 lines, real values), samples/trajectory parquet column schemas, events vocabulary table - pandas snippet for cross-run P99 TTFT regression by git SHA - three sweep walkthroughs (cartesian, paired-topology, constraint-validated) - six Threshold predicates as YAML - end-to-end safety_violation failure walkthrough - W7 before/after table for the 8 security/correctness fixes - workstream DAG mermaid Anchor links throughout point back into v1-spec.md for full prose. --- docs/prd/pr-body.md | 781 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 781 insertions(+) create mode 100644 docs/prd/pr-body.md diff --git a/docs/prd/pr-body.md b/docs/prd/pr-body.md new file mode 100644 index 00000000..0438da5b --- /dev/null +++ b/docs/prd/pr-body.md @@ -0,0 +1,781 @@ +# CVS DTNI v1 — implementation spec (digestible) + +**Status:** Draft for team review. Full prose lives in [`cvs-dtni-v1-spec.md`](cvs-dtni-v1-spec.md); this file is the PR body and the entry point. + +--- + +## TLDR + +Two of CVS's real sglang test wrappers are **429 lines each**. They differ by **one line** — `'llama-70b'` vs `'deepseek-r1'` on line 297. v1 collapses both into one parametrized test consuming two YAMLs, deletes `cvs/lib/sglang_disagg_lib.py` (1261 LOC), and stops the HF token from being logged in `phdl.exec` debug output on every multi-node run. Net change: ~**−5000 LOC** across `cvs/lib/` and `cvs/tests/`, **+1** lifecycle driver, **+6** adapters, **+6** typed threshold predicates, **+1** content-addressable manifest tree that pandas / DuckDB read directly. + +The matrix is the artifact: pick any axis a benchmark engineer cares about — framework, model, knob (`aiter`, `fa`, `fp4`, `mooncake`), threshold, git SHA, cluster — and `pytest -m "..."` slices it on day one. + +**Reading order if you only have N minutes:** + +- **5 min:** §1 data flow → §2 sglang before/after → §13 W7 security fixes (independently shippable) +- **15 min:** add §3 one config → N pytest IDs → §8 manifest + cross-run query → §11 failure walkthrough +- **Full:** §1–§16 below, with anchors into [`cvs-dtni-v1-spec.md`](cvs-dtni-v1-spec.md) for the long form + +--- + +## §1. End-to-end data flow + +One picture covers the whole thing. + +```mermaid +flowchart LR + cfg["config.yaml
cluster.json"] --> plan["cvs plan
(sweep expansion + binder)"] + plan -->|"N cells"| job["Job.run per cell
prepare → launch → await → parse → verify → teardown"] + job --> art["per-run dir:
manifest.json
samples.parquet
trajectory.parquet
events.jsonl
config.resolved.yaml
logs/"] + art --> pytest["pytest reads manifest
N test IDs per cell"] + art --> export["cvs export"] + export --> fact["fact.parquet
(M runs × all metrics)"] + fact --> nb["notebook /
dashboard /
regression alert"] +``` + +The persistent artifacts are everything under the per-run dir. `cvs plan` and pytest invocation are ephemeral. `cvs export` is the data-science seam — flatten N runs into one Parquet fact table; pandas/DuckDB consume it without a service. + +Full prose: [W4 in spec](cvs-dtni-v1-spec.md#w4-manifest-sidecars-and-cross-run-export), [W5 in spec](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). + +--- + +## §2. The sglang before/after + +The single most persuasive change in the PR. + +**Before** (`cvs/tests/inference/sglang/`): + +```text +sglang_llama_70b_distributed.py 429 LOC +sglang_deepseek_r1_671b_distributed.py 429 LOC + ───────── + 858 LOC +``` + +These two files are **byte-identical except for line 297**: + +```python +# sglang_llama_70b_distributed.py:297 +bp_dict = benchmark_params_dict['llama-70b'] + +# sglang_deepseek_r1_671b_distributed.py:297 +bp_dict = benchmark_params_dict['deepseek-r1'] +``` + +Plus the library they both consume: `cvs/lib/sglang_disagg_lib.py` at **1261 LOC** — a single monolithic class that cannot be partially reused. + +**After:** + +```text +cvs/tests/inference/test_inference.py (one parametrized test) +cvs/input/config_file/inference/sglang/llama_70b_disagg.yaml (~80 LOC YAML) +cvs/input/config_file/inference/sglang/deepseek_r1_disagg.yaml (~80 LOC YAML) +cvs/lib/adapters/sglang_disagg.py (~400 LOC adapter) +``` + +**Net delta:** −2119 LOC of duplicated Python and monolithic library; +1 reusable adapter; +N typed configs (one per model). The same shape applies to the other 14 wrappers (see §5). + +Full prose: [W2 in spec](cvs-dtni-v1-spec.md#w2-six-concrete-adapters). + +--- + +## §3. One config → N pytest IDs + +The matrix story, end-to-end. + +### 3a. The config + +```yaml +# cvs/input/config_file/inference/vllm/gpt-oss-120b_mi355x_aiter.yaml +schema_version: "2" +test_id: vllm_gpt_oss_120b_mi355x_aiter +target_gpu: mi355x + +framework: vllm +workload_kind: inference +topology: + roles: + server: {count: 1, gpus_per_node: 8, selector: "mi355x"} + +model: gpt-oss-120b +knobs: + attention: aiter + quant: fp4 + backend: vllm-native + +params: + tensor_parallelism: 1 + max_model_length: 9216 + num_prompts: 3200 + +sweep: + concurrency: [16, 32, 64] + isl_osl: + - {isl: 1024, osl: 1024, name: balanced} + - {isl: 4096, osl: 128, name: prefill_heavy} + +benchmarks: [throughput, ttft_p99, tpot_p99] + +thresholds: + - {kind: Rate, metric: throughput, per_unit: sec, op: ">=", min_rate: 1200} + - {kind: Percentile, metric: ttft_ms, percentile: 99, op: "<=", value: 50} + - {kind: Percentile, metric: tpot_ms, percentile: 99, op: "<=", value: 25} +``` + +### 3b. `cvs plan` output + +```text +$ cvs plan --cluster cluster.json --config gpt-oss-120b_mi355x_aiter.yaml + +6 cells (concurrency × isl_osl, cartesian): + +Cell Bindings Tests Markers +────────────────────────────────────── ─────────────── ────── ───────────────────────────────────────── +[balanced-conc16] server=[n1] 13 framework_vllm, model_gpt_oss_120b, +[balanced-conc32] server=[n1] 13 knob_attention_aiter, knob_quant_fp4, +[balanced-conc64] server=[n1] 13 topology_single, workload_inference, +[prefill_heavy-conc16] server=[n1] 13 gpu_mi355x, benchmark_throughput, +[prefill_heavy-conc32] server=[n1] 13 benchmark_ttft_p99, benchmark_tpot_p99, +[prefill_heavy-conc64] server=[n1] 13 tier_1, tier_3, tier_4, tier_5 + +Tests collected per cell (after collect-skip): + logistics/test_prepare.py::test_image_pullable + logistics/test_launch.py::test_container_up + logistics/test_launch.py::test_role_ready + logistics/test_teardown.py::test_no_orphans + logistics/test_teardown.py::test_dmesg_clean + inference/test_serving.py::test_server_health + inference/test_serving.py::test_request_success_rate + frameworks/test_vllm.py::test_aiter_flags_active + frameworks/test_vllm.py::test_attention_backend_matches[aiter] + benchmarks/test_throughput.py::test_throughput_min + benchmarks/test_latency.py::test_ttft_p99 + benchmarks/test_latency.py::test_tpot_p99 + models/test_gpt_oss.py::test_quant_conversion_consistent + +Total: 78 pytest IDs (6 cells × 13 tests). +Estimated wall time: ~24 min (based on 3 prior matching runs). +``` + +### 3c. CLI slicing — three real queries + +```bash +# All vLLM + AITER cells across every model and concurrency +cvs run -m "framework_vllm and knob_attention_aiter" + +# Just the P99 TTFT claim across the whole nightly sweep +cvs run -m "tier_5 and benchmark_ttft_p99" + +# Distributed training cells only, just convergence claims, FP8 quant only +cvs run -m "workload_training and topology_distributed and knob_quant_fp8 and benchmark_convergence" +``` + +Each query lowers to standard `pytest -m` (markers auto-applied at collection from config fields — see §6 derivation table). Slicing along framework, model, knob, topology, benchmark, GPU, or tier all work on day one. + +Full prose: [W6 in spec](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). + +--- + +## §4. The lifecycle and its driver + +```mermaid +flowchart TD + prepare["prepare()"] --> launch["launch()"] + launch --> awaitc["await_completion()
(polls progress_predicate)"] + awaitc --> parse["parse()"] + parse --> verify["verify()
(evaluates thresholds)"] + verify --> teardown["teardown()
(RAII; always runs)"] + + prepare -.->|"raises"| setup["setup_failure"] + launch -.->|"raises"| setup + awaitc -.->|"predicate broke"| safety["safety_violation"] + awaitc -.->|"timeout"| liveness["liveness_failure"] + parse -.->|"pattern hit"| pattern["failure_pattern_matched"] + verify -.->|"threshold False"| verif["verification_failure"] + + setup --> teardown + safety --> teardown + liveness --> teardown + pattern --> teardown + verif --> teardown +``` + +The `Job` driver runs the same six-step body for every workload — no `if mode == "training"` branching. The five failure categories are classified at the boundary where they originate, not by post-hoc stack-trace inspection. + +### The Protocol + +```python +# cvs/lib/adapter_protocol.py +class WorkloadAdapter(Protocol): + def prepare(self, ctx: Context) -> None: ... + def launch(self, ctx: Context) -> AdapterRun: ... + def await_completion(self, run: AdapterRun) -> None: ... + def progress_predicate(self, run: AdapterRun) -> ProgressStatus: ... + def parse(self, run: AdapterRun, manifest: Manifest) -> WorkloadResult: ... + def verify(self, result: WorkloadResult, thresholds: list[Threshold]) -> list[Verdict]: ... + def teardown(self, run: AdapterRun) -> None: ... +``` + +### The driver + +```python +# cvs/lib/job.py +class Job: + def run(self) -> Manifest: + run = None + try: + self.adapter.prepare(self.ctx) # raises -> setup_failure + run = self.adapter.launch(self.ctx) # raises -> setup_failure + self._await_with_progress(run) # raises -> safety / liveness + result = self.adapter.parse(run, self.manifest) + verdicts = self.adapter.verify(result, self.ctx.cfg.thresholds) + self.manifest.record_verdicts(verdicts) # status -> pass | verification_failure + except SetupFailure as e: + self.manifest.record_failure("setup_failure", e.evidence) + except SafetyViolation as e: + self.manifest.record_failure("safety_violation", e.predicate, e.evidence) + except LivenessFailure as e: + self.manifest.record_failure("liveness_failure", e.evidence) + except FailurePatternMatched as e: + self.manifest.record_failure("failure_pattern_matched", e.pattern_id, e.line) + finally: + if run is not None: + self.adapter.teardown(run) # RAII: always runs + self.manifest.flush() + return self.manifest +``` + +Full prose: [W1 in spec](cvs-dtni-v1-spec.md#w1-core-lifecycle-and-adapter-framework). + +--- + +## §5. What changes in the repo + +### `cvs/lib/` before vs after + +```text +BEFORE LOC AFTER LOC +───────────────────────────────────────────────── ───── ────────────────────────────────────────── ───── +cvs/lib/sglang_disagg_lib.py 1261 cvs/lib/adapter_protocol.py ~30 +cvs/lib/megatron_training_lib.py ~830 cvs/lib/base_adapter.py ~150 +cvs/lib/jax_training_lib.py ~830 cvs/lib/job.py ~120 +cvs/lib/inference/base.py ~720 cvs/lib/failure_taxonomy.py ~40 +cvs/lib/inference/vllm.py ~60 cvs/lib/registry.py ~30 +cvs/lib/inference/inference_max.py ~60 cvs/lib/adapters/vllm.py ~250 +cvs/lib/inference_lib.py (orphan; broken) ~80 cvs/lib/adapters/inferencemax.py ~200 + cvs/lib/adapters/sglang_disagg.py ~400 + cvs/lib/adapters/pytorch_xdit.py ~250 + cvs/lib/adapters/megatron.py ~350 + cvs/lib/adapters/jax.py ~350 + cvs/lib/config/ (Pydantic schemas, params) ~400 + cvs/lib/handles/secret.py + container.py ~150 + cvs/lib/manifest.py ~300 + cvs/lib/binder.py ~200 +───────────────────────────────────────────────── ───── ────────────────────────────────────────── ───── + ~3841 ~3220 +``` + +`cvs/tests/{training,inference}/` collapses from **16 wrappers** (7 training + 9 inference, ~3800 LOC of which ~1120 is byte-identical fixture boilerplate) into **~6 parametrized test files** (~300 LOC total) plus the per-suite `conftest.py`. + +### Class hierarchy + +```mermaid +flowchart TD + proto["WorkloadAdapter (Protocol)"] + proto --> base["BaseWorkloadAdapter (abc.ABC)
concrete: teardown, await_completion, prepare"] + base --> vllm["VllmAdapter
overrides: launch, parse, progress_predicate"] + base --> imax["InferenceMaxAdapter
overrides: launch, parse, progress_predicate"] + base --> sgl["SglangDisaggAdapter
overrides: prepare, launch, parse, progress_predicate, teardown"] + base --> xdit["PytorchXditAdapter
overrides: launch, parse, progress_predicate"] + base --> meg["MegatronAdapter
overrides: launch, parse, progress_predicate"] + base --> jax["JaxAdapter
overrides: launch, parse, progress_predicate"] +``` + +Most adapters override 3 of 7 methods. `SglangDisaggAdapter` overrides 5 because it manages multi-role orchestration internally. If a future adapter needs to override all 7, the abstraction is failing — refactor to a Composite (deferred to a later release; one multi-role workload doesn't justify it yet). + +### `cvs/tests/` tier tree + +```text +cvs/tests/ +├── conftest.py # session: workload_run fixture, marker registration, collect-skip +├── logistics/ # TIER 1 — every config runs +│ ├── test_prepare.py +│ ├── test_launch.py +│ └── test_teardown.py +├── training/ # TIER 2 — workload_kind=training only +│ ├── conftest.py +│ ├── test_trajectory.py +│ └── test_distributed.py +├── inference/ # TIER 3 — workload_kind=inference only +│ ├── conftest.py +│ ├── test_serving.py +│ ├── test_disagg.py +│ └── test_distributed.py +├── frameworks/ # TIER 4 — one file per framework +│ ├── test_vllm.py +│ ├── test_sglang.py +│ ├── test_inferencemax.py +│ ├── test_pytorch_xdit.py +│ ├── test_megatron.py +│ └── test_jax.py +├── benchmarks/ # TIER 5 — opt-in via config benchmarks: [...] +│ ├── test_throughput.py +│ ├── test_latency.py +│ ├── test_accuracy.py +│ └── test_convergence.py +└── models/ # TIER 6 — model-family edge cases (often empty) + ├── test_gpt_oss.py + └── test_llama.py +``` + +Tier = directory. Claim = test function. One workload run feeds many independent assertions. + +Full prose: [W2](cvs-dtni-v1-spec.md#w2-six-concrete-adapters), [W6](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). + +--- + +## §6. Markers — derivation rules + +| Config field | Marker pattern | Example value → marker | Scope | +|---|---|---|---| +| `framework` | `framework_` | `vllm` → `framework_vllm` | always | +| `model` | `model_` (underscores normalized) | `gpt-oss-120b` → `model_gpt_oss_120b` | always | +| `workload_kind` | `workload_` | `inference` → `workload_inference` | always | +| `topology` (single/distributed/disagg) | `topology_` | `disagg` → `topology_disagg` | always | +| `target_gpu` | `gpu_` | `mi355x` → `gpu_mi355x` | always | +| `knobs.` (scalar) | `knob__` | `attention: aiter` → `knob_attention_aiter` | always | +| `benchmarks: [...]` (list) | `benchmark_` per entry | `[throughput, ttft_p99]` → `benchmark_throughput`, `benchmark_ttft_p99` | opt-in | +| tier (from directory) | `tier_N` | `cvs/tests/benchmarks/` → `tier_5` | always | +| skip reason (from binder) | `skipped_` | `insufficient_nodes` → `skipped_insufficient_nodes` | conditional | + +Registered via `pytest_configure` so `-m` queries don't warn. List-valued config fields fan out into multiple markers. Nested dict knobs flatten with one underscore. + +Full prose: [W6](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). + +--- + +## §7. Cluster + binder walkthrough + +### Cluster file (pool only — no role assignments) + +```yaml +# cluster.json +nodes: + n1: {ip: 10.0.0.11, user: atnair, ssh_key: ~/.ssh/id, gpus: 8, labels: [mi355x]} + n2: {ip: 10.0.0.12, user: atnair, ssh_key: ~/.ssh/id, gpus: 8, labels: [mi355x]} + n3: {ip: 10.0.0.13, user: atnair, ssh_key: ~/.ssh/id, gpus: 8, labels: [mi355x]} + n4: {ip: 10.0.0.14, user: atnair, ssh_key: ~/.ssh/id, gpus: 0, labels: [mi355x, cpu_only]} +``` + +### Config role requirements (in the test config) + +```yaml +topology: + roles: + prefill: {count: 2, gpus_per_node: 8, selector: "mi355x"} + decode: {count: 1, gpus_per_node: 8, selector: "mi355x"} + router: {count: 1, gpus_per_node: 0} +``` + +### Binder behavior — small dev cluster (3 GPU nodes, 1 CPU node) + +```text +$ cvs plan --cluster small_cluster.json --config sglang_2p1d.yaml + +Cell Bindings Status +───────────── ───────────────────────────────────────────── ──────────────────────────────────── +[conc32] prefill=[n1,n2] decode=[n3] router=[n4] OK +[conc64] prefill=[n1,n2] decode=[n3] router=[n4] OK + +Plan summary: 2 cells will execute, 0 will skip +``` + +Now swap in a config that needs 4 GPU nodes: + +```text +$ cvs plan --cluster small_cluster.json --config sglang_2p2d.yaml + +Cell Bindings Status +───────────── ───────────────────────────────────────────── ──────────────────────────────────── +[conc32] — SKIP: insufficient GPU nodes (need 4, have 3) +[conc64] — SKIP: insufficient GPU nodes (need 4, have 3) + +Plan summary: 0 cells will execute, 2 will skip +Hint: cells require a 4-node topology this cluster cannot satisfy. +``` + +Skipped cells still produce manifests (with `status: skipped`, `reason: "insufficient_nodes ..."`), still appear in `pytest --collect-only` with a `skipped_insufficient_nodes` marker, and remain queryable in dashboards. Partial coverage on a small cluster is a feature, not an error. + +The binder is **deterministic** (first-fit by cluster-file declaration order): same cluster + same config → same bindings, always. This is load-bearing for future caching features. + +Full prose: [W5](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). + +--- + +## §8. Manifest + sidecars + cross-run analysis + +### Per-run directory layout + +```text +/vllm_gpt_oss_120b_mi355x_aiter/balanced-conc64/sha7d3a/0193a8e2-71c1.../ +├── manifest.json # 5-50 KB; metadata + verdicts + scalars + sidecar pointers +├── events.jsonl # append-only; closed vocab events +├── samples.parquet # per-request rows (long format) +├── trajectory.parquet # per-step rows (long format) +├── config.resolved.yaml # full resolved config for reproducibility +└── logs/ + ├── stdout.log + ├── stderr.log + ├── dmesg.n2.pre.txt + ├── dmesg.n2.post.txt + ├── gpu_state.n2.pre.json + └── gpu_state.n2.post.json +``` + +### Sample `manifest.json` (abbreviated, real values) + +```json +{ + "schema_version": "1.0", + "manifest_kind": "workload_run", + "run_id": "0193a8e2-71c1-7e0f-9c1a-7d5e8e1f4a02", + "test_id": "vllm_gpt_oss_120b_mi355x_aiter", + "cell_id": "balanced-conc64", + "config_hash": "sha256:91a2...e44b", + "workload_hash": "sha256:7d3a...b21f", + "verification_hash": "sha256:9c1e...8f12", + "experiment_id": "vllm/gpt-oss-120b/fp4+aiter+vllm-native/mi355x", + "cvs_version": "2.4.0", + "cvs_git_sha": "a4f1e2c", + "framework_image_digest": "sha256:7d3a...b21f", + "framework_versions": {"vllm": "0.10.2", "torch": "2.7.1", "rocm": "6.4.0"}, + "timestamp_start": "2026-05-28T20:01:08Z", + "timestamp_end": "2026-05-28T20:14:52Z", + "hosts": [{"hostname": "n1", "ip": "10.0.0.11", "role": "server", "role_index": 0}], + "model_descriptor": {"hf_repo": "openai/gpt-oss-120b", "revision": "main", "precision": "fp4"}, + "phases": { + "prepare": {"duration_s": 4.3, "status": "ok"}, + "launch": {"duration_s": 41.7, "status": "ok"}, + "await": {"duration_s": 720.0, "status": "ok"}, + "parse": {"duration_s": 1.8, "status": "ok"}, + "verify": {"duration_s": 0.1, "status": "failed"}, + "teardown": {"duration_s": 6.9, "status": "ok"} + }, + "status": "failed_verification", + "failure": { + "category": "verification_failure", + "originated_in_phase": "verify", + "message": "P99 TTFT 73.4ms exceeds threshold 50.0ms" + }, + "verdicts": [ + {"kind": "Percentile", "metric": "ttft_ms", "percentile": 99, "op": "<=", + "expected": 50.0, "actual": 73.4, "passed": false, "margin": -23.4}, + {"kind": "Percentile", "metric": "tpot_ms", "percentile": 99, "op": "<=", + "expected": 25.0, "actual": 14.2, "passed": true, "margin": 10.8}, + {"kind": "Rate", "metric": "throughput", "per_unit": "sec", "op": ">=", + "expected": 1200.0, "actual": 1318.0, "passed": true, "margin": 118.0} + ], + "result": { + "scalars": {"ttft_p99_ms": 73.4, "tpot_p99_ms": 14.2, "throughput_tps": 1318.0} + }, + "resource_summary": {"n1": {"gpu_util_mean": 78.1, "hbm_used_max_gb": 142.0, "oom_killed": false}}, + "samples_path": "samples.parquet", + "trajectory_path": "trajectory.parquet", + "events_path": "events.jsonl" +} +``` + +### `samples.parquet` schema (long format) + +| Column | Type | Semantics | Example | +|---|---|---|---| +| `request_id` | string | unique per request | `"req-00042"` | +| `ts` | timestamp | request arrival time | `2026-05-28T20:02:14.110Z` | +| `ttft_ms` | float64 | time to first token | `43.2` | +| `tpot_ms` | float64 | time per output token | `8.1` | +| `itl_ms` | float64 | inter-token latency | `7.9` | +| `e2el_ms` | float64 | end-to-end latency | `1124.0` | +| `output_tokens` | int32 | output token count | `128` | +| `role` | string | role that handled (for composites) | `"decode"` | +| `host` | string | hostname | `"n1"` | + +Long format means new metrics (`memory_pressure`, `kv_cache_util`) become new rows, not new columns — no schema migration. + +### `trajectory.parquet` schema + +| Column | Type | Example | +|---|---|---| +| `step` | int64 | `100` | +| `ts` | timestamp | `2026-05-28T20:05:00Z` | +| `metric` | string | `"loss"`, `"throughput_tps"`, `"grad_norm"`, `"router_queue"` | +| `value` | float64 | `4.21` | +| `role` | string | `"worker"`, `"router"` | +| `host` | string | `"n1"` | + +### `events.jsonl` — closed vocabulary + +| Event | Emitted by | Example payload (besides `ts`) | +|---|---|---| +| `prepare.start` | Job | `{}` | +| `prepare.done` | Job | `{phase_duration_s: 4.3}` | +| `launch.container_up` | Adapter | `{role: "server", host: "n1"}` | +| `launch.role_ready` | Adapter | `{role: "server", host: "n1"}` | +| `step` | Adapter (training) | `{step: 100, loss: 4.21, throughput: 1342}` | +| `request` | Adapter (inference) | `{request_id: "...", ttft_ms: 43.2}` | +| `safety.violated` | Job | `{predicate: "loss_is_finite", detail: "NaN at step 412"}` | +| `pattern.matched` | FailurePatternScanner | `{pattern_id: "oom_kill", source: "dmesg", line: "..."}` | +| `parse.done` | Job | `{samples_rows: 3200, trajectory_rows: 412}` | +| `verify.failed` | Job | `{metric: "ttft_ms", actual: 73.4, expected_max: 50.0}` | +| `teardown.done` | Job | `{}` | + +Adding an event name is a schema change reviewed in PR, not a free-for-all `log.info`. + +### Cross-run regression query (the data-science seam) + +```bash +# Flatten N runs into one fact table +cvs export --artifact-dir ./runs --since 30d -o fact.parquet +``` + +```python +# notebook +import pandas as pd +import matplotlib.pyplot as plt + +fact = pd.read_parquet("fact.parquet") + +# P99 TTFT trend for vLLM gpt-oss-120b with AITER + FP4 on mi355x, by CVS commit +sub = fact.query( + "framework == 'vllm' and model == 'gpt_oss_120b' " + "and knob_attention == 'aiter' and knob_quant == 'fp4' " + "and gpu == 'mi355x' and cell_id == 'balanced-conc64'" +) +sub.groupby("cvs_git_sha")["ttft_p99_ms"].median().plot( + marker="o", title="P99 TTFT regression (gpt-oss-120b)" +) +plt.axhline(50.0, color="red", linestyle="--", label="threshold") +plt.show() +``` + +Three lines of pandas catches a regression. No service, no dashboard infra required. A real dashboard (Grafana / Streamlit) reads the same Parquet. + +Full prose: [W4](cvs-dtni-v1-spec.md#w4-manifest-sidecars-and-cross-run-export), [W8](cvs-dtni-v1-spec.md#w8-tooling-and-documentation). + +--- + +## §9. Sweep semantics — three walkthroughs + +### Cartesian (the common case) + +```yaml +sweep: + concurrency: [16, 32, 64] + isl_osl: + - {isl: 1024, osl: 1024, name: balanced} + - {isl: 4096, osl: 128, name: prefill_heavy} +``` + +→ **6 cells:** `[balanced-conc16]`, `[balanced-conc32]`, `[balanced-conc64]`, `[prefill_heavy-conc16]`, `[prefill_heavy-conc32]`, `[prefill_heavy-conc64]`. + +### Paired with topology change (sglang P/D split) + +```yaml +sweep: + pd_splits: + - name: 2p2d + topology: + roles: + prefill: {count: 2, gpus_per_node: 8, selector: "mi300x"} + decode: {count: 2, gpus_per_node: 8, selector: "mi300x"} + - name: 1p3d + topology: + roles: + prefill: {count: 1, gpus_per_node: 8, selector: "mi300x"} + decode: {count: 3, gpus_per_node: 8, selector: "mi300x"} + concurrency: [32, 64] +``` + +→ **4 cells**, binder re-evaluates per cell (different node assignments for 2p2d vs 1p3d). `cvs plan` shows the bindings before any container starts. + +### Constraint-validated (megatron parallelism) + +```yaml +topology: + roles: + worker: {count: 4, gpus_per_node: 8, selector: "mi300x"} # 32 GPUs total + +sweep: + parallelism_combos: + - {tp: 8, pp: 1, dp: 4, fsdp: 1, name: tp8_dp4} # 8*1*4*1 = 32 ✓ + - {tp: 4, pp: 2, dp: 4, fsdp: 1, name: tp4_pp2_dp4} # 4*2*4*1 = 32 ✓ + - {tp: 1, pp: 1, dp: 1, fsdp: 32, name: fsdp_only} # 1*1*1*32 = 32 ✓ + # {tp: 8, pp: 2, dp: 4, fsdp: 1, name: bad} # 8*2*4*1 = 64 ✗ — Pydantic validator rejects at parse + micro_batch_size: [1, 2] +``` + +→ **6 cells.** The `product(parallelism) == total_gpus` constraint is a Pydantic validator on `MegatronSweepParams`; nonsense combos fail at `model_validate()`, not 20 minutes into the run. + +Full prose: [W3](cvs-dtni-v1-spec.md#w3-typed-config-schema), [W5](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). + +--- + +## §10. Threshold predicates + +Six kinds; explicit `op:` (never inferred from metric name): + +```yaml +# 1. Percentile (over samples) +- {kind: Percentile, metric: ttft_ms, percentile: 99, op: "<=", value: 50} + +# 2. Monotonicity (over trajectory) +- {kind: Monotonicity, metric: loss, window: last_quarter, + direction: non_increasing, tolerance: 0.02} + +# 3. Convergence (over trajectory) +- {kind: Convergence, metric: loss, target: 2.1, epsilon: 0.1, + by_wallclock_sec: 14400} + +# 4. Stability (rolling variance, samples or trajectory) +- {kind: Stability, metric: step_time_ms, window_size: 50, max_variance: 25.0} + +# 5. Rate (derived rate) +- {kind: Rate, metric: throughput, per_unit: sec, op: ">=", min_rate: 1200} + +# 6. Goodput (filtered rate — the MLPerf headline) +- {kind: Goodput, + metric_pair: {ttft: ttft_ms, tpot: tpot_ms}, + ttft_max_ms: 450, tpot_max_ms: 40, + op: ">=", min_qps: 600} +``` + +Each evaluates against the manifest's `samples` or `trajectory` carriers and emits a `Verdict` row with `expected`, `actual`, `passed`, `margin`. The `margin` field powers regression alerts ("p99 TTFT margin shrank from +12 ms to +2 ms over the last 10 runs" is one DuckDB query away). + +Full prose: [W3](cvs-dtni-v1-spec.md#w3-typed-config-schema). + +--- + +## §11. Failure walkthrough — `safety_violation` end-to-end + +A training run hits `NaN` loss at step 412. Here's the trail. + +**1. Adapter's `progress_predicate` returns ProgressStatus(ok=False, predicate_name="loss_is_finite", evidence="NaN at step 412").** + +**2. `Job._await_with_progress` raises `SafetyViolation`. `Job.run`'s `except` clause classifies and records:** + +```python +except SafetyViolation as e: + self.manifest.record_failure("safety_violation", + predicate=e.predicate, + evidence=e.evidence) +``` + +**3. `events.jsonl` gets a row:** + +```json +{"ts": "2026-05-28T20:08:33.012Z", "event": "safety.violated", + "predicate": "loss_is_finite", "detail": "NaN at step 412"} +``` + +**4. `manifest.json` reflects:** + +```json +"status": "failed_safety", +"failure": { + "category": "safety_violation", + "originated_in_phase": "await_completion", + "originated_at_ts": "2026-05-28T20:08:33.012Z", + "message": "loss_is_finite broke: NaN at step 412" +} +``` + +**5. `finally` block runs `adapter.teardown(run)` — containers `docker rm`'d by label, logs + dmesg captured.** + +**6. pytest terminal summary, end of session:** + +```text +FAILED training/test_trajectory.py::test_loss_finite[megatron-llama_70b-tp8_dp4-mbs2] + safety_violation: loss_is_finite broke at step 412 + manifest: runs/megatron_llama_70b/tp8_dp4-mbs2/.../manifest.json +``` + +The pytest-html report carries the same line plus the manifest path; from there, the notebook query in §8 finds whether this is a one-off or a regression. + +Full prose: [W1](cvs-dtni-v1-spec.md#w1-core-lifecycle-and-adapter-framework), [W8](cvs-dtni-v1-spec.md#w8-tooling-and-documentation). + +--- + +## §12. Security and correctness fixes (W7) + +Eight independently-shippable fixes. None depend on the rest of the architecture; can land as a standalone PR series. + +| Defect today | Fix in v1 | +|---|---| +| HF token logged plaintext in `phdl.exec` debug output on every multi-node run | `SecretValue` wrapper; `repr`/`str` redact; `.reveal()` only at env-file write; sentinel-leak CI test | +| `docker run --privileged` + `seccomp=unconfined` default | `ContainerHandle` non-privileged default; readiness probe in `__enter__`; label-scoped cleanup | +| `docker system prune --force` wipes other users' containers on shared hosts | `docker rm` by `run_id` label only | +| `fail_test()` appends to `globals.error_list` and returns; multi-step tests march past first failure | `fail_test()` calls `pytest.fail` | +| Threshold direction inferred from substring `"ms"` in metric name (any `latency_seconds` flips comparison) | explicit `op:` field in every `Threshold` | +| Inferencemax silently passes when threshold is missing from config | `InferenceMaxAdapter.verify` raises on missing threshold | +| `UnboundLocalError` in HF-token fixture on missing token file | clean `pytest.skip(reason="...")` | +| `mi355` vs `mi355x` literal inconsistency across 4 files | single `GpuPlatform.detect()` source of truth | +| `sudo rm -rf $log_dir` unquoted (user-controlled paths) | quoted everywhere; lint check in CI | + +Full prose: [W7](cvs-dtni-v1-spec.md#w7-security-and-correctness-fixes). + +--- + +## §13. Workstream DAG + +```mermaid +flowchart LR + W7["W7 security/correctness
(standalone)"] + W3["W3 typed configs"] + W1["W1 lifecycle + adapters"] + W4["W4 manifest"] + W5["W5 cluster/binder/sweep"] + W2["W2 six adapters"] + W6["W6 pytest layer"] + W8["W8 tooling + docs"] + + W1 --> W2 + W3 --> W2 + W3 --> W6 + W3 --> W5 + W1 --> W6 + W4 --> W6 + W5 --> W6 + W3 --> W8 + W4 --> W8 + W5 --> W8 + W6 --> W8 +``` + +W7 ships first (no upstream). W8 ships last. W3 is the most depended-on. + +--- + +## §14. Adoption + +- **One-shot:** `cvs migrate-config` rewrites every existing JSON under `cvs/input/config_file/{training,inference}/` to v2 YAML. No backwards-compat reader. +- **Wrappers deleted:** all 16 (7 training + 9 inference) replaced by ~6 parametrized test files. +- **CLI unchanged:** `cvs run --cluster_file=... --config_file=...` works identically. Existing automation unaffected. +- **Markers additive:** new `pytest -m "..."` queries become possible; nothing previously possible breaks. + +Full prose: [Migration story in spec](cvs-dtni-v1-spec.md#migration-story). + +--- + +## §15. Decision wanted + +**W7 as a standalone PR series, or merged with the rest?** The eight security/correctness fixes (§12) have zero upstream dependencies in the redesign. Reviewer preference welcome on: + +- **Land first, standalone:** smaller PRs, faster review, fixes ship before the architecture work completes. Recommended. +- **Land bundled with v1:** one cohesive review, but blocks security fixes behind architecture review cycle. + +--- + +## §16. Pointers + +- Full spec (long-form prose): [`docs/prd/cvs-dtni-v1-spec.md`](cvs-dtni-v1-spec.md) +- Original architecture PRD (superseded): preserved at commit [`0d0dd1a`](https://github.com/ROCm/cvs/blob/0d0dd1a/docs/prd/cvs-dtni-suite-expansion-prd.md) on this branch +- Per-workstream history: see commits `625c8f2`, `f6c84ba`, `0d0dd1a`, and HEAD on `atnair/prd-dtni-refactor` From 7392e67bc25ecba32219c1c1abff994266640a77 Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Fri, 29 May 2026 02:18:01 -0400 Subject: [PATCH 6/7] Restructure pr-body.md around design philosophy and capabilities Replaces the prior "sledgehammer before/after" framing with a forward- looking overview of what the redesign enables, organized into seven sections per reviewer guidance: 1. Code structure and design philosophy (uniform lifecycle + factory/ registry + state-on-disk; flow diagram; Protocol + class hierarchy; "framework emits, CVS retains" + "one workload run, many sliceable claims" principles) 2. Tiered tests (the six tiers with examples; why tier structure makes the matrix queryable) 3. Pytest invocation and lifecycle (the matrix story end-to-end with the lifecycle mermaid, marker derivation table, sample test ID showing every axis, three CLI slicing queries, workload_run fixture sketch) 4. Config files - categories of dials (annotated YAML with sections; one-paragraph description of each dial category: identity, workload, model, knobs, params, sweep, benchmarks, thresholds, topology, secrets) 5. Sweeps (three semantics with examples; how cells propagate into pytest parametrize IDs; cvs plan dry-run) 6. Metrics and benchmarks supported per framework (two matrix tables: captured metrics per framework, benchmarks per framework; six Threshold predicates as YAML; how new metrics/benchmarks/predicates are added cheaply) 7. Manifest and sidecars (directory layout; sample manifest.json; samples/trajectory schemas; events vocabulary; cross-run pandas regression query; why the design choices enable cheap re-verify, re-parse, and dashboard consumption) Dropped: sglang before/after hook, repo file-tree before/after, dedicated cluster/binder section, standalone failure walkthrough, workstream DAG. W7 security fixes moved to appendix. Adds prose throughout to frame the snippets and diagrams; the goal of the overview is to show philosophy and capabilities, not enumerate line counts. --- docs/prd/pr-body.md | 995 ++++++++++++++++++++------------------------ 1 file changed, 457 insertions(+), 538 deletions(-) diff --git a/docs/prd/pr-body.md b/docs/prd/pr-body.md index 0438da5b..11412f96 100644 --- a/docs/prd/pr-body.md +++ b/docs/prd/pr-body.md @@ -1,235 +1,193 @@ -# CVS DTNI v1 — implementation spec (digestible) +# CVS DTNI v1 — refactor overview **Status:** Draft for team review. Full prose lives in [`cvs-dtni-v1-spec.md`](cvs-dtni-v1-spec.md); this file is the PR body and the entry point. ---- - -## TLDR +This PR is a refactor of the DTNI (data-center training and inference) suite — turning a fork-per-workload structure (one Python wrapper per `(framework, model, single/distributed)` tuple, one monolithic library per framework) into a **uniform lifecycle-driven workload runner**. The goal of this overview is to show what the redesign *enables*, not to enumerate every line that moves. Four capabilities are the headline: -Two of CVS's real sglang test wrappers are **429 lines each**. They differ by **one line** — `'llama-70b'` vs `'deepseek-r1'` on line 297. v1 collapses both into one parametrized test consuming two YAMLs, deletes `cvs/lib/sglang_disagg_lib.py` (1261 LOC), and stops the HF token from being logged in `phdl.exec` debug output on every multi-node run. Net change: ~**−5000 LOC** across `cvs/lib/` and `cvs/tests/`, **+1** lifecycle driver, **+6** adapters, **+6** typed threshold predicates, **+1** content-addressable manifest tree that pandas / DuckDB read directly. +- **Typed declarative configs** — one YAML per `(framework, model)`, Pydantic-validated, fail-fast on typos. +- **Pytest-as-first-class matrix slicing** — every axis a benchmark engineer cares about (framework, model, dataset, benchmark, metric, hyperparameter, backend knob like Mooncake or AITER) is a sliceable pytest marker on day one. +- **Durable on-disk artifacts** — per-cell content-addressable directory with a manifest + Parquet sidecars + raw logs, queryable directly from pandas / DuckDB. No regression-analysis service required. +- **Clean separation** between framework-agnostic orchestration (one Job driver, one lifecycle) and per-framework specifics (one adapter per framework, registered via a factory). -The matrix is the artifact: pick any axis a benchmark engineer cares about — framework, model, knob (`aiter`, `fa`, `fp4`, `mooncake`), threshold, git SHA, cluster — and `pytest -m "..."` slices it on day one. +**Reading guide** — seven sections, ~15 min end-to-end: -**Reading order if you only have N minutes:** +1. Code structure and design philosophy +2. Tiered tests — what they are and why +3. Pytest invocation and lifecycle — the matrix story +4. Config files — categories of dials +5. Sweeps — how the matrix expands +6. Metrics and benchmarks supported per framework +7. Manifest and sidecars — durable runs and regression analysis -- **5 min:** §1 data flow → §2 sglang before/after → §13 W7 security fixes (independently shippable) -- **15 min:** add §3 one config → N pytest IDs → §8 manifest + cross-run query → §11 failure walkthrough -- **Full:** §1–§16 below, with anchors into [`cvs-dtni-v1-spec.md`](cvs-dtni-v1-spec.md) for the long form +An appendix covers adoption, security/correctness fixes (W7), and the open reviewer decision. --- -## §1. End-to-end data flow +## §1. Code structure and design philosophy -One picture covers the whole thing. +The redesign rests on three primitives. -```mermaid -flowchart LR - cfg["config.yaml
cluster.json"] --> plan["cvs plan
(sweep expansion + binder)"] - plan -->|"N cells"| job["Job.run per cell
prepare → launch → await → parse → verify → teardown"] - job --> art["per-run dir:
manifest.json
samples.parquet
trajectory.parquet
events.jsonl
config.resolved.yaml
logs/"] - art --> pytest["pytest reads manifest
N test IDs per cell"] - art --> export["cvs export"] - export --> fact["fact.parquet
(M runs × all metrics)"] - fact --> nb["notebook /
dashboard /
regression alert"] -``` +### Three primitives -The persistent artifacts are everything under the per-run dir. `cvs plan` and pytest invocation are ephemeral. `cvs export` is the data-science seam — flatten N runs into one Parquet fact table; pandas/DuckDB consume it without a service. +1. **A uniform lifecycle.** Every workload — training or inference, single-role or multi-role — executes the same six phases: `prepare → launch → await_completion → parse → verify → teardown`. The `Job` driver runs this lifecycle the same way for every workload; per-framework specialization lives entirely inside the adapter that implements those six methods. -Full prose: [W4 in spec](cvs-dtni-v1-spec.md#w4-manifest-sidecars-and-cross-run-export), [W5 in spec](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). +2. **A factory + registry.** A typed config selects which adapter handles it. `INFERENCE_REGISTRY` and `TRAINING_REGISTRY` map `framework: vllm` (or `megatron`, `sglang_disagg`, etc.) to the concrete adapter class. The `Job` driver never branches on framework name; adding a new framework is one new adapter + one registry line. ---- +3. **State on disk, not in memory.** Every cell of a run produces a content-addressable directory containing `manifest.json` + Parquet sidecars + raw logs. Tests, dashboards, and CI consumers all read from that directory. Nothing depends on module-level Python state surviving a process exit. -## §2. The sglang before/after +### The flow -The single most persuasive change in the PR. +```mermaid +flowchart LR + cfg["Typed config
(framework: vllm)"] --> reg["Registry
(INFERENCE / TRAINING)"] + reg --> adapter["Concrete adapter
(VllmAdapter, MegatronAdapter, ...)"] + adapter --> job["Job driver
(6-step lifecycle)"] + job --> dir["Per-run dir on disk
(manifest + parquet + logs)"] + dir --> tests["pytest functions
(read manifest, fire assertions)"] + dir --> exp["cvs export
(flatten to fact.parquet)"] + exp --> nb["pandas / DuckDB / dashboard"] +``` -**Before** (`cvs/tests/inference/sglang/`): +The dashed arrows from `dir` indicate that the manifest tree is the single source of truth: pytest functions, cross-run exports, and any future dashboard all read the same artifacts. -```text -sglang_llama_70b_distributed.py 429 LOC -sglang_deepseek_r1_671b_distributed.py 429 LOC - ───────── - 858 LOC -``` +### The adapter contract -These two files are **byte-identical except for line 297**: +The factory hands a registered class to the Job driver; the Job driver calls these seven methods in order: ```python -# sglang_llama_70b_distributed.py:297 -bp_dict = benchmark_params_dict['llama-70b'] - -# sglang_deepseek_r1_671b_distributed.py:297 -bp_dict = benchmark_params_dict['deepseek-r1'] +# cvs/lib/adapter_protocol.py +class WorkloadAdapter(Protocol): + def prepare(self, ctx: Context) -> None: ... + def launch(self, ctx: Context) -> AdapterRun: ... + def await_completion(self, run: AdapterRun) -> None: ... + def progress_predicate(self, run: AdapterRun) -> ProgressStatus: ... + def parse(self, run: AdapterRun, manifest: Manifest) -> WorkloadResult: ... + def verify(self, result: WorkloadResult, thresholds: list[Threshold]) -> list[Verdict]: ... + def teardown(self, run: AdapterRun) -> None: ... ``` -Plus the library they both consume: `cvs/lib/sglang_disagg_lib.py` at **1261 LOC** — a single monolithic class that cannot be partially reused. +`BaseWorkloadAdapter` provides concrete defaults for `teardown` (always capture logs + dmesg + GPU state, then `docker rm` by label), `await_completion` (poll the predicate with timeout), and `prepare` (no-op). Most adapters override 3 of the 7 methods. -**After:** - -```text -cvs/tests/inference/test_inference.py (one parametrized test) -cvs/input/config_file/inference/sglang/llama_70b_disagg.yaml (~80 LOC YAML) -cvs/input/config_file/inference/sglang/deepseek_r1_disagg.yaml (~80 LOC YAML) -cvs/lib/adapters/sglang_disagg.py (~400 LOC adapter) +```mermaid +flowchart TD + proto["WorkloadAdapter (Protocol)"] + proto --> base["BaseWorkloadAdapter
(concrete defaults: teardown, await_completion, prepare)"] + base --> vllm["VllmAdapter"] + base --> imax["InferenceMaxAdapter"] + base --> sgl["SglangDisaggAdapter"] + base --> xdit["PytorchXditAdapter"] + base --> meg["MegatronAdapter"] + base --> jax["JaxAdapter"] ``` -**Net delta:** −2119 LOC of duplicated Python and monolithic library; +1 reusable adapter; +N typed configs (one per model). The same shape applies to the other 14 wrappers (see §5). +### Design philosophy -Full prose: [W2 in spec](cvs-dtni-v1-spec.md#w2-six-concrete-adapters). +Two principles drove the choices above. ---- +**Framework emits → CVS retains.** Today the frameworks already emit rich data — per-request JSONL (vLLM), per-step trajectories (Megatron, JAX), Prometheus metrics (vLLM, sglang), per-batch JSON (InferenceMAX), per-step latency tracelogs (xDiT). Today CVS tail-greps the console and discards everything else. The redesign routes the framework's native emission directly into Parquet sidecars; nothing is invented and nothing is lost. Adding a metric is a `parse()` change, not a new pipeline. -## §3. One config → N pytest IDs +**One workload run → many sliceable claims.** A single cell of a sweep produces one manifest. Many pytest test functions (logistics, framework-specific, benchmark-specific, model-specific) read that one manifest and assert independent properties. Failure of one assertion doesn't invalidate the run; the manifest is durable; rerunning a subset of claims against an existing manifest becomes one CLI flag (see §7). -The matrix story, end-to-end. +The abstraction is intentionally shallow. If a hypothetical future workload needs to override all seven adapter methods, the abstraction has failed for that workload and the right move is to refactor at that point — not build for it speculatively now. -### 3a. The config +Full prose: [W1](cvs-dtni-v1-spec.md#w1-core-lifecycle-and-adapter-framework), [W2](cvs-dtni-v1-spec.md#w2-six-concrete-adapters). -```yaml -# cvs/input/config_file/inference/vllm/gpt-oss-120b_mi355x_aiter.yaml -schema_version: "2" -test_id: vllm_gpt_oss_120b_mi355x_aiter -target_gpu: mi355x - -framework: vllm -workload_kind: inference -topology: - roles: - server: {count: 1, gpus_per_node: 8, selector: "mi355x"} - -model: gpt-oss-120b -knobs: - attention: aiter - quant: fp4 - backend: vllm-native - -params: - tensor_parallelism: 1 - max_model_length: 9216 - num_prompts: 3200 - -sweep: - concurrency: [16, 32, 64] - isl_osl: - - {isl: 1024, osl: 1024, name: balanced} - - {isl: 4096, osl: 128, name: prefill_heavy} +--- -benchmarks: [throughput, ttft_p99, tpot_p99] +## §2. Tiered tests — what they are and why -thresholds: - - {kind: Rate, metric: throughput, per_unit: sec, op: ">=", min_rate: 1200} - - {kind: Percentile, metric: ttft_ms, percentile: 99, op: "<=", value: 50} - - {kind: Percentile, metric: tpot_ms, percentile: 99, op: "<=", value: 25} -``` +A "test" in v1 isn't "did the workload run end-to-end and produce one pass/fail line." It's a layered stack of independent claims about the same workload run. Tier = directory under `cvs/tests/` = abstraction layer of the claim. -### 3b. `cvs plan` output +| Tier | What it claims | Applies to | Example test functions | +|---|---|---|---| +| 1. Logistics | The workload started, ran, and cleaned up | Every config | `test_image_pullable`, `test_container_up`, `test_role_ready`, `test_no_orphans`, `test_dmesg_clean` | +| 2. Workload-kind | Training/inference invariants | training or inference configs respectively | `test_loss_finite`, `test_request_success_rate`, `test_no_5xx_burst` | +| 3. Topology | Distribution/disagg invariants | only distributed or disagg configs | `test_per_rank_step_sync`, `test_no_straggler`, `test_router_balance` | +| 4. Framework | Framework knobs were applied correctly | only the matching framework | `test_aiter_flags_active`, `test_xla_flags_applied`, `test_attention_backend_matches` | +| 5. Benchmark | The perf claims declared in `benchmarks:` | opt-in per config | `test_throughput_min`, `test_ttft_p99`, `test_convergence`, `test_goodput` | +| 6. Model | Known model-family edge cases | only the matching model | `test_quant_conversion_consistent` (gpt-oss-120b) | ```text -$ cvs plan --cluster cluster.json --config gpt-oss-120b_mi355x_aiter.yaml - -6 cells (concurrency × isl_osl, cartesian): - -Cell Bindings Tests Markers -────────────────────────────────────── ─────────────── ────── ───────────────────────────────────────── -[balanced-conc16] server=[n1] 13 framework_vllm, model_gpt_oss_120b, -[balanced-conc32] server=[n1] 13 knob_attention_aiter, knob_quant_fp4, -[balanced-conc64] server=[n1] 13 topology_single, workload_inference, -[prefill_heavy-conc16] server=[n1] 13 gpu_mi355x, benchmark_throughput, -[prefill_heavy-conc32] server=[n1] 13 benchmark_ttft_p99, benchmark_tpot_p99, -[prefill_heavy-conc64] server=[n1] 13 tier_1, tier_3, tier_4, tier_5 - -Tests collected per cell (after collect-skip): - logistics/test_prepare.py::test_image_pullable - logistics/test_launch.py::test_container_up - logistics/test_launch.py::test_role_ready - logistics/test_teardown.py::test_no_orphans - logistics/test_teardown.py::test_dmesg_clean - inference/test_serving.py::test_server_health - inference/test_serving.py::test_request_success_rate - frameworks/test_vllm.py::test_aiter_flags_active - frameworks/test_vllm.py::test_attention_backend_matches[aiter] - benchmarks/test_throughput.py::test_throughput_min - benchmarks/test_latency.py::test_ttft_p99 - benchmarks/test_latency.py::test_tpot_p99 - models/test_gpt_oss.py::test_quant_conversion_consistent - -Total: 78 pytest IDs (6 cells × 13 tests). -Estimated wall time: ~24 min (based on 3 prior matching runs). +cvs/tests/ +├── conftest.py +├── logistics/ # tier 1 +├── training/ # tier 2 + 3 (distributed subset) +├── inference/ # tier 2 + 3 (disagg / distributed subsets) +├── frameworks/ # tier 4 — one file per framework +├── benchmarks/ # tier 5 — opt-in via config benchmarks: [...] +└── models/ # tier 6 — rarely used; documents quirks ``` -### 3c. CLI slicing — three real queries +### How tiers are collected -```bash -# All vLLM + AITER cells across every model and concurrency -cvs run -m "framework_vllm and knob_attention_aiter" +The framework collects tier 1 for every config. Tiers 2–4 are gated by a `collect-skip` hook in `pytest_collection_modifyitems` that compares each test's tier predicate against the cell's config (`workload_kind`, `topology`, `framework`) and **deselects** mismatched items (not skipped — deselected, so the report stays clean). Tier 5 is opt-in via the config's `benchmarks: [...]` list and a `@requires_benchmark("name")` decorator. Tier 6 is rare and routes by `model:`. -# Just the P99 TTFT claim across the whole nightly sweep -cvs run -m "tier_5 and benchmark_ttft_p99" +A typical inference cell collects ~13 test IDs (5 logistics + 2 inference-kind + 2 framework + 3 benchmark + 1 model). A typical training cell collects ~10. -# Distributed training cells only, just convergence claims, FP8 quant only -cvs run -m "workload_training and topology_distributed and knob_quant_fp8 and benchmark_convergence" -``` +### Why tier structure matters -Each query lowers to standard `pytest -m` (markers auto-applied at collection from config fields — see §6 derivation table). Slicing along framework, model, knob, topology, benchmark, GPU, or tier all work on day one. +Slicing. A reviewer who wants only "did training loss diverge anywhere last night?" runs `pytest -m "tier_2 and benchmark_loss_finite"` and gets exactly those claims, without needing to know which configs declared `loss_finite` as a check. A user investigating an HF token leak runs `pytest -m "tier_1 and not skipped_insufficient_nodes"` to see only logistics across the whole nightly sweep. Tiers are the structure that makes the test matrix queryable rather than monolithic. -Full prose: [W6 in spec](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). +Full prose: [W6](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). --- -## §4. The lifecycle and its driver +## §3. Pytest invocation and lifecycle — the matrix story -```mermaid -flowchart TD - prepare["prepare()"] --> launch["launch()"] - launch --> awaitc["await_completion()
(polls progress_predicate)"] - awaitc --> parse["parse()"] - parse --> verify["verify()
(evaluates thresholds)"] - verify --> teardown["teardown()
(RAII; always runs)"] - - prepare -.->|"raises"| setup["setup_failure"] - launch -.->|"raises"| setup - awaitc -.->|"predicate broke"| safety["safety_violation"] - awaitc -.->|"timeout"| liveness["liveness_failure"] - parse -.->|"pattern hit"| pattern["failure_pattern_matched"] - verify -.->|"threshold False"| verif["verification_failure"] - - setup --> teardown - safety --> teardown - liveness --> teardown - pattern --> teardown - verif --> teardown +### What happens when a user runs `cvs run` + +The user types: + +```bash +cvs run --cluster cluster.json --config configs/inference/vllm/gpt-oss-120b.yaml ``` -The `Job` driver runs the same six-step body for every workload — no `if mode == "training"` branching. The five failure categories are classified at the boundary where they originate, not by post-hoc stack-trace inspection. +Internally: -### The Protocol +1. The config is parsed and Pydantic-validated (`extra = "forbid"`, so typos fail here). +2. The sweep block expands into N cells (§5). +3. The binder assigns physical nodes from the cluster pool to each cell's role requirements (per-cell, deterministic; §4 covers the topology dial). +4. Pytest collects test functions for each cell, applies markers derived from config fields (§2's tier predicate determines which functions actually run for that cell), runs the lifecycle once per cell via the `workload_run` session-scoped fixture, and fires the collected test functions against the resulting manifest. +5. A `pytest_terminal_summary` hook aggregates verdicts across all cells. -```python -# cvs/lib/adapter_protocol.py -class WorkloadAdapter(Protocol): - def prepare(self, ctx: Context) -> None: ... - def launch(self, ctx: Context) -> AdapterRun: ... - def await_completion(self, run: AdapterRun) -> None: ... - def progress_predicate(self, run: AdapterRun) -> ProgressStatus: ... - def parse(self, run: AdapterRun, manifest: Manifest) -> WorkloadResult: ... - def verify(self, result: WorkloadResult, thresholds: list[Threshold]) -> list[Verdict]: ... - def teardown(self, run: AdapterRun) -> None: ... +### The lifecycle + +```mermaid +flowchart TD + prep["prepare()"] --> lau["launch()"] + lau --> aw["await_completion()
(polls progress_predicate)"] + aw --> par["parse()"] + par --> ver["verify()
(evaluates thresholds)"] + ver --> td["teardown()
(RAII; always runs)"] + + prep -.->|"raises"| setup["setup_failure"] + lau -.->|"raises"| setup + aw -.->|"predicate broke"| safety["safety_violation"] + aw -.->|"timeout"| liveness["liveness_failure"] + par -.->|"pattern hit"| pattern["failure_pattern_matched"] + ver -.->|"threshold False"| verif["verification_failure"] + + setup --> td + safety --> td + liveness --> td + pattern --> td + verif --> td ``` -### The driver +The 6-step body is identical for every workload — no `if mode == "training"` branching in the driver. Failures are classified at the boundary where they originate; `teardown` always runs in `finally`. The five failure categories map to actionable next steps: `setup_failure` means your config or environment is wrong; `safety_violation` means the workload broke its own invariants mid-run (NaN loss, server health probe failing, etc.); `verification_failure` means it ran cleanly but missed a threshold. ```python -# cvs/lib/job.py +# cvs/lib/job.py (skeleton) class Job: def run(self) -> Manifest: run = None try: - self.adapter.prepare(self.ctx) # raises -> setup_failure - run = self.adapter.launch(self.ctx) # raises -> setup_failure - self._await_with_progress(run) # raises -> safety / liveness + self.adapter.prepare(self.ctx) + run = self.adapter.launch(self.ctx) # raises -> setup_failure + self._await_with_progress(run) # raises -> safety / liveness result = self.adapter.parse(run, self.manifest) verdicts = self.adapter.verify(result, self.ctx.cfg.thresholds) - self.manifest.record_verdicts(verdicts) # status -> pass | verification_failure + self.manifest.record_verdicts(verdicts) # status: pass | verification_failure except SetupFailure as e: self.manifest.record_failure("setup_failure", e.evidence) except SafetyViolation as e: @@ -240,332 +198,165 @@ class Job: self.manifest.record_failure("failure_pattern_matched", e.pattern_id, e.line) finally: if run is not None: - self.adapter.teardown(run) # RAII: always runs + self.adapter.teardown(run) # always runs self.manifest.flush() return self.manifest ``` -Full prose: [W1 in spec](cvs-dtni-v1-spec.md#w1-core-lifecycle-and-adapter-framework). +### Markers — the matrix surface ---- +Pytest markers are auto-derived from config fields at collection time. This is the surface a user actually queries: -## §5. What changes in the repo +| Config field | Marker pattern | Example value → marker | +|---|---|---| +| `framework` | `framework_` | `vllm` → `framework_vllm` | +| `model` | `model_` (underscores normalized) | `gpt-oss-120b` → `model_gpt_oss_120b` | +| `workload_kind` | `workload_` | `inference` → `workload_inference` | +| `topology` | `topology_` | `disagg` → `topology_disagg` | +| `target_gpu` | `gpu_` | `mi355x` → `gpu_mi355x` | +| `knobs.` (scalar) | `knob__` | `attention: aiter` → `knob_attention_aiter` | +| `benchmarks: [...]` (list) | `benchmark_` per entry | `[throughput, ttft_p99]` → `benchmark_throughput`, `benchmark_ttft_p99` | +| tier (from directory) | `tier_N` | `cvs/tests/benchmarks/` → `tier_5` | +| skip reason (binder) | `skipped_` | `insufficient_nodes` → `skipped_insufficient_nodes` | + +Registered via `pytest_configure` so `-m` queries don't emit unknown-marker warnings. List-valued config fields fan out into multiple markers. -### `cvs/lib/` before vs after +### What a test ID looks like ```text -BEFORE LOC AFTER LOC -───────────────────────────────────────────────── ───── ────────────────────────────────────────── ───── -cvs/lib/sglang_disagg_lib.py 1261 cvs/lib/adapter_protocol.py ~30 -cvs/lib/megatron_training_lib.py ~830 cvs/lib/base_adapter.py ~150 -cvs/lib/jax_training_lib.py ~830 cvs/lib/job.py ~120 -cvs/lib/inference/base.py ~720 cvs/lib/failure_taxonomy.py ~40 -cvs/lib/inference/vllm.py ~60 cvs/lib/registry.py ~30 -cvs/lib/inference/inference_max.py ~60 cvs/lib/adapters/vllm.py ~250 -cvs/lib/inference_lib.py (orphan; broken) ~80 cvs/lib/adapters/inferencemax.py ~200 - cvs/lib/adapters/sglang_disagg.py ~400 - cvs/lib/adapters/pytorch_xdit.py ~250 - cvs/lib/adapters/megatron.py ~350 - cvs/lib/adapters/jax.py ~350 - cvs/lib/config/ (Pydantic schemas, params) ~400 - cvs/lib/handles/secret.py + container.py ~150 - cvs/lib/manifest.py ~300 - cvs/lib/binder.py ~200 -───────────────────────────────────────────────── ───── ────────────────────────────────────────── ───── - ~3841 ~3220 +benchmarks/test_latency.py::test_ttft_p99[vllm-gpt_oss_120b-mi355x-aiter-fp4-balanced-conc64] ``` -`cvs/tests/{training,inference}/` collapses from **16 wrappers** (7 training + 9 inference, ~3800 LOC of which ~1120 is byte-identical fixture boilerplate) into **~6 parametrized test files** (~300 LOC total) plus the per-suite `conftest.py`. +Every axis a benchmark engineer might want to slice on — framework, model, GPU, attention knob, quant knob, sweep cell (`balanced-conc64`) — appears in the parametrize bracket. The test function name (`test_ttft_p99`) names the claim. The marker set on this item includes `framework_vllm`, `model_gpt_oss_120b`, `gpu_mi355x`, `knob_attention_aiter`, `knob_quant_fp4`, `workload_inference`, `topology_single`, `benchmark_ttft_p99`, `tier_5`. -### Class hierarchy +### Three real CLI queries -```mermaid -flowchart TD - proto["WorkloadAdapter (Protocol)"] - proto --> base["BaseWorkloadAdapter (abc.ABC)
concrete: teardown, await_completion, prepare"] - base --> vllm["VllmAdapter
overrides: launch, parse, progress_predicate"] - base --> imax["InferenceMaxAdapter
overrides: launch, parse, progress_predicate"] - base --> sgl["SglangDisaggAdapter
overrides: prepare, launch, parse, progress_predicate, teardown"] - base --> xdit["PytorchXditAdapter
overrides: launch, parse, progress_predicate"] - base --> meg["MegatronAdapter
overrides: launch, parse, progress_predicate"] - base --> jax["JaxAdapter
overrides: launch, parse, progress_predicate"] -``` - -Most adapters override 3 of 7 methods. `SglangDisaggAdapter` overrides 5 because it manages multi-role orchestration internally. If a future adapter needs to override all 7, the abstraction is failing — refactor to a Composite (deferred to a later release; one multi-role workload doesn't justify it yet). +```bash +# All vLLM + AITER cells across every model and concurrency +cvs run -m "framework_vllm and knob_attention_aiter" -### `cvs/tests/` tier tree +# Just the P99 TTFT claim across the whole nightly sweep, all frameworks +cvs run -m "benchmark_ttft_p99" -```text -cvs/tests/ -├── conftest.py # session: workload_run fixture, marker registration, collect-skip -├── logistics/ # TIER 1 — every config runs -│ ├── test_prepare.py -│ ├── test_launch.py -│ └── test_teardown.py -├── training/ # TIER 2 — workload_kind=training only -│ ├── conftest.py -│ ├── test_trajectory.py -│ └── test_distributed.py -├── inference/ # TIER 3 — workload_kind=inference only -│ ├── conftest.py -│ ├── test_serving.py -│ ├── test_disagg.py -│ └── test_distributed.py -├── frameworks/ # TIER 4 — one file per framework -│ ├── test_vllm.py -│ ├── test_sglang.py -│ ├── test_inferencemax.py -│ ├── test_pytorch_xdit.py -│ ├── test_megatron.py -│ └── test_jax.py -├── benchmarks/ # TIER 5 — opt-in via config benchmarks: [...] -│ ├── test_throughput.py -│ ├── test_latency.py -│ ├── test_accuracy.py -│ └── test_convergence.py -└── models/ # TIER 6 — model-family edge cases (often empty) - ├── test_gpt_oss.py - └── test_llama.py +# Distributed training cells only, FP8 quant, just convergence claims +cvs run -m "workload_training and topology_distributed and knob_quant_fp8 and benchmark_convergence" ``` -Tier = directory. Claim = test function. One workload run feeds many independent assertions. +### One workload run, many independent claims -Full prose: [W2](cvs-dtni-v1-spec.md#w2-six-concrete-adapters), [W6](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). - ---- +The `workload_run` fixture in `cvs/tests/conftest.py` is session-scoped. For each cell, it instantiates the adapter (via the registry), runs `Job.run()` once, and yields the resulting manifest. Every test function for that cell then consumes the same manifest object: -## §6. Markers — derivation rules +```python +# cvs/tests/conftest.py (sketch) +@pytest.fixture(scope="session") +def workload_run(config_cell, cluster): + adapter_cls = REGISTRY[config_cell.framework] + adapter = adapter_cls(config_cell, cluster, gpu, secrets) + manifest = Job(adapter, config_cell, cluster, gpu, secrets).run() + return manifest +``` -| Config field | Marker pattern | Example value → marker | Scope | -|---|---|---|---| -| `framework` | `framework_` | `vllm` → `framework_vllm` | always | -| `model` | `model_` (underscores normalized) | `gpt-oss-120b` → `model_gpt_oss_120b` | always | -| `workload_kind` | `workload_` | `inference` → `workload_inference` | always | -| `topology` (single/distributed/disagg) | `topology_` | `disagg` → `topology_disagg` | always | -| `target_gpu` | `gpu_` | `mi355x` → `gpu_mi355x` | always | -| `knobs.` (scalar) | `knob__` | `attention: aiter` → `knob_attention_aiter` | always | -| `benchmarks: [...]` (list) | `benchmark_` per entry | `[throughput, ttft_p99]` → `benchmark_throughput`, `benchmark_ttft_p99` | opt-in | -| tier (from directory) | `tier_N` | `cvs/tests/benchmarks/` → `tier_5` | always | -| skip reason (from binder) | `skipped_` | `insufficient_nodes` → `skipped_insufficient_nodes` | conditional | - -Registered via `pytest_configure` so `-m` queries don't warn. List-valued config fields fan out into multiple markers. Nested dict knobs flatten with one underscore. +Thirteen test IDs per cell does not mean thirteen workload launches — it means one launch and thirteen independent verdicts. Most tests are pure manifest-reads (`assert manifest.scalars["ttft_p99_ms"] <= threshold.value`), so post-launch verification adds milliseconds per assertion. -Full prose: [W6](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). +Full prose: [W1](cvs-dtni-v1-spec.md#w1-core-lifecycle-and-adapter-framework), [W6](cvs-dtni-v1-spec.md#w6-pytest-layer-and-test-taxonomy). --- -## §7. Cluster + binder walkthrough +## §4. Config files — categories of dials + +Today's config story is fragmented: cluster JSON declares hostnames, the test wrapper hard-codes role lists, the per-framework library applies defaults via `dict.setdefault`, threshold values live in a stringified-key dict (`"ISL=1024,OSL=1024,TP=8,CONC=64"`), and typos silently fall through to defaults. v1 collapses all of this into **one Pydantic-validated YAML per `(framework, model)`**. The schema is `extra = "forbid"`, so any unknown field is a `model_validate()` error at parse time. -### Cluster file (pool only — no role assignments) +### Anatomy of a config ```yaml -# cluster.json -nodes: - n1: {ip: 10.0.0.11, user: atnair, ssh_key: ~/.ssh/id, gpus: 8, labels: [mi355x]} - n2: {ip: 10.0.0.12, user: atnair, ssh_key: ~/.ssh/id, gpus: 8, labels: [mi355x]} - n3: {ip: 10.0.0.13, user: atnair, ssh_key: ~/.ssh/id, gpus: 8, labels: [mi355x]} - n4: {ip: 10.0.0.14, user: atnair, ssh_key: ~/.ssh/id, gpus: 0, labels: [mi355x, cpu_only]} -``` +# cvs/input/config_file/inference/vllm/gpt-oss-120b_mi355x_aiter.yaml -### Config role requirements (in the test config) +# ---- Identity ---- +schema_version: "2" +test_id: vllm_gpt_oss_120b_mi355x_aiter +target_gpu: mi355x -```yaml +# ---- Workload kind & framework ---- +framework: vllm +workload_kind: inference topology: roles: - prefill: {count: 2, gpus_per_node: 8, selector: "mi355x"} - decode: {count: 1, gpus_per_node: 8, selector: "mi355x"} - router: {count: 1, gpus_per_node: 0} -``` - -### Binder behavior — small dev cluster (3 GPU nodes, 1 CPU node) + server: {count: 1, gpus_per_node: 8, selector: "mi355x"} -```text -$ cvs plan --cluster small_cluster.json --config sglang_2p1d.yaml +# ---- Model ---- +model: gpt-oss-120b -Cell Bindings Status -───────────── ───────────────────────────────────────────── ──────────────────────────────────── -[conc32] prefill=[n1,n2] decode=[n3] router=[n4] OK -[conc64] prefill=[n1,n2] decode=[n3] router=[n4] OK +# ---- Knobs (first-class; become pytest markers) ---- +knobs: + attention: aiter + quant: fp4 + backend: vllm-native + fused_moe: aiter_a16w4 -Plan summary: 2 cells will execute, 0 will skip -``` +# ---- Framework-specific params ---- +params: + tensor_parallelism: 1 + max_model_length: 9216 + num_prompts: 3200 -Now swap in a config that needs 4 GPU nodes: +# ---- Sweep (see §5) ---- +sweep: + concurrency: [16, 32, 64] + isl_osl: + - {isl: 1024, osl: 1024, name: balanced} + - {isl: 4096, osl: 128, name: prefill_heavy} -```text -$ cvs plan --cluster small_cluster.json --config sglang_2p2d.yaml +# ---- Tier-5 benchmark opt-in ---- +benchmarks: [throughput, ttft_p99, tpot_p99] -Cell Bindings Status -───────────── ───────────────────────────────────────────── ──────────────────────────────────── -[conc32] — SKIP: insufficient GPU nodes (need 4, have 3) -[conc64] — SKIP: insufficient GPU nodes (need 4, have 3) +# ---- Typed thresholds ---- +thresholds: + - {kind: Rate, metric: throughput, per_unit: sec, op: ">=", min_rate: 1200} + - {kind: Percentile, metric: ttft_ms, percentile: 99, op: "<=", value: 50} + - {kind: Percentile, metric: tpot_ms, percentile: 99, op: "<=", value: 25} -Plan summary: 0 cells will execute, 2 will skip -Hint: cells require a 4-node topology this cluster cannot satisfy. +# ---- Secrets (type-redacted) ---- +secrets: + hf_token: {kind: SecretValue, source: env, env_var: HF_TOKEN} ``` -Skipped cells still produce manifests (with `status: skipped`, `reason: "insufficient_nodes ..."`), still appear in `pytest --collect-only` with a `skipped_insufficient_nodes` marker, and remain queryable in dashboards. Partial coverage on a small cluster is a feature, not an error. +### Each category, one paragraph -The binder is **deterministic** (first-fit by cluster-file declaration order): same cluster + same config → same bindings, always. This is load-bearing for future caching features. +**Identity** — `schema_version`, `test_id`, `target_gpu`. `target_gpu` is asserted against `GpuPlatform.detect()` at config load; running a `mi355x`-targeted config on an `mi300x` cluster is a fail-fast at config load, not a 20-minute crash. -Full prose: [W5](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). +**Workload** — `framework`, `workload_kind`, `topology`. The framework Literal routes through `INFERENCE_REGISTRY` or `TRAINING_REGISTRY`. `topology.roles` declares what the workload needs (count, GPUs per node, optional label selector); the binder maps roles onto cluster nodes at run time. The same config runs on any cluster that has enough nodes matching the selector — no hostname is ever baked into the config. ---- +**Model** — single string. Becomes the `model_` marker; routes tier-6 model-specific tests. -## §8. Manifest + sidecars + cross-run analysis +**Knobs** — first-class dict for backend-stack details that benchmark engineers care to slice on. Currently used: `attention` (`aiter` / `fa` / `te`), `quant` (`fp4` / `fp8` / `bf16`), `backend` (engine variant such as `vllm-native` vs `mooncake` vs `sglang-native`), `fused_moe` (kernel variant). Each becomes a `knob__` marker; slicing the matrix on "all Mooncake configs" is one `-m "knob_backend_mooncake"` query. -### Per-run directory layout +**Params** — framework-specific scalars. Inference: `tensor_parallelism`, `max_model_length`, `num_prompts`. Training: `tp` / `pp` / `dp` / `fsdp` parallelism degrees, `micro_batch_size`, `sequence_length`. Per-framework Pydantic classes (`VllmParams`, `MegatronParams`, …) own validation; Megatron has a validator that asserts `product(parallelism) == total_gpus`, so an invalid combo is caught at parse. -```text -/vllm_gpt_oss_120b_mi355x_aiter/balanced-conc64/sha7d3a/0193a8e2-71c1.../ -├── manifest.json # 5-50 KB; metadata + verdicts + scalars + sidecar pointers -├── events.jsonl # append-only; closed vocab events -├── samples.parquet # per-request rows (long format) -├── trajectory.parquet # per-step rows (long format) -├── config.resolved.yaml # full resolved config for reproducibility -└── logs/ - ├── stdout.log - ├── stderr.log - ├── dmesg.n2.pre.txt - ├── dmesg.n2.post.txt - ├── gpu_state.n2.pre.json - └── gpu_state.n2.post.json -``` +**Sweep** — declarative axis expansion; full coverage in §5. -### Sample `manifest.json` (abbreviated, real values) +**Benchmarks** — opt-in list naming which tier-5 claim families this config asks for. Configs that don't list `convergence` won't have the `test_convergence` function collected for them, even though the function exists in the test tree. -```json -{ - "schema_version": "1.0", - "manifest_kind": "workload_run", - "run_id": "0193a8e2-71c1-7e0f-9c1a-7d5e8e1f4a02", - "test_id": "vllm_gpt_oss_120b_mi355x_aiter", - "cell_id": "balanced-conc64", - "config_hash": "sha256:91a2...e44b", - "workload_hash": "sha256:7d3a...b21f", - "verification_hash": "sha256:9c1e...8f12", - "experiment_id": "vllm/gpt-oss-120b/fp4+aiter+vllm-native/mi355x", - "cvs_version": "2.4.0", - "cvs_git_sha": "a4f1e2c", - "framework_image_digest": "sha256:7d3a...b21f", - "framework_versions": {"vllm": "0.10.2", "torch": "2.7.1", "rocm": "6.4.0"}, - "timestamp_start": "2026-05-28T20:01:08Z", - "timestamp_end": "2026-05-28T20:14:52Z", - "hosts": [{"hostname": "n1", "ip": "10.0.0.11", "role": "server", "role_index": 0}], - "model_descriptor": {"hf_repo": "openai/gpt-oss-120b", "revision": "main", "precision": "fp4"}, - "phases": { - "prepare": {"duration_s": 4.3, "status": "ok"}, - "launch": {"duration_s": 41.7, "status": "ok"}, - "await": {"duration_s": 720.0, "status": "ok"}, - "parse": {"duration_s": 1.8, "status": "ok"}, - "verify": {"duration_s": 0.1, "status": "failed"}, - "teardown": {"duration_s": 6.9, "status": "ok"} - }, - "status": "failed_verification", - "failure": { - "category": "verification_failure", - "originated_in_phase": "verify", - "message": "P99 TTFT 73.4ms exceeds threshold 50.0ms" - }, - "verdicts": [ - {"kind": "Percentile", "metric": "ttft_ms", "percentile": 99, "op": "<=", - "expected": 50.0, "actual": 73.4, "passed": false, "margin": -23.4}, - {"kind": "Percentile", "metric": "tpot_ms", "percentile": 99, "op": "<=", - "expected": 25.0, "actual": 14.2, "passed": true, "margin": 10.8}, - {"kind": "Rate", "metric": "throughput", "per_unit": "sec", "op": ">=", - "expected": 1200.0, "actual": 1318.0, "passed": true, "margin": 118.0} - ], - "result": { - "scalars": {"ttft_p99_ms": 73.4, "tpot_p99_ms": 14.2, "throughput_tps": 1318.0} - }, - "resource_summary": {"n1": {"gpu_util_mean": 78.1, "hbm_used_max_gb": 142.0, "oom_killed": false}}, - "samples_path": "samples.parquet", - "trajectory_path": "trajectory.parquet", - "events_path": "events.jsonl" -} -``` - -### `samples.parquet` schema (long format) - -| Column | Type | Semantics | Example | -|---|---|---|---| -| `request_id` | string | unique per request | `"req-00042"` | -| `ts` | timestamp | request arrival time | `2026-05-28T20:02:14.110Z` | -| `ttft_ms` | float64 | time to first token | `43.2` | -| `tpot_ms` | float64 | time per output token | `8.1` | -| `itl_ms` | float64 | inter-token latency | `7.9` | -| `e2el_ms` | float64 | end-to-end latency | `1124.0` | -| `output_tokens` | int32 | output token count | `128` | -| `role` | string | role that handled (for composites) | `"decode"` | -| `host` | string | hostname | `"n1"` | - -Long format means new metrics (`memory_pressure`, `kv_cache_util`) become new rows, not new columns — no schema migration. - -### `trajectory.parquet` schema - -| Column | Type | Example | -|---|---|---| -| `step` | int64 | `100` | -| `ts` | timestamp | `2026-05-28T20:05:00Z` | -| `metric` | string | `"loss"`, `"throughput_tps"`, `"grad_norm"`, `"router_queue"` | -| `value` | float64 | `4.21` | -| `role` | string | `"worker"`, `"router"` | -| `host` | string | `"n1"` | - -### `events.jsonl` — closed vocabulary +**Thresholds** — list of typed predicates that name a metric, an operator, and a target value or window. Six kinds: `Percentile`, `Monotonicity`, `Convergence`, `Stability`, `Rate`, `Goodput`. Direction always comes from the explicit `op:` field — never inferred from the metric name (today, the substring `"ms"` flips the comparison; a future `latency_seconds` field would invert). -| Event | Emitted by | Example payload (besides `ts`) | -|---|---|---| -| `prepare.start` | Job | `{}` | -| `prepare.done` | Job | `{phase_duration_s: 4.3}` | -| `launch.container_up` | Adapter | `{role: "server", host: "n1"}` | -| `launch.role_ready` | Adapter | `{role: "server", host: "n1"}` | -| `step` | Adapter (training) | `{step: 100, loss: 4.21, throughput: 1342}` | -| `request` | Adapter (inference) | `{request_id: "...", ttft_ms: 43.2}` | -| `safety.violated` | Job | `{predicate: "loss_is_finite", detail: "NaN at step 412"}` | -| `pattern.matched` | FailurePatternScanner | `{pattern_id: "oom_kill", source: "dmesg", line: "..."}` | -| `parse.done` | Job | `{samples_rows: 3200, trajectory_rows: 412}` | -| `verify.failed` | Job | `{metric: "ttft_ms", actual: 73.4, expected_max: 50.0}` | -| `teardown.done` | Job | `{}` | - -Adding an event name is a schema change reviewed in PR, not a free-for-all `log.info`. - -### Cross-run regression query (the data-science seam) - -```bash -# Flatten N runs into one fact table -cvs export --artifact-dir ./runs --since 30d -o fact.parquet -``` +**Topology requirements** — covered above under Workload. Worth saying again: the cluster file is a pool of nodes only (hostnames, GPUs, labels). All role assignment happens at run time in the binder, per cell. -```python -# notebook -import pandas as pd -import matplotlib.pyplot as plt +**Secrets** — `SecretValue` wrapper. Stringification redacts (``); `.reveal()` is only invoked at env-file write time inside the container. HF token never appears in command-line logs. -fact = pd.read_parquet("fact.parquet") +Full prose: [W3](cvs-dtni-v1-spec.md#w3-typed-config-schema), [W5](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). -# P99 TTFT trend for vLLM gpt-oss-120b with AITER + FP4 on mi355x, by CVS commit -sub = fact.query( - "framework == 'vllm' and model == 'gpt_oss_120b' " - "and knob_attention == 'aiter' and knob_quant == 'fp4' " - "and gpu == 'mi355x' and cell_id == 'balanced-conc64'" -) -sub.groupby("cvs_git_sha")["ttft_p99_ms"].median().plot( - marker="o", title="P99 TTFT regression (gpt-oss-120b)" -) -plt.axhline(50.0, color="red", linestyle="--", label="threshold") -plt.show() -``` +--- -Three lines of pandas catches a regression. No service, no dashboard infra required. A real dashboard (Grafana / Streamlit) reads the same Parquet. +## §5. Sweeps — how the matrix expands -Full prose: [W4](cvs-dtni-v1-spec.md#w4-manifest-sidecars-and-cross-run-export), [W8](cvs-dtni-v1-spec.md#w8-tooling-and-documentation). +A sweep declares one config that expands into N cells. Three semantics, all YAML-driven: ---- +- **Cartesian** (default): scalar lists cross with each other. +- **Paired** (no cross): a single list-of-objects, where each object is one cell. Each entry can include a `name:` that becomes the parametrize ID. +- **Constraint-validated**: per-framework `SweepParams` Pydantic classes enforce invariants (e.g. parallelism product must equal GPU count). -## §9. Sweep semantics — three walkthroughs +Topology-changing axes (P/D split, node count, parallelism degrees) carry a per-cell `topology` block; the binder re-evaluates node assignments per cell. -### Cartesian (the common case) +### Example A — Cartesian (the common case) ```yaml sweep: @@ -577,7 +368,7 @@ sweep: → **6 cells:** `[balanced-conc16]`, `[balanced-conc32]`, `[balanced-conc64]`, `[prefill_heavy-conc16]`, `[prefill_heavy-conc32]`, `[prefill_heavy-conc64]`. -### Paired with topology change (sglang P/D split) +### Example B — Paired with topology change (sglang P/D split) ```yaml sweep: @@ -595,9 +386,9 @@ sweep: concurrency: [32, 64] ``` -→ **4 cells**, binder re-evaluates per cell (different node assignments for 2p2d vs 1p3d). `cvs plan` shows the bindings before any container starts. +→ **4 cells**; binder re-evaluates per cell because each `pd_split` carries its own topology block. -### Constraint-validated (megatron parallelism) +### Example C — Constraint-validated (Megatron parallelism) ```yaml topology: @@ -609,19 +400,81 @@ sweep: - {tp: 8, pp: 1, dp: 4, fsdp: 1, name: tp8_dp4} # 8*1*4*1 = 32 ✓ - {tp: 4, pp: 2, dp: 4, fsdp: 1, name: tp4_pp2_dp4} # 4*2*4*1 = 32 ✓ - {tp: 1, pp: 1, dp: 1, fsdp: 32, name: fsdp_only} # 1*1*1*32 = 32 ✓ - # {tp: 8, pp: 2, dp: 4, fsdp: 1, name: bad} # 8*2*4*1 = 64 ✗ — Pydantic validator rejects at parse + # {tp: 8, pp: 2, dp: 4, fsdp: 1, name: bad} # 8*2*4*1 = 64 ✗ — rejected at parse micro_batch_size: [1, 2] ``` → **6 cells.** The `product(parallelism) == total_gpus` constraint is a Pydantic validator on `MegatronSweepParams`; nonsense combos fail at `model_validate()`, not 20 minutes into the run. -Full prose: [W3](cvs-dtni-v1-spec.md#w3-typed-config-schema), [W5](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion). +### How sweeps propagate into pytest + +Each cell becomes a pytest parametrize ID via its `name:` field (or an auto-derived name from scalar values). The full pipeline: + +```text +config sweep block + ↓ (sweep expansion) +N cells, each with its own resolved params + topology + ↓ (binder) +N cells, each with role → host bindings (or "skipped: insufficient_nodes") + ↓ (pytest_generate_tests) +N parametrize IDs per applicable test function + ↓ (workload_run fixture, session-scoped per cell) +N workload runs, one manifest each + ↓ (test functions read manifests) +N × M test verdicts (M = collected test functions per cell) +``` + +A cell that the cluster can't satisfy gets a manifest with `status: skipped` and a `skipped_` marker on its pytest items. The cell still appears in `cvs plan` output and remains queryable in `pytest -m`. The matrix degrades gracefully on under-resourced clusters — useful for dev boxes. + +### Dry-running the matrix + +`cvs plan --cluster cluster.json --config foo.yaml` is the matrix-preview command. Same code path as `cvs run` up to the point where the `Job` driver would call `adapter.prepare()`, then prints what would happen and exits. Output includes: cells, per-cell role-to-host bindings, selected test functions per cell, skip reasons, estimated wall time. Useful to catch "config doesn't fit cluster" in 2 seconds rather than 30 minutes. + +Full prose: [W5](cvs-dtni-v1-spec.md#w5-cluster-pool-deterministic-binder-sweep-expansion), [W8](cvs-dtni-v1-spec.md#w8-tooling-and-documentation). --- -## §10. Threshold predicates +## §6. Metrics and benchmarks supported per framework + +The "framework emits → CVS retains" principle (§1) means each framework's native telemetry routes directly into the manifest's Parquet sidecars. The contract is uniform across adapters: `samples.parquet` is request-grained or sample-grained (inference); `trajectory.parquet` is time-grained (training, or inference time-series). Benchmarks (tier-5 claims) opt in per config. + +### Captured metrics per framework -Six kinds; explicit `op:` (never inferred from metric name): +| Framework | `samples.parquet` columns | `trajectory.parquet` series | +|---|---|---| +| **vllm** | `request_id`, `ttft_ms`, `tpot_ms`, `itl_ms`, `e2el_ms`, `output_tokens` | `queue_depth`, `memory_pressure`, `gpu_util` | +| **sglang_disagg** | (above) + `kv_transfer_ms` (P→D handoff), `prefill_done_ns`, `decode_start_ns` | (above) + `router_queue`, `decode_kv_cache_util`, `per_role_startup_ms` | +| **inferencemax** | per-batch `latency_ms`, `tokens`, `throughput_tps` | per-batch series | +| **pytorch_xdit (Flux)** | per-image `generation_ms`, `prompt_id`, `seed`, `num_inference_steps` | per-step `latency_ms` | +| **pytorch_xdit (Wan)** | per-frame `latency_ms`, `frame_idx`, `bitrate_kbps` | `fps`, `aggregate_bitrate` | +| **megatron** | (training: trajectory only) | `loss`, `throughput`, `step_time_ms`, `grad_norm`, `mem_used_gb` (per-rank) | +| **jax** | (training: trajectory only) | `loss`, `throughput`, `step_time_ms`, per-host metrics from coordinator | + +Long-format Parquet means adding a new metric is a new row, not a new column — no schema migration on existing manifests. A new field that's already in the framework's emission costs one line in `parse()` and zero elsewhere. + +### Benchmarks (tier-5 claims) supported per framework + +| Benchmark | Threshold kind | Frameworks supported | +|---|---|---| +| `throughput` | `Rate` | vllm, sglang, inferencemax, xdit, megatron, jax | +| `ttft_p99` | `Percentile` | vllm, sglang, inferencemax | +| `tpot_p99` | `Percentile` | vllm, sglang, inferencemax | +| `itl_p99` | `Percentile` | vllm, sglang | +| `goodput` | `Goodput` | vllm, sglang | +| `handoff_latency` | `Percentile` | sglang_disagg | +| `router_balance` | (progress predicate) | sglang_disagg | +| `image_throughput` | `Rate` | xdit (Flux) | +| `step_time_p99` | `Percentile` | xdit | +| `video_throughput` | `Rate` | xdit (Wan) | +| `convergence` | `Convergence` | megatron, jax | +| `loss_finite` | (progress predicate) | megatron, jax | +| `monotonic_loss` | `Monotonicity` | megatron, jax | +| `step_time_stability` | `Stability` | megatron, jax | +| `no_straggler` | (progress predicate, distributed only) | megatron, jax (distributed) | + +### Threshold predicates + +All six kinds, as they appear in config: ```yaml # 1. Percentile (over samples) @@ -642,140 +495,206 @@ Six kinds; explicit `op:` (never inferred from metric name): - {kind: Rate, metric: throughput, per_unit: sec, op: ">=", min_rate: 1200} # 6. Goodput (filtered rate — the MLPerf headline) -- {kind: Goodput, - metric_pair: {ttft: ttft_ms, tpot: tpot_ms}, - ttft_max_ms: 450, tpot_max_ms: 40, - op: ">=", min_qps: 600} +- {kind: Goodput, metric_pair: {ttft: ttft_ms, tpot: tpot_ms}, + ttft_max_ms: 450, tpot_max_ms: 40, op: ">=", min_qps: 600} ``` -Each evaluates against the manifest's `samples` or `trajectory` carriers and emits a `Verdict` row with `expected`, `actual`, `passed`, `margin`. The `margin` field powers regression alerts ("p99 TTFT margin shrank from +12 ms to +2 ms over the last 10 runs" is one DuckDB query away). +Each evaluates against the manifest's `samples` or `trajectory` carriers and emits a `Verdict` row with `expected`, `actual`, `passed`, `margin`. The `margin` field powers regression alerts — "P99 TTFT margin shrank from +12 ms to +2 ms over the last 10 runs" is one DuckDB query away (§7). -Full prose: [W3](cvs-dtni-v1-spec.md#w3-typed-config-schema). +### Adding new coverage ---- +The three common add-flows are intentionally cheap: -## §11. Failure walkthrough — `safety_violation` end-to-end +- **New metric on an existing framework** — one extra column write in `parse()`. No schema migration. Existing manifests don't know about the new metric; new manifests do. +- **New benchmark on an existing config** — append the name to `benchmarks: [...]` in the YAML. The matching tier-5 test function collects automatically. +- **New threshold predicate kind** — new Pydantic class + an evaluator function. Doesn't invalidate existing manifests (they don't reference the new kind). -A training run hits `NaN` loss at step 412. Here's the trail. +Full prose: [W2](cvs-dtni-v1-spec.md#w2-six-concrete-adapters), [W3](cvs-dtni-v1-spec.md#w3-typed-config-schema). -**1. Adapter's `progress_predicate` returns ProgressStatus(ok=False, predicate_name="loss_is_finite", evidence="NaN at step 412").** +--- -**2. `Job._await_with_progress` raises `SafetyViolation`. `Job.run`'s `except` clause classifies and records:** +## §7. Manifest and sidecars — durable runs and regression analysis -```python -except SafetyViolation as e: - self.manifest.record_failure("safety_violation", - predicate=e.predicate, - evidence=e.evidence) -``` +The manifest is the contract between "the workload ran" and every downstream consumer (tests, dashboards, regression analysis). It exists on disk at a content-addressable path, survives pytest's process, and is the single artifact every test function reads. Today, a CVS run prints results to stdout and forgets them; v1 makes the run a queryable record. -**3. `events.jsonl` gets a row:** +### Per-run directory layout -```json -{"ts": "2026-05-28T20:08:33.012Z", "event": "safety.violated", - "predicate": "loss_is_finite", "detail": "NaN at step 412"} +```text +///// +├── manifest.json # 5-50 KB; metadata + verdicts + scalars + sidecar pointers +├── events.jsonl # append-only; closed-vocab events +├── samples.parquet # per-request rows (long format) +├── trajectory.parquet # per-step rows (long format) +├── config.resolved.yaml # full resolved config for reproducibility +└── logs/ + ├── stdout.log + ├── stderr.log + ├── dmesg..pre.txt + ├── dmesg..post.txt + └── gpu_state..pre.json + └── gpu_state..post.json ``` -**4. `manifest.json` reflects:** +Content-addressable directory key = `` of (workload-defining inputs + framework image digest + bindings). Same config + same cluster always lands at the same path. + +### Sample `manifest.json` (abbreviated, real values) ```json -"status": "failed_safety", -"failure": { - "category": "safety_violation", - "originated_in_phase": "await_completion", - "originated_at_ts": "2026-05-28T20:08:33.012Z", - "message": "loss_is_finite broke: NaN at step 412" +{ + "schema_version": "1.0", + "run_id": "0193a8e2-71c1-7e0f-9c1a-7d5e8e1f4a02", + "test_id": "vllm_gpt_oss_120b_mi355x_aiter", + "cell_id": "balanced-conc64", + "config_hash": "sha256:91a2...e44b", + "workload_hash": "sha256:7d3a...b21f", + "verification_hash": "sha256:9c1e...8f12", + "experiment_id": "vllm/gpt-oss-120b/fp4+aiter+vllm-native/mi355x", + "cvs_git_sha": "a4f1e2c", + "framework_image_digest": "sha256:7d3a...b21f", + "framework_versions": {"vllm": "0.10.2", "torch": "2.7.1", "rocm": "6.4.0"}, + "timestamp_start": "2026-05-28T20:01:08Z", + "timestamp_end": "2026-05-28T20:14:52Z", + "hosts": [{"hostname": "n1", "ip": "10.0.0.11", "role": "server"}], + "model_descriptor": {"hf_repo": "openai/gpt-oss-120b", "precision": "fp4"}, + "phases": { + "prepare": {"duration_s": 4.3, "status": "ok"}, + "launch": {"duration_s": 41.7, "status": "ok"}, + "await": {"duration_s": 720.0, "status": "ok"}, + "parse": {"duration_s": 1.8, "status": "ok"}, + "verify": {"duration_s": 0.1, "status": "failed"}, + "teardown": {"duration_s": 6.9, "status": "ok"} + }, + "status": "failed_verification", + "failure": { + "category": "verification_failure", + "originated_in_phase": "verify", + "message": "P99 TTFT 73.4ms exceeds threshold 50.0ms" + }, + "verdicts": [ + {"kind": "Percentile", "metric": "ttft_ms", "op": "<=", + "expected": 50.0, "actual": 73.4, "passed": false, "margin": -23.4}, + {"kind": "Percentile", "metric": "tpot_ms", "op": "<=", + "expected": 25.0, "actual": 14.2, "passed": true, "margin": 10.8}, + {"kind": "Rate", "metric": "throughput", "op": ">=", + "expected": 1200.0, "actual": 1318.0, "passed": true, "margin": 118.0} + ], + "result": {"scalars": {"ttft_p99_ms": 73.4, "tpot_p99_ms": 14.2, "throughput_tps": 1318.0}}, + "samples_path": "samples.parquet", + "trajectory_path": "trajectory.parquet", + "events_path": "events.jsonl" } ``` -**5. `finally` block runs `adapter.teardown(run)` — containers `docker rm`'d by label, logs + dmesg captured.** +### Sidecar schemas -**6. pytest terminal summary, end of session:** +**`samples.parquet`** (long-format — one row per request/sample): -```text -FAILED training/test_trajectory.py::test_loss_finite[megatron-llama_70b-tp8_dp4-mbs2] - safety_violation: loss_is_finite broke at step 412 - manifest: runs/megatron_llama_70b/tp8_dp4-mbs2/.../manifest.json -``` +| Column | Type | Semantics | +|---|---|---| +| `request_id` | string | unique per request | +| `ts` | timestamp | request arrival | +| `ttft_ms` | float64 | time to first token | +| `tpot_ms` | float64 | time per output token | +| `itl_ms` | float64 | inter-token latency | +| `e2el_ms` | float64 | end-to-end latency | +| `output_tokens` | int32 | output token count | +| `role` | string | role that handled (for composites) | +| `host` | string | hostname | + +**`trajectory.parquet`** (long-format — one row per (step, metric)): -The pytest-html report carries the same line plus the manifest path; from there, the notebook query in §8 finds whether this is a one-off or a regression. +| Column | Type | Example | +|---|---|---| +| `step` | int64 | `100` | +| `ts` | timestamp | `2026-05-28T20:05:00Z` | +| `metric` | string | `"loss"`, `"throughput_tps"`, `"router_queue"` | +| `value` | float64 | `4.21` | +| `role` | string | `"worker"`, `"router"` | +| `host` | string | `"n1"` | -Full prose: [W1](cvs-dtni-v1-spec.md#w1-core-lifecycle-and-adapter-framework), [W8](cvs-dtni-v1-spec.md#w8-tooling-and-documentation). +**`events.jsonl`** — closed vocabulary, one line per event: ---- +| Event | Emitted by | Payload (besides `ts`) | +|---|---|---| +| `prepare.start` / `prepare.done` | Job | `phase_duration_s` | +| `launch.container_up` / `launch.role_ready` | Adapter | `role`, `host` | +| `step` | Adapter (training) | `step`, `loss`, `throughput` | +| `request` | Adapter (inference) | `request_id`, `ttft_ms`, … | +| `safety.violated` | Job | `predicate`, `detail` | +| `pattern.matched` | FailurePatternScanner | `pattern_id`, `source`, `line` | +| `parse.done` | Job | `samples_rows`, `trajectory_rows` | +| `verify.failed` | Job | `metric`, `actual`, `expected_max` | +| `teardown.done` | Job | (empty) | -## §12. Security and correctness fixes (W7) +Adding an event is a schema change reviewed in PR, not a free-for-all `log.info`. -Eight independently-shippable fixes. None depend on the rest of the architecture; can land as a standalone PR series. +### Cross-run regression analysis (the data-science seam) -| Defect today | Fix in v1 | -|---|---| -| HF token logged plaintext in `phdl.exec` debug output on every multi-node run | `SecretValue` wrapper; `repr`/`str` redact; `.reveal()` only at env-file write; sentinel-leak CI test | -| `docker run --privileged` + `seccomp=unconfined` default | `ContainerHandle` non-privileged default; readiness probe in `__enter__`; label-scoped cleanup | -| `docker system prune --force` wipes other users' containers on shared hosts | `docker rm` by `run_id` label only | -| `fail_test()` appends to `globals.error_list` and returns; multi-step tests march past first failure | `fail_test()` calls `pytest.fail` | -| Threshold direction inferred from substring `"ms"` in metric name (any `latency_seconds` flips comparison) | explicit `op:` field in every `Threshold` | -| Inferencemax silently passes when threshold is missing from config | `InferenceMaxAdapter.verify` raises on missing threshold | -| `UnboundLocalError` in HF-token fixture on missing token file | clean `pytest.skip(reason="...")` | -| `mi355` vs `mi355x` literal inconsistency across 4 files | single `GpuPlatform.detect()` source of truth | -| `sudo rm -rf $log_dir` unquoted (user-controlled paths) | quoted everywhere; lint check in CI | +```bash +# Flatten N runs into one fact table +cvs export --artifact-dir ./runs --since 30d -o fact.parquet +``` -Full prose: [W7](cvs-dtni-v1-spec.md#w7-security-and-correctness-fixes). +```python +# notebook +import pandas as pd, matplotlib.pyplot as plt +fact = pd.read_parquet("fact.parquet") ---- +# P99 TTFT trend for vLLM gpt-oss-120b with AITER + FP4 on mi355x, by CVS commit +sub = fact.query( + "framework == 'vllm' and model == 'gpt_oss_120b' " + "and knob_attention == 'aiter' and knob_quant == 'fp4' " + "and gpu == 'mi355x' and cell_id == 'balanced-conc64'" +) +sub.groupby("cvs_git_sha")["ttft_p99_ms"].median().plot( + marker="o", title="P99 TTFT regression (gpt-oss-120b)" +) +plt.axhline(50.0, color="red", linestyle="--", label="threshold") +plt.show() +``` -## §13. Workstream DAG +Three lines of pandas catches a regression. Same Parquet file feeds a Grafana panel, a Streamlit dashboard, or a nightly alert script. No CVS service required. -```mermaid -flowchart LR - W7["W7 security/correctness
(standalone)"] - W3["W3 typed configs"] - W1["W1 lifecycle + adapters"] - W4["W4 manifest"] - W5["W5 cluster/binder/sweep"] - W2["W2 six adapters"] - W6["W6 pytest layer"] - W8["W8 tooling + docs"] - - W1 --> W2 - W3 --> W2 - W3 --> W6 - W3 --> W5 - W1 --> W6 - W4 --> W6 - W5 --> W6 - W3 --> W8 - W4 --> W8 - W5 --> W8 - W6 --> W8 -``` +### Why the manifest design makes future capabilities cheap -W7 ships first (no upstream). W8 ships last. W3 is the most depended-on. +Three things fall out of this schema for free: ---- +1. **Re-verify without re-running.** The manifest splits `workload_hash` (workload-defining inputs: framework, image digest, model, dataset, knobs, params, bindings, seed) from `verification_hash` (thresholds + pattern catalog). If only thresholds change, a future `--reuse-manifests` flag can re-evaluate verdicts against cached `samples.parquet` and rewrite the verdicts block — no workload launch. The hashes are recorded from day one even though the flag isn't shipped in v1, so the feature lands later as ~100 LOC with no historical-manifest migration. -## §14. Adoption +2. **Re-parse logs into new metrics.** Raw logs persist in `logs/`. If a new metric becomes valuable post-hoc (e.g. a tail-latency percentile we forgot to capture), a `--reparse` flag can re-run the current parser against the saved logs and rewrite `samples.parquet`. Same content-addressable path; same manifest gets a new parse-time verdict block. -- **One-shot:** `cvs migrate-config` rewrites every existing JSON under `cvs/input/config_file/{training,inference}/` to v2 YAML. No backwards-compat reader. -- **Wrappers deleted:** all 16 (7 training + 9 inference) replaced by ~6 parametrized test files. -- **CLI unchanged:** `cvs run --cluster_file=... --config_file=...` works identically. Existing automation unaffected. -- **Markers additive:** new `pytest -m "..."` queries become possible; nothing previously possible breaks. +3. **Dashboards consume Parquet directly.** Long-format columnar layout means a new metric becomes a new row group, not a schema migration. Cross-run aggregations are DuckDB one-liners. No bespoke ingestion pipeline needs to be built before the data is queryable. -Full prose: [Migration story in spec](cvs-dtni-v1-spec.md#migration-story). +Full prose: [W4](cvs-dtni-v1-spec.md#w4-manifest-sidecars-and-cross-run-export), [W8](cvs-dtni-v1-spec.md#w8-tooling-and-documentation). --- -## §15. Decision wanted +## Appendix -**W7 as a standalone PR series, or merged with the rest?** The eight security/correctness fixes (§12) have zero upstream dependencies in the redesign. Reviewer preference welcome on: +### Adoption -- **Land first, standalone:** smaller PRs, faster review, fixes ship before the architecture work completes. Recommended. -- **Land bundled with v1:** one cohesive review, but blocks security fixes behind architecture review cycle. +- **One-shot config migration.** `cvs migrate-config` rewrites every existing JSON under `cvs/input/config_file/{training,inference}/` to v2 YAML in a single pass. The new schema is the only schema; no backwards-compat reader path. +- **Wrappers deleted.** The 16 existing wrappers under `cvs/tests/{training,inference}/` (7 training + 9 inference) are replaced by ~6 parametrized test files. Adding a new model is a new YAML, not a new Python file. +- **CLI unchanged.** `cvs run --cluster_file=... --config_file=...` works identically. Existing automation continues to work. +- **Markers are additive.** New `pytest -m "..."` queries become possible; nothing previously possible breaks. ---- +### Security and correctness fixes (W7) + +W7 is a set of eight latent-defect fixes that ride alongside the redesign but have no architectural dependency on it. They can ship as a standalone PR series before the rest of v1 lands. Highlights: + +- `SecretValue` wrapper for HF tokens — type-level redaction; `.reveal()` only at env-file write time; sentinel-leak CI test. +- `ContainerHandle` — non-privileged default; readiness probe in `__enter__`; label-scoped cleanup; **no more `docker system prune --force`** wiping other users' containers on shared nodes. +- `fail_test()` actually calls `pytest.fail` (today it appends to `globals.error_list` and returns). +- Explicit `op:` field in every `Threshold` — kills the substring-`"ms"` direction inference. +- `InferenceMaxAdapter.verify` raises on missing threshold instead of silently passing. +- `mi355` / `mi355x` literal normalization via a single GPU-family detection path. +- Quoted `$log_dir` in every `sudo rm -rf` call. + +### Open decision for reviewers + +**W7 as a standalone PR series, or merged with the rest?** Recommended: standalone first. Smaller PRs, faster review, security fixes ship without waiting on the architecture work. -## §16. Pointers +### Pointers - Full spec (long-form prose): [`docs/prd/cvs-dtni-v1-spec.md`](cvs-dtni-v1-spec.md) -- Original architecture PRD (superseded): preserved at commit [`0d0dd1a`](https://github.com/ROCm/cvs/blob/0d0dd1a/docs/prd/cvs-dtni-suite-expansion-prd.md) on this branch -- Per-workstream history: see commits `625c8f2`, `f6c84ba`, `0d0dd1a`, and HEAD on `atnair/prd-dtni-refactor` +- Original architecture PRD (superseded): commit [`0d0dd1a`](https://github.com/ROCm/cvs/blob/0d0dd1a/docs/prd/cvs-dtni-suite-expansion-prd.md) on this branch From daebfb8f1155f6f2ae8b105e48505503b966cdbe Mon Sep 17 00:00:00 2001 From: Atul Nair Date: Fri, 5 Jun 2026 12:53:02 -0400 Subject: [PATCH 7/7] Sync PRD adapter contract with shipped implementation The pr-body.md adapter-contract block described a threaded AdapterRun object with rich typed params/returns (Context, WorkloadResult, Manifest, Threshold, Verdict) that never shipped. The real WorkloadAdapter Protocol threads a single ctx (RunContext) through all seven methods and returns None/List, with progress_predicate declared ahead of await_completion. Also document the multi-role launch/readiness plumbing (_launch_role, _wait_http_pool) that PR-A2 (#214) moved into BaseWorkloadAdapter, in both the base-adapter prose and the adapter-tree diagram, and align the W1 method list ordering in the spec. --- docs/prd/cvs-dtni-v1-spec.md | 3 ++- docs/prd/pr-body.md | 23 +++++++++++++---------- 2 files changed, 15 insertions(+), 11 deletions(-) diff --git a/docs/prd/cvs-dtni-v1-spec.md b/docs/prd/cvs-dtni-v1-spec.md index c4196121..4bca4dd0 100644 --- a/docs/prd/cvs-dtni-v1-spec.md +++ b/docs/prd/cvs-dtni-v1-spec.md @@ -14,11 +14,12 @@ Centralize the prepare → launch → await → parse → verify → teardown li **Deliverables** -- `cvs/lib/adapter_protocol.py` — `WorkloadAdapter` Protocol with seven methods: `prepare`, `launch`, `await_completion`, `progress_predicate`, `parse`, `verify`, `teardown`. The Protocol shape is v1's commitment; it is not claimed to be a closed contract that handles every conceivable future workload. +- `cvs/lib/adapter_protocol.py` — `WorkloadAdapter` Protocol with seven methods: `prepare`, `launch`, `progress_predicate`, `await_completion`, `parse`, `verify`, `teardown`. Every method takes the same `ctx` (the `RunContext`) and communicates through it; there is no separate run object threaded between methods. The Protocol shape is v1's commitment; it is not claimed to be a closed contract that handles every conceivable future workload. - `cvs/lib/base_adapter.py` — `BaseWorkloadAdapter` (`abc.ABC`) providing concrete defaults that most adapters inherit: - `teardown` always captures container logs, dmesg snapshots, and GPU state, then removes containers by `run_id` label. - `await_completion` polls `progress_predicate` at a configurable interval and raises on timeout. - `prepare` is a no-op by default. + - `_launch_role` / `_wait_http_pool` provide the shared multi-role launch + readiness plumbing (fan out one container per host bound to a role via the per-host runner; concurrent HTTP readiness across every handle of a role), so single-role and multi-role disagg adapters share one launch path. - `cvs/lib/job.py` — `Job` driver running the six-step lifecycle. Mode-blind body: no `if mode == "training"` anywhere in the driver. Failures are classified at the boundary where they originate, not by post-hoc inspection of a stack trace. `teardown` always runs in `finally`. - `cvs/lib/failure_taxonomy.py` — five disjoint failure categories, evaluated in priority order: - `setup_failure` — `prepare` or `launch` raised before the workload started. diff --git a/docs/prd/pr-body.md b/docs/prd/pr-body.md index 11412f96..c256b1d5 100644 --- a/docs/prd/pr-body.md +++ b/docs/prd/pr-body.md @@ -52,26 +52,29 @@ The dashed arrows from `dir` indicate that the manifest tree is the single sourc ### The adapter contract -The factory hands a registered class to the Job driver; the Job driver calls these seven methods in order: +The factory hands a registered class to the Job driver, which depends on these seven lifecycle methods. Every method takes the same `ctx` (the `RunContext`) and communicates through it — there is no separate run object threaded between methods: `parse` writes carriers onto the manifest via `ctx`, and `await_completion` polls `progress_predicate`. ```python # cvs/lib/adapter_protocol.py +@runtime_checkable class WorkloadAdapter(Protocol): - def prepare(self, ctx: Context) -> None: ... - def launch(self, ctx: Context) -> AdapterRun: ... - def await_completion(self, run: AdapterRun) -> None: ... - def progress_predicate(self, run: AdapterRun) -> ProgressStatus: ... - def parse(self, run: AdapterRun, manifest: Manifest) -> WorkloadResult: ... - def verify(self, result: WorkloadResult, thresholds: list[Threshold]) -> list[Verdict]: ... - def teardown(self, run: AdapterRun) -> None: ... + framework: str + + def prepare(self, ctx: Any) -> None: ... + def launch(self, ctx: Any) -> None: ... + def progress_predicate(self, ctx: Any) -> Progress: ... + def await_completion(self, ctx: Any) -> None: ... + def parse(self, ctx: Any) -> None: ... + def verify(self, ctx: Any) -> List: ... + def teardown(self, ctx: Any) -> None: ... ``` -`BaseWorkloadAdapter` provides concrete defaults for `teardown` (always capture logs + dmesg + GPU state, then `docker rm` by label), `await_completion` (poll the predicate with timeout), and `prepare` (no-op). Most adapters override 3 of the 7 methods. +`BaseWorkloadAdapter` provides concrete defaults for `teardown` (always capture logs + dmesg + GPU state, then `docker rm` by label), `await_completion` (poll the predicate with timeout), and `prepare` (no-op). It also owns the shared **multi-role launch/readiness plumbing** — `_launch_role` (fan out one container per host bound to a role, each scoped to its per-host runner) and `_wait_http_pool` (concurrent HTTP readiness across every handle of a role) — so single-role (vLLM) and multi-role disagg (`SglangDisaggAdapter`) adapters share one launch path. Most adapters override 3 of the 7 lifecycle methods. ```mermaid flowchart TD proto["WorkloadAdapter (Protocol)"] - proto --> base["BaseWorkloadAdapter
(concrete defaults: teardown, await_completion, prepare)"] + proto --> base["BaseWorkloadAdapter
(concrete defaults: teardown, await_completion, prepare;
multi-role launch helpers: _launch_role, _wait_http_pool)"] base --> vllm["VllmAdapter"] base --> imax["InferenceMaxAdapter"] base --> sgl["SglangDisaggAdapter"]