Skip to content

Commit 4da2a15

Browse files
authored
Archive MiniCPM Seed-TTS details and retain verifiable core evidence (#435)
* Record MiniCPM S3 replay evidence and Seed-TTS validation limits Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: record MiniCPM service screens and rejected reference padding experiment Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: retain MiniCPM follow-up screens and interrupted replica experiment Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: record matched MiniCPM ngram and TensorRT development screens Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: retain MiniCPM replica and batching development evidence Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: record MiniCPM reference and batching ablations with SV evidence Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: retain MiniCPM asynchronous history service experiments Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * docs: record MiniCPM sampling and matched precision diagnostics Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * Record checked sampling and prefetch waiter experiments * Record background reference preparation regression * Normalize empty patch context lines * Record clean Flow FP16 service screen * Restore MiniCPM service evidence provenance and reproducibility Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> * Archive MiniCPM trial details and retain verifiable core evidence Publish the unchanged snapshot as a versioned evidence asset, derive compact summaries and a complete retained-trial index, and add bounded explicit verification with offline inventory integration. Agent-assisted-by: Codex (implementation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> --------- Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
1 parent 4535315 commit 4da2a15

21 files changed

Lines changed: 7671 additions & 2 deletions
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# MiniCPM Seed-TTS evidence
2+
3+
The service result remains **INCONCLUSIVE**: the retained tuning runs do not
4+
establish lower RTF than the independent engine at C1/C16/C32 with unchanged
5+
speech quality. Acceptance is [#431](https://github.com/InferMatrixLab/InferRSI/issues/431);
6+
implementation is [vLLM-Omni #10331](https://github.com/vllm-project/vllm-omni/pull/10331).
7+
8+
This is a bounded pilot of proposed [RFC #464](https://github.com/InferMatrixLab/InferRSI/issues/464).
9+
Git retains conclusions, numerical summaries, every retained trial's index,
10+
acceptance/configuration/source identities and executable checks. The versioned
11+
external archive preserves **all 56 files / 10,430,698 bytes** from
12+
`dec7a734f27215092f62728f78b98fe26cd58ba2`, byte-for-byte, including the historical
13+
README, raw request records, rejected patches and frozen validator. No experiment
14+
was rerun and no historical verdict or frozen file was rewritten.
15+
16+
## Results and limits
17+
18+
The independent S3 final's median component call latency was **4.2791 → 3.7187 ms
19+
(13.09% lower)** across three alternating measurements per arm. Exact codec tokens
20+
and lengths were checked on synthetic inputs and 16 real references. That local
21+
PASS covers only repeated 600/1200-frame S3 replay, with a two-shape capacity;
22+
it does not qualify the service. The minimal component patch and frozen validator
23+
remain in `patch/s3-component.patch` and `validator/check.py`.
24+
25+
The latest service comparisons each use 64 tuning requests, two repetitions per
26+
concurrency, disjoint warmups and the same reserved GPU. RTF below is the mean of
27+
request RTFs, averaged across the two repetitions; brackets show their min–max.
28+
These are within-server repetitions, not fresh-server uncertainty estimates.
29+
30+
| Mean RTF [repetition range] | C1 | C16 | C32 |
31+
| --- | --- | --- | --- |
32+
| Phase-14 FP32 control | 0.06070 [0.06032–0.06109] | 0.23153 [0.21428–0.24879] | 0.33311 [0.32437–0.34185] |
33+
| Phase-15 Flow FP16 | 0.06069 [0.06029–0.06109] | 0.23159 [0.19614–0.26703] | 0.30565 [0.30310–0.30819] |
34+
| Phase-14 paired engine | 0.06424 [0.06313–0.06535] | 0.21633 [0.21328–0.21937] | 0.26031 [0.25178–0.26884] |
35+
36+
All cells above completed without request failures. Flow FP16 reuses the immediately
37+
preceding control and engine; it has no new paired control server. C16/C32 still
38+
miss the target. WER, embedding-cosine SIM, separately labelled WavLM speaker
39+
verification and UTMOS are descriptive saved-audio scores. Five fresh-server,
40+
held-out comparisons and a resolved speech-quality non-inferiority result remain
41+
missing. FP16 per-replica capture logs, chunk-gap arrays and private traces are
42+
unavailable publicly; aggregate flags do not establish those observations.
43+
44+
Negative and incomplete results remain discoverable in the index and archive:
45+
46+
- The four-worker CPU reference configuration worsened C16 RTF to 0.39736 despite
47+
speeding up the isolated workload. Its isolated speedup is not a service gain.
48+
- The four-replica run was interrupted; its last C32 cell failed 61/64 requests.
49+
All six cells of that run are excluded from comparative conclusions.
50+
- Three early screens used a locally modified S3 dependency and were excluded;
51+
their original raw measurements were never published. Later screens isolate
52+
the published S3 0.3.0 wheel.
53+
- Pruned configurations, rejected unpadded/onset changes, asynchronous/batched
54+
history and sampling screens are retained. Matching HiFT precision to the
55+
running service removed the apparent microbenchmark gain; no service treatment
56+
was run for that hypothesis. The historical README preserves each interpretation.
57+
58+
## What to inspect
59+
60+
| Record | Purpose |
61+
| --- | --- |
62+
| [summary.json](results/summary.json) | All 144 service groups, counts, failures, repetition mean/min/max/stdev, quality metrics, configurations and both component assessments |
63+
| [trials.jsonl](results/trials.jsonl) | 340 stable trial IDs with archive ID, original file/JSON pointer, execution status, eligibility and conclusion/scope |
64+
| [artifacts.json](results/artifacts.json) | The sole authority for the archive URL, SHA256, bytes, format, source revision, key paths, versions, retention owner and availability limits |
65+
| [core-integrity.json](results/core-integrity.json) | Hashes of retained core files, separate from the frozen archive's original integrity map |
66+
| [pilot.json](results/pilot.json) | Git-blob size comparison, coverage and executed archive verification |
67+
| [service-review-provenance.json](plan/service-review-provenance.json) | Exact source/config chains, dependency and validator identities, launch and workload context |
68+
69+
The index contains 288 service cells, 30 diagnostics, 16 managed component
70+
executions, three pruned proposals and three excluded early screens. Execution
71+
statuses are 333 completed, one interrupted, three not run and three not recorded;
72+
completed means execution completed, not experiment accepted. Of the service cells,
73+
270 retain 17,280 request records; 18 are aggregate-only. Each source/variant/C/repeat
74+
is distinct. Per-request samples and per-utterance diagnostic rows are not trials.
75+
Coverage means all retained public result trials, not unknown private executions.
76+
77+
`plan/` retains the acceptance, RunSpec and exact config chains. Phase-5 source is
78+
reconstructible from the recorded baseline and `patch/service-phase5-source.patch`;
79+
other experimental patches remain at their original paths inside the archive.
80+
The frozen component validator retains its generic `/data/models` mount and needs
81+
the pinned private/local model and dataset assets for a GPU reproduction.
82+
83+
## Verification and retention
84+
85+
From the repository root, using Python with NumPy and PyYAML installed:
86+
87+
```bash
88+
# Explicit network access; no download is added to default CI.
89+
python evidence/minicpm-vllm-seedtts/validator/verify_archive.py --download
90+
# The same verification using a separately downloaded archive:
91+
python evidence/minicpm-vllm-seedtts/validator/verify_archive.py --archive <ARCHIVE_PATH>
92+
# Regenerate summary/index from the verified, extracted frozen bundle:
93+
python evidence/minicpm-vllm-seedtts/validator/build_core.py --snapshot <EXTRACTED_BUNDLE> --output <OUTPUT_DIR>
94+
```
95+
96+
The checker validates metadata, downloads from the pinned Release location, checks
97+
compressed bytes and SHA256 **before extraction**, bounds files/expanded bytes/time,
98+
rejects unsafe paths/links/special files, verifies all original file hashes, then
99+
recomputes the entire index and summary. Structure, IDs and integers compare exactly;
100+
finite floats use absolute tolerance 1e-12. It also runs the preserved CPU validator:
101+
270 service cells, 17,280 timestamp-derived RTFs, 12 configs, nine phase-6 bootstrap
102+
intervals and six speaker-verification records. Errors identify their stage and
103+
return nonzero without changing historical verdicts. This does not rerun GPUs,
104+
validate inaccessible audio/models or qualify the service.
105+
106+
The repository inventory references `artifacts.json` through its existing
107+
`archive-sha256` adapter. Offline checks validate metadata and report the absent
108+
local archive as unavailable; only the explicit download check establishes remote
109+
availability at that time. Proposed shared tooling in PR #328 is not a dependency.
110+
The named retention owner and immutable-version policy live in `artifacts.json`.
111+
Missing original private artifacts remain explicit; hashes alone do not prove access.
112+
113+
The public asset was uploaded and downloaded successfully before removing raw
114+
files from this branch's final tree. The pilot's 512 KiB/file and 2 MiB/net-evidence
115+
thresholds are reporting references only. **Squash merge is required** to avoid
116+
introducing this PR's earlier large blobs into main ancestry; this follow-up commit
117+
alone does not erase them from branch history. No main history rewrite or global
118+
policy/CI change is included.
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
--- /dev/null
2+
+++ b/vllm_omni/model_executor/models/minicpmo_4_5/s3_cuda_graph.py
3+
@@ -0,0 +1,52 @@
4+
+# SPDX-License-Identifier: Apache-2.0
5+
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
6+
+"""Exact-shape CUDA replay for MiniCPM's short S3 reference tokenizer."""
7+
+
8+
+from copy import copy
9+
+
10+
+import torch
11+
+from s3tokenizer.model_v2 import AudioEncoderV2, S3TokenizerV2
12+
+from vllm.config import VllmConfig
13+
+from vllm.config.multimodal import MultiModalConfig
14+
+
15+
+from .encoder_cuda_graph import EncoderCudaGraph
16+
+from .speaker_cuda_graph import _precision_policy
17+
+
18+
+
19+
+class S3CudaGraph:
20+
+ """Keep the original S3 modules, masks, arithmetic and quantizer.
21+
+
22+
+ The caller supplies a single unpadded short reference. The pinned
23+
+ s3tokenizer 0.3.0 encoder uses tensor widths for its masks and is capturable
24+
+ once the caller has moved its rotary table to the device. The existing
25+
+ graph manager bounds shapes and owns replay outputs.
26+
+ """
27+
+
28+
+ def __init__(self, tokenizer: S3TokenizerV2, vllm_config: VllmConfig, *, max_graphs: int = 2):
29+
+ if type(tokenizer.encoder) is not AudioEncoderV2:
30+
+ raise ValueError("S3 CUDA graphs require the S3 AudioEncoderV2")
31+
+ self.tokenizer = tokenizer
32+
+ config = copy(vllm_config)
33+
+ config.model_config = copy(config.model_config)
34+
+ if config.model_config.multimodal_config is None:
35+
+ config.model_config.multimodal_config = MultiModalConfig()
36+
+ self.graph = EncoderCudaGraph(self._forward, config, max_graphs=max_graphs)
37+
+ self._policy: tuple[bool, ...] | None = None
38+
+
39+
+ def _forward(self, mels: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:
40+
+ hidden, lengths = self.tokenizer.encoder(mels, lengths)
41+
+ codes = self.tokenizer.quantizer.encode(hidden)
42+
+ # The graph manager returns one owned tensor. Keep token and length
43+
+ # outputs together so both survive subsequent graph replays.
44+
+ return torch.cat((codes, lengths.to(codes.dtype).unsqueeze(1)), dim=1)
45+
+
46+
+ @torch.inference_mode()
47+
+ def __call__(self, mels: torch.Tensor, lengths: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
48+
+ policy = _precision_policy()
49+
+ if self._policy is None:
50+
+ self._policy = policy
51+
+ if policy != self._policy or torch.is_autocast_enabled("cuda"):
52+
+ hidden, code_lengths = self.tokenizer.encoder(mels, lengths)
53+
+ return self.tokenizer.quantizer.encode(hidden), code_lengths
54+
+ packed = self.graph(mels, lengths)
55+
+ return packed[:, :-1], packed[:, -1].to(lengths.dtype)
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
--- a/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py
2+
+++ b/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py
3+
@@ -1920,6 +1920,9 @@
4+
if read_bool_flag(extra, "token2wav_spk_cuda_graph", False):
5+
token2wav.enable_speaker_cuda_graph(self.vllm_config)
6+
logger.info("MiniCPM-o Code2Wav: exact-shape Torch speaker CUDA graphs enabled")
7+
+ if read_bool_flag(extra, "token2wav_s3_cuda_graph", False):
8+
+ token2wav.enable_s3_cuda_graph(self.vllm_config)
9+
+ logger.info("MiniCPM-o Code2Wav: exact-shape S3 CUDA graphs enabled")
10+
self.backend = BatchedToken2Wav(
11+
token2wav,
12+
trt_stepper=trt_stepper,
13+
--- a/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_token2wav.py
14+
+++ b/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_token2wav.py
15+
@@ -21,6 +21,7 @@
16+
17+
import numpy as np
18+
import torch
19+
+from vllm.config import VllmConfig
20+
21+
from vllm_omni.model_executor.models.step_audio2.step_audio2_token2wav import (
22+
StepAudio2Token2WavCore,
23+
@@ -140,6 +141,14 @@
24+
raise ValueError("Speaker CUDA graphs require the CUDA Torch speaker model")
25+
self._core._spk_model = SpeakerCudaGraph(speaker, vllm_config)
26+
27+
+ def enable_s3_cuda_graph(self, vllm_config: VllmConfig) -> None:
28+
+ """Replay short reference tokenization with unchanged S3 weights/math."""
29+
+ from .s3_cuda_graph import S3CudaGraph
30+
+
31+
+ if self._core.device.type != "cuda":
32+
+ raise ValueError("S3 CUDA graphs require a CUDA tokenizer")
33+
+ self._core._s3_cuda_graph = S3CudaGraph(self._core.audio_tokenizer, vllm_config)
34+
+
35+
def __call__(self, generated_speech_tokens, prompt_wav) -> bytes:
36+
"""One-shot tokens → 24 kHz WAV bytes."""
37+
return self._core.forward(
38+
--- a/vllm_omni/model_executor/models/step_audio2/step_audio2_token2wav.py
39+
+++ b/vllm_omni/model_executor/models/step_audio2/step_audio2_token2wav.py
40+
@@ -4,7 +4,7 @@
41+
import io
42+
import os
43+
import threading
44+
-from collections.abc import Iterable
45+
+from collections.abc import Callable, Iterable
46+
from dataclasses import dataclass, field
47+
from typing import Any
48+
49+
@@ -201,6 +201,7 @@
50+
self._s3_failed: set = set()
51+
self._s3_graph_lock = threading.Lock()
52+
self._s3_graph_done = None
53+
+ self._s3_cuda_graph: Callable[[torch.Tensor, torch.Tensor], tuple[torch.Tensor, torch.Tensor]] | None = None
54+
# Replay/eager counters, logged on capture and at milestones so a
55+
# benchmark log shows the realized replay rate per run.
56+
self._s3_replay_calls = 0
57+
@@ -414,6 +415,8 @@
58+
Graph replay is ~6ms with bit-identical tokens. Eager fallback
59+
off-NPU / on capture failure / bucket overflow.
60+
"""
61+
+ if self._s3_cuda_graph is not None:
62+
+ return self._s3_cuda_graph(mels, mels_lens)
63+
tok = self.audio_tokenizer
64+
frames = int(mels.shape[2])
65+
with self._s3_graph_lock:
66+
--- /dev/null
67+
+++ b/vllm_omni/model_executor/models/minicpmo_4_5/s3_cuda_graph.py
68+
@@ -0,0 +1,52 @@
69+
+# SPDX-License-Identifier: Apache-2.0
70+
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
71+
+"""Exact-shape CUDA replay for MiniCPM's short S3 reference tokenizer."""
72+
+
73+
+from copy import copy
74+
+
75+
+import torch
76+
+from s3tokenizer.model_v2 import AudioEncoderV2, S3TokenizerV2
77+
+from vllm.config import VllmConfig
78+
+from vllm.config.multimodal import MultiModalConfig
79+
+
80+
+from .encoder_cuda_graph import EncoderCudaGraph
81+
+from .speaker_cuda_graph import _precision_policy
82+
+
83+
+
84+
+class S3CudaGraph:
85+
+ """Keep the original S3 modules, masks, arithmetic and quantizer.
86+
+
87+
+ The caller supplies a single unpadded short reference. The pinned
88+
+ s3tokenizer 0.3.0 encoder uses tensor widths for its masks and is capturable
89+
+ once the caller has moved its rotary table to the device. The existing
90+
+ graph manager bounds shapes and owns replay outputs.
91+
+ """
92+
+
93+
+ def __init__(self, tokenizer: S3TokenizerV2, vllm_config: VllmConfig, *, max_graphs: int = 2):
94+
+ if type(tokenizer.encoder) is not AudioEncoderV2:
95+
+ raise ValueError("S3 CUDA graphs require the S3 AudioEncoderV2")
96+
+ self.tokenizer = tokenizer
97+
+ config = copy(vllm_config)
98+
+ config.model_config = copy(config.model_config)
99+
+ if config.model_config.multimodal_config is None:
100+
+ config.model_config.multimodal_config = MultiModalConfig()
101+
+ self.graph = EncoderCudaGraph(self._forward, config, max_graphs=max_graphs)
102+
+ self._policy: tuple[bool, ...] | None = None
103+
+
104+
+ def _forward(self, mels: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:
105+
+ hidden, lengths = self.tokenizer.encoder(mels, lengths)
106+
+ codes = self.tokenizer.quantizer.encode(hidden)
107+
+ # The graph manager returns one owned tensor. Keep token and length
108+
+ # outputs together so both survive subsequent graph replays.
109+
+ return torch.cat((codes, lengths.to(codes.dtype).unsqueeze(1)), dim=1)
110+
+
111+
+ @torch.inference_mode()
112+
+ def __call__(self, mels: torch.Tensor, lengths: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
113+
+ policy = _precision_policy()
114+
+ if self._policy is None:
115+
+ self._policy = policy
116+
+ if policy != self._policy or torch.is_autocast_enabled("cuda"):
117+
+ hidden, code_lengths = self.tokenizer.encoder(mels, lengths)
118+
+ return self.tokenizer.quantizer.encode(hidden), code_lengths
119+
+ packed = self.graph(mels, lengths)
120+
+ return packed[:, :-1], packed[:, -1].to(lengths.dtype)
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
## Outcome and user need
2+
3+
Optimize MiniCPM-o 4.5 turn-mode streaming voice cloning in vLLM-Omni for users needing lower waiting time and higher single-GPU concurrency. The user explicitly approved profiling Code2Wav, batching and inter-stage scheduling, implementing measured improvements in one vLLM-Omni PR, and contributing the evidence and reusable lessons to InferRSI.
4+
5+
## Fixed comparison
6+
7+
- Hardware: one exclusively allocated NVIDIA L20X per comparison, with private MPS; sequential engines on the same physical device.
8+
- Initial vLLM-Omni revision: d111310d548486118a2d22a943fa0a10fcff65e2. An unrelated local duplex modification is excluded from the contribution.
9+
- Independent engine revision: b6a6cb16f68fba336170e1600067af4f1a5eb5c4, profile `track_s_fast_r16`.
10+
- InferRSI runtime: a590ccb25596396a592f237105166ade4aedeaaa.
11+
- Model: openbmb/MiniCPM-o-4_5 revision 503e754207c94da6bb26850b4469f367c9ea3582.
12+
- Workload: Seed-TTS-eval English voice cloning, dataset revision 8f5e1aa2a35d42f42e940074c1983358b9491f89. Reuse the independent engine's canonical requests, disjoint warmup/tuning/evaluation pools, sampling parameters and client metric implementation.
13+
- Required concurrency: 1, 16, 32. RTF is the unweighted mean of per-request (last unique audio arrival minus dispatch) / delivered audio duration. Report each concurrency separately.
14+
15+
## Acceptance and evidence
16+
17+
The user's performance gate is strictly lower mean RTF than the independent engine at all three concurrencies, with no speech-quality regression. Historical published engine numbers are references, not a substitute for a matched local comparison.
18+
19+
Use tuning requests for development only. Final evaluation uses 320 requests at C1/C16 and 640 at C32, 2*C warmup requests, a discarded initial run and five measured fresh-server runs. Preserve failures, per-request timings and generated audio. Compare identical request and sampling inputs. Report repetitions and variance; a missing cell or unresolved measurement/quality evidence remains INCONCLUSIVE.
20+
21+
Retain exact-stage parity tests for lossless changes. Validate end-to-end speech with WER, speaker similarity and UTMOS against the frozen unmodified vLLM-Omni reference; do not approve approximations based only on speed or numerical kernel tests. No quality tolerance relaxation is authorized by this campaign. Any unresolved quality comparison prevents a PASS claim.
22+
23+
Implementation candidates and final validation must retain source/configuration identities. Keep development screens distinct from qualification. Submit only evidenced improvements, with limitations and unsuccessful hypotheses documented. PR readiness and campaign success are separate decisions.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Acceptance issue: https://github.com/InferMatrixLab/InferRSI/issues/431
2+
This subexperiment evaluates exact S3 CUDA replay at repeated 600/1200-frame shapes
3+
with real tokenizer weights and deterministic synthetic mels. Quality requires
4+
bit-identical codec tokens and lengths on those inputs and 16 distinct reference
5+
WAVs from the disjoint tuning pool, with input hashes retained. The official
6+
s3tokenizer 0.3.0 model source hash is checked, excluding the locally patched
7+
shared installation. Metric is synchronized host latency per
8+
call including graph copies. Three alternating process measurements and an
9+
independent final compare to the pristine vLLM-Omni baseline's eager tokenizer.
10+
No service RTF or generated-audio quality claim follows from this
11+
component experiment. The separate Seed-TTS service gate remains mandatory.

0 commit comments

Comments
 (0)