|
| 1 | +# MiniCPM Seed-TTS evidence |
| 2 | + |
| 3 | +The service result remains **INCONCLUSIVE**: the retained tuning runs do not |
| 4 | +establish lower RTF than the independent engine at C1/C16/C32 with unchanged |
| 5 | +speech quality. Acceptance is [#431](https://github.com/InferMatrixLab/InferRSI/issues/431); |
| 6 | +implementation is [vLLM-Omni #10331](https://github.com/vllm-project/vllm-omni/pull/10331). |
| 7 | + |
| 8 | +This is a bounded pilot of proposed [RFC #464](https://github.com/InferMatrixLab/InferRSI/issues/464). |
| 9 | +Git retains conclusions, numerical summaries, every retained trial's index, |
| 10 | +acceptance/configuration/source identities and executable checks. The versioned |
| 11 | +external archive preserves **all 56 files / 10,430,698 bytes** from |
| 12 | +`dec7a734f27215092f62728f78b98fe26cd58ba2`, byte-for-byte, including the historical |
| 13 | +README, raw request records, rejected patches and frozen validator. No experiment |
| 14 | +was rerun and no historical verdict or frozen file was rewritten. |
| 15 | + |
| 16 | +## Results and limits |
| 17 | + |
| 18 | +The independent S3 final's median component call latency was **4.2791 → 3.7187 ms |
| 19 | +(13.09% lower)** across three alternating measurements per arm. Exact codec tokens |
| 20 | +and lengths were checked on synthetic inputs and 16 real references. That local |
| 21 | +PASS covers only repeated 600/1200-frame S3 replay, with a two-shape capacity; |
| 22 | +it does not qualify the service. The minimal component patch and frozen validator |
| 23 | +remain in `patch/s3-component.patch` and `validator/check.py`. |
| 24 | + |
| 25 | +The latest service comparisons each use 64 tuning requests, two repetitions per |
| 26 | +concurrency, disjoint warmups and the same reserved GPU. RTF below is the mean of |
| 27 | +request RTFs, averaged across the two repetitions; brackets show their min–max. |
| 28 | +These are within-server repetitions, not fresh-server uncertainty estimates. |
| 29 | + |
| 30 | +| Mean RTF [repetition range] | C1 | C16 | C32 | |
| 31 | +| --- | --- | --- | --- | |
| 32 | +| Phase-14 FP32 control | 0.06070 [0.06032–0.06109] | 0.23153 [0.21428–0.24879] | 0.33311 [0.32437–0.34185] | |
| 33 | +| Phase-15 Flow FP16 | 0.06069 [0.06029–0.06109] | 0.23159 [0.19614–0.26703] | 0.30565 [0.30310–0.30819] | |
| 34 | +| Phase-14 paired engine | 0.06424 [0.06313–0.06535] | 0.21633 [0.21328–0.21937] | 0.26031 [0.25178–0.26884] | |
| 35 | + |
| 36 | +All cells above completed without request failures. Flow FP16 reuses the immediately |
| 37 | +preceding control and engine; it has no new paired control server. C16/C32 still |
| 38 | +miss the target. WER, embedding-cosine SIM, separately labelled WavLM speaker |
| 39 | +verification and UTMOS are descriptive saved-audio scores. Five fresh-server, |
| 40 | +held-out comparisons and a resolved speech-quality non-inferiority result remain |
| 41 | +missing. FP16 per-replica capture logs, chunk-gap arrays and private traces are |
| 42 | +unavailable publicly; aggregate flags do not establish those observations. |
| 43 | + |
| 44 | +Negative and incomplete results remain discoverable in the index and archive: |
| 45 | + |
| 46 | +- The four-worker CPU reference configuration worsened C16 RTF to 0.39736 despite |
| 47 | + speeding up the isolated workload. Its isolated speedup is not a service gain. |
| 48 | +- The four-replica run was interrupted; its last C32 cell failed 61/64 requests. |
| 49 | + All six cells of that run are excluded from comparative conclusions. |
| 50 | +- Three early screens used a locally modified S3 dependency and were excluded; |
| 51 | + their original raw measurements were never published. Later screens isolate |
| 52 | + the published S3 0.3.0 wheel. |
| 53 | +- Pruned configurations, rejected unpadded/onset changes, asynchronous/batched |
| 54 | + history and sampling screens are retained. Matching HiFT precision to the |
| 55 | + running service removed the apparent microbenchmark gain; no service treatment |
| 56 | + was run for that hypothesis. The historical README preserves each interpretation. |
| 57 | + |
| 58 | +## What to inspect |
| 59 | + |
| 60 | +| Record | Purpose | |
| 61 | +| --- | --- | |
| 62 | +| [summary.json](results/summary.json) | All 144 service groups, counts, failures, repetition mean/min/max/stdev, quality metrics, configurations and both component assessments | |
| 63 | +| [trials.jsonl](results/trials.jsonl) | 340 stable trial IDs with archive ID, original file/JSON pointer, execution status, eligibility and conclusion/scope | |
| 64 | +| [artifacts.json](results/artifacts.json) | The sole authority for the archive URL, SHA256, bytes, format, source revision, key paths, versions, retention owner and availability limits | |
| 65 | +| [core-integrity.json](results/core-integrity.json) | Hashes of retained core files, separate from the frozen archive's original integrity map | |
| 66 | +| [pilot.json](results/pilot.json) | Git-blob size comparison, coverage and executed archive verification | |
| 67 | +| [service-review-provenance.json](plan/service-review-provenance.json) | Exact source/config chains, dependency and validator identities, launch and workload context | |
| 68 | + |
| 69 | +The index contains 288 service cells, 30 diagnostics, 16 managed component |
| 70 | +executions, three pruned proposals and three excluded early screens. Execution |
| 71 | +statuses are 333 completed, one interrupted, three not run and three not recorded; |
| 72 | +completed means execution completed, not experiment accepted. Of the service cells, |
| 73 | +270 retain 17,280 request records; 18 are aggregate-only. Each source/variant/C/repeat |
| 74 | +is distinct. Per-request samples and per-utterance diagnostic rows are not trials. |
| 75 | +Coverage means all retained public result trials, not unknown private executions. |
| 76 | + |
| 77 | +`plan/` retains the acceptance, RunSpec and exact config chains. Phase-5 source is |
| 78 | +reconstructible from the recorded baseline and `patch/service-phase5-source.patch`; |
| 79 | +other experimental patches remain at their original paths inside the archive. |
| 80 | +The frozen component validator retains its generic `/data/models` mount and needs |
| 81 | +the pinned private/local model and dataset assets for a GPU reproduction. |
| 82 | + |
| 83 | +## Verification and retention |
| 84 | + |
| 85 | +From the repository root, using Python with NumPy and PyYAML installed: |
| 86 | + |
| 87 | +```bash |
| 88 | +# Explicit network access; no download is added to default CI. |
| 89 | +python evidence/minicpm-vllm-seedtts/validator/verify_archive.py --download |
| 90 | +# The same verification using a separately downloaded archive: |
| 91 | +python evidence/minicpm-vllm-seedtts/validator/verify_archive.py --archive <ARCHIVE_PATH> |
| 92 | +# Regenerate summary/index from the verified, extracted frozen bundle: |
| 93 | +python evidence/minicpm-vllm-seedtts/validator/build_core.py --snapshot <EXTRACTED_BUNDLE> --output <OUTPUT_DIR> |
| 94 | +``` |
| 95 | + |
| 96 | +The checker validates metadata, downloads from the pinned Release location, checks |
| 97 | +compressed bytes and SHA256 **before extraction**, bounds files/expanded bytes/time, |
| 98 | +rejects unsafe paths/links/special files, verifies all original file hashes, then |
| 99 | +recomputes the entire index and summary. Structure, IDs and integers compare exactly; |
| 100 | +finite floats use absolute tolerance 1e-12. It also runs the preserved CPU validator: |
| 101 | +270 service cells, 17,280 timestamp-derived RTFs, 12 configs, nine phase-6 bootstrap |
| 102 | +intervals and six speaker-verification records. Errors identify their stage and |
| 103 | +return nonzero without changing historical verdicts. This does not rerun GPUs, |
| 104 | +validate inaccessible audio/models or qualify the service. |
| 105 | + |
| 106 | +The repository inventory references `artifacts.json` through its existing |
| 107 | +`archive-sha256` adapter. Offline checks validate metadata and report the absent |
| 108 | +local archive as unavailable; only the explicit download check establishes remote |
| 109 | +availability at that time. Proposed shared tooling in PR #328 is not a dependency. |
| 110 | +The named retention owner and immutable-version policy live in `artifacts.json`. |
| 111 | +Missing original private artifacts remain explicit; hashes alone do not prove access. |
| 112 | + |
| 113 | +The public asset was uploaded and downloaded successfully before removing raw |
| 114 | +files from this branch's final tree. The pilot's 512 KiB/file and 2 MiB/net-evidence |
| 115 | +thresholds are reporting references only. **Squash merge is required** to avoid |
| 116 | +introducing this PR's earlier large blobs into main ancestry; this follow-up commit |
| 117 | +alone does not erase them from branch history. No main history rewrite or global |
| 118 | +policy/CI change is included. |
0 commit comments