Repository navigation
Acceptance: MiniCPM-o 4.5 single-L20X vLLM-Omni throughput optimization #49
Description
Optimize MiniCPM-o 4.5 on vLLM-Omni for single-GPU high-concurrency throughput, while preserving speech quality and reporting TTFA p50/p95 and mean RTF. The requester explicitly selected high-concurrency throughput as the primary objective and requested implementation submissions to vLLM-Omni and evidence/knowledge submissions here.
Scope:
- Pin and measure current vLLM-Omni main aa3fd1d9b7 and SGLang-Omni main dbd923cd on one NVIDIA L20X, using the same checkpoint bytes and Seed-TTS English requests.
- Measure C=1,2,4,8,16,32. Throughput is 25 times delivered audio seconds divided by measured wall time; TTFA ends at first delivered audio samples; request RTF ends at last delivered audio samples.
- Prioritize C16/C32 throughput. Record latency tradeoffs, request failures, durations, stream chunk counts and playback stalls.
- Use Strata, vLLM-Omni, SGLang-Omni knowledge and the source-pinned minicpm-o-4.5-single-gpu-engine as hypotheses; imported results are not new measurements.
- Profile before candidate selection; prefer existing lossless configuration/runtime mechanisms and avoid changing weights, sampling, output length limits or precision to manufacture gains.
- Record all attempted configurations, startup failures and rejected candidates.
Validation before any PASS claim:
- Freeze exact workload/model hashes, reference baseline, candidate hypothesis and validator commands before candidate measurement.
- Repeated served A/B comparisons with equivalent warmup/cache policies; exploratory no-restart sweeps are explicitly exploratory.
- Speech quality using existing Seed-TTS WER and speaker similarity evaluation, with full scoring coverage; preserve baseline scores and report any degradation. Use UTMOS where a qualified scorer is available. Missing required quality evidence remains INCONCLUSIVE.
- Appropriate numerical/state regression checks for the actual modified mechanism and relevant repository checks.
- Complete plan/results/patch/validator evidence, including raw per-request measurements and repeat variability, before publishing qualified speedup claims.
Non-goals: global optimality claims, unrelated model/runtime rewrites, replacing official main with local uncommitted modifications, merging upstream PRs, or closing this issue without acceptance.
Current status: prior speaker/MPS candidates failed the frozen quality gates and remain unqualified. The requester explicitly asked to continue until throughput exceeds the source-pinned single-GPU peer engine. Round two is active and has not achieved acceptance. Final acceptance/closure remains with the requester/maintainers.
Round-two acceptance scope (requester-directed continuation):
- Exceed
InferMatrixLab/minicpm-o-4.5-single-gpu-enginerevisiona4f4778f312791c2d190b48e521141887ed160c8, profiletrack_s_fast_r11, at both C16 and C32 in repeated, equivalent single-L20X comparisons. That profile is a throughput target, not a qualified speech-quality reference. - Keep quality reference pinned to vLLM-Omni main
15bce345863c6e2db882534876b8977f5b1ae1e3and official model revision503e754207c94da6bb26850b4469f367c9ea3582. - Authorized noninferiority margins remain WER +0.20 percentage points, SIM x100 drop0.50, UTMOS drop0.05. All nine one-sided95% paired-bootstrap upper degradation bounds across C1/C16/C32 must pass (20000 draws, seed20261022).
- Reserve the audited256 previously unused prompt IDs in the frozen round-two plan. Select candidates on reused tuning data before generating this holdout. Earlier failed verdicts remain unchanged.
- New precision-path experiments are opt-in, separately recorded and subject to numerical and speech gates; no performance gain alone qualifies them. Model weights, request sampling and output-length limits stay fixed.
- Independent profiler windows are excluded from performance aggregates. Peer C1 run1 was affected by two short stack snapshots and is explicitly excluded; final paired confirmation must be unprofiled.
Frozen round-two plan, validators, rejected-candidate evidence and reproducible patches: campaign evidence. Current work tests pipeline replication, selective CFM precision, exact speaker graphs and audio-window geometry; no candidate is claimed to exceed the peer with accepted quality.
Pre-generation repeat-design addendum (frozen before any reserved holdout output): run one discarded repetition plus three measured repetitions at C1/C16/C32 with unchanged sampling seed42. Aggregate within each of the256 paired prompts (sum WER counts, average SIM/UTMOS), then bootstrap the256 prompt clusters, never768 independent waveforms. All original margins,20000 draws, seed20261022 and all-nine requirement remain unchanged. The original plan is SHA256-bound by plan/peer-round2-quality-repeats.json; eight synthetic validator controls pass. This change does not rejudge earlier FAILs.
Activity
amy-why-3459 commentedon Oct 21, 2026
New separately scoped noninferiority validation, authorized by the user to select margins: WER +0.20 percentage points, SIM (×100) −0.50, UTMOS −0.05. Use unused eval_pool[320:640], C=1/16/32, paired prompt bootstrap 20,000 resamples (seed 20261022), one-sided 95% upper bound of degradation; all nine checks must pass. Original strict zero-regression results retain their verdicts. Margins are engineering choices, not established perceptual thresholds. Select speaker-only vs speaker+MPS by repeated C16/C32 throughput before opening holdout; no holdout-driven retuning. Frozen plan: evidence/minicpmo45-l20x-20261021/plan/noninferiority-holdout.json.
amy-why-3459 commentedon Oct 21, 2026
Upstream advanced during the campaign: vLLM-Omni main
15bce34586includes #9178, which enables Torch CUDA speaker extraction by default. Historical baselineaa3fd1d9and candidatef2462435remain unchanged and the frozen unused-holdout run continues against those revisions.The PR is adapted at
227a43e898f8b96236f48f16f399431c8ad0ddcc, preserving new Torch defaults and adding explicittoken2wav_spk_cpu_reference=true; its speaker graph/IEEE TensorRT implementation matches the historical candidate. The full six-concurrency, 320-request, three-repeat comparison is being rerun against the updated main. First-cohort C1/16/32 quality uses the already authorized margins and frozen bootstrap, explicitly as post-integration regression on reused prompts, not a new untouched holdout. The selected MPS profile is unchanged; there is no quality-driven retuning or margin relaxation. Plans:plan/latest-main-addendum.jsonandplan/integrated-comparison.json.amy-why-3459 commentedon Oct 21, 2026
The separately frozen unused-cohort noninferiority test is FAIL for historical candidate
f2462435+ MPS againstaa3fd1d9. C1 passes all three metrics. C16 WER degradation is +0.4047 percentage points (one-sided 95% upper bound +0.9043); C32 is +1.4166 points (upper +4.7298), exceeding the frozen +0.20-point margin. SIM and UTMOS bounds pass at all three cells. All prompts and original scores remain included.At C32 one prompt contributes 57 additional errors against a net increase of 49. Repeated ASR submission of the candidate WAV reproduces the same result. This is a diagnostic, not a reason to delete the prompt or relax the gate. Baseline C32 ASR resubmissions differ from its original by one whitespace token; those frozen scores also remain unchanged.
Results:
results/noninferiority-holdout.json,results/holdout-quality.json.gz,results/holdout-error-attribution.json, andresults/outlier-asr-repeat.json. Updated-main/adapted-candidate measurements remain a separately scoped integration comparison; they do not overwrite this failed decision.amy-why-3459 commentedon Oct 21, 2026
Final bounded comparison is complete. Overall acceptance remains FAIL, and this issue stays open.
15bce345863c6e2db882534876b8977f5b1ae1e3versus adapted candidate tree227a43e898f8b96236f48f16f399431c8ad0ddcc: C1/2/4/8/16/32, three measured 320-request repeats each, zero measured request failures. PR head172395ef6ea072c531a89dc18ab868ff2f9ada12has the identical Git tree.Immutable evidence, tables, CSV, figures and reproduction commands. Implementation: vllm-project/vllm-omni#9188. Knowledge/evidence: #51. Both remain draft; no optimality, combined quality PASS, multilingual or full-duplex qualification is claimed.
amy-why-3459 commentedon Oct 21, 2026
Continue per user request until vLLM-Omni exceeds peer engine throughput at C16 and C32, with the existing quality margins and full six-C TTFA/RTF reporting. Peer target: a4f4778 / track_s_fast_r11, MPS enabled; its own unresolved quality gates remain explicit. Same model503e754, L20X and canonical workload. Before search, reserve256 genuinely unused prompts (original tuning indices128:384); audit includes saved service runs and stage profiling. A proposed320-prompt pool failed the pre-freeze reuse assertion and was rejected. Quality baseline remains vLLM main15bce345; WER+0.20pp, SIM×100 drop0.50, UTMOS drop0.05, all nine one-sided95 paired-bootstrap bounds. Previous FAIL verdicts unchanged. See plan/peer-round2.json for exact IDs and scope.