iframe-proxy

damienlaine · 2026-06-09T13:55:27Z

Fixes for Voxtral realtime, all in the model and the realtime entrypoint. The unbounded-duration part is #45833 (stacked).

Update 2026-10-05

Reworked after @andylolu2's review: no core changes anymore. Scheduler, runner and output_processor changes dropped (#58942 covers ERROR/ABORT, #42502 the generic scheduler guard).

Fixes

Boot OOM ([Bug]: Voxtral-Mini-4B-Realtime hangs/crashes on multiple sessions due to encoder_cache_usage saturation on 16GB GPU #38233): profiling runs one audio item of max_model_len tokens. With the default max_model_len (131072) that OOMs at boot on 16 GB. New VoxtralRealtimeProcessingInfo: the profiling audio is the largest streaming chunk (the prefix, 39 tokens), {"audio": 1} per update. The item size stays max_model_len because it also sizes the encoder cache: a paused session keeps its last chunk cached until it resumes, and a cache sized to one chunk deadlocks concurrent sessions (0/12 at --max-num-batched-tokens 256).
Engine crash at max_model_len: a streaming session reaching the limit kills the EngineCore (illegal memory access) and every other session with it. VoxtralRealtimeBuffer now counts submitted tokens and ends the stream before max_model_len, so the session finishes normally.
WebSocket disconnect: cancelling the generation task doesn't abort the engine request, the slot stays held. Cleanup now aborts it.

Tests

RTX 4090 Laptop 16 GB, --hf-overrides sliding window 256, PIECEWISE:

--max-model-len 131072: main OOMs during profiling, this PR boots.
concurrent realtime streams: same transcripts, throughput and memory as with the previous runner change, no OOM. 12 sessions at --max-num-batched-tokens 256: 12/12, same as main.
--max-model-len 512, 70 s streams over websocket: streams stop at the cap, client gets transcription.done, next session OK.

pytest tests/models/multimodal/generation/test_voxtral_realtime.py -k max_model_len   # 2 passed

test_voxtral_realtime_generator_max_model_len fails with EngineDeadError without the buffer cap.

AI-assisted; reviewed and tested by me.

github-actions · 2026-06-09T13:55:43Z

mergify · 2026-06-09T13:56:20Z

Documentation preview: https://vllm--45022.org.readthedocs.build/en/45022/

mergify · 2026-06-09T21:46:50Z

Documentation preview: https://vllm--45022.org.readthedocs.build/en/45022/

hmellor · 2026-06-09T21:47:03Z

Re-opening after appeal on Slack. I'll give it a review once it's been trimmed a bit

damienlaine · 2026-06-10T09:54:41Z

Thanks for re-opening, and to recap the Slack thread here for anyone reading.

Quick context: I want Voxtral-Mini-4B-Realtime to run on a single low-VRAM GPU (≤16 GiB) and not hit a hard duration limit. Three separate bugs break it on small cards today (boot-OOM #38233, silent-hang on long silences #36015/#35863, engine crash at max_model_len), and there's a default-off flag that removes the duration cap with RoPE re-anchoring. We run this in prod at LinTO.

To be upfront: a lot of the docs and the PR body was written with AI help. I was trying to make the PR look "clean" and pull the test results, docs and fix together.

I also put everything in one big PR with flags instead of small separate ones, which is probably the other reason it's hard to review.

So: how do you want it trimmed? I can keep just the three bug fixes and move the unbounded-duration part to a later PR, or cut more. Tell me what shape you want and I'll redo it.

damienlaine · 2026-10-05T22:08:08Z

@andylolu2 one change vs your suggestion. With get_max_audio_tokens = prefix, the encoder cache shrinks to max(max_num_batched_tokens, 39). A paused session keeps its last chunk cached until it resumes (with async scheduling the last position is recomputed, so it can't be freed earlier), so at start each session pins ~41 embeds. With --max-num-batched-tokens 256, 12 sessions deadlock: 0/12 transcribed, main does 12/12.

So the subclass now only shrinks the profiling audio (get_max_audio_array_len) and keeps the item size at max_model_len, the encoder cache is the same as on main. Boot at max_model_len 131072 still OK, 12/12 at 256.

The encoder cache being sized only by max(budget, item size) is a core issue for streaming sessions in general, I can open an issue for it.

andylolu2

Great thank you @damienlaine!

Could you fix the pre-commit checks so we can trigger the CI run?

damienlaine · 2026-10-07T03:49:41Z

Thanks @andylolu2. Pre-commit passes locally on 188b1cc5e8 (ruff, mypy 3.10-3.13, typos, test tethering). On CI the pre-commit job is skipped, not failed: pre-run-check gates it on the ready/verified label or 4 merged PRs, and this is my first.

github-actions · 2026-10-07T15:20:58Z

✅ @damienlaine, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

…nk, stop the stream at max_model_len Signed-off-by: Damien Laine <damien.laine@gmail.com>

…sconnect Signed-off-by: Damien Laine <damien.laine@gmail.com>

…ep the encoder cache size A paused streaming session keeps its last chunk in the encoder cache until it resumes. Sizing the cache to one chunk deadlocks a handful of concurrent sessions when max_num_batched_tokens is small. Signed-off-by: Damien Laine <damien.laine@gmail.com>

damienlaine · 2026-10-07T15:34:56Z

/ci run

github-actions · 2026-10-07T15:36:01Z

✅ Triggered Buildkite CI #93347 for commit c87cc56a1405.

vllm-agent · 2026-10-07T15:55:02Z

CI selector (shadow): 9 test steps (34 jobs) instead of 64 (99 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	64 (99)	9 (34)	56 (68)	1 (3)
AMD mirrors	60 (87)	6 (24)	54 (63)	0 (0)

Selector would run (9)

arm-cpu-test ×3
basic-models-tests-extra-initialization ×14
basic-models-tests-other
cpu-language-generation-and-pooling-model-tests ×3
cpu-multi-modal-model-tests-n ×4
distributed-model-tests-2-gpus ×3
entrypoints-integration-speech_to_text
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4

Would skip (today's rules run them) (56)

amd-lm-eval-small-models-harness
amd-qwen3-next-mtp-async-eplb-accuracy
ascend-npu-test
async-engine-inputs-utils-worker
basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-test-other-cpu
basic-models-tests-initialization
benchmarks-cli-test
cpu-qwen2-5-vl-multimodal-tests
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-pooling
entrypoints-integration-responses-api
entrypoints-unit-tests
examples
fusion-e2e-quick-h100
fusion-e2e-tp2-b200
fusion-e2e-tp2-quick-h100
kernels-fla-ops-test-b200
kernels-mhc-test-b200
kernels-root-misc-test-b200
language-models-tests-extra-standard ×2
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
metrics-tracing-2-gpus
model-executor
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-processor-cpu ×4
pipeline-context-parallelism-4-gpus
pytorch-compilation-dynamic-shapes
pytorch-compilation-passes-unit-tests
pytorch-compilation-unit-tests
pytorch-compilation-unit-tests-h100
pytorch-fullgraph-cudagraph-l4-compatibility
pytorch-fullgraph-test
regression
rl-entrypoints-tests
v1-core
v1-executor-worker
v1-kv-connectors ×4
v1-kv-offload
v1-logits-oracle
v1-metrics-lmeval
v1-sample
v1-spec-decode

Would add (today's rules do not run them) (1)

arm-cpu-test ×3 (code map)

AMD mirrors: would skip (54)

async-engine-inputs-utils-worker
basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-tests-initialization
benchmarks-cli-test
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-pooling
entrypoints-integration-responses-api
entrypoints-unit-tests
examples
fusion-e2e-quick-h100
fusion-e2e-tp2-ar-rms-config-sweep-h100
fusion-e2e-tp2-b200
fusion-e2e-tp2-quick-h100
kernels-fla-ops-test-b200
kernels-mhc-test-b200
kernels-root-misc-test-b200
language-models-tests-extra-standard ×2
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
lm-eval-small-models
metrics-tracing-2-gpus
model-executor
mrcr-eval-small-models
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
pipeline-context-parallelism-4-gpus
platform-tests
pytorch-compilation-dynamic-shapes
pytorch-compilation-passes-unit-tests
pytorch-compilation-unit-tests
pytorch-compilation-unit-tests-h100
pytorch-fullgraph-cudagraph-l4-compatibility
pytorch-fullgraph-test
regression
rl-entrypoints-tests
v1-core
v1-executor-worker
v1-kv-connectors ×4
v1-kv-offload
v1-logits-oracle
v1-metrics-lmeval
v1-sample
v1-spec-decode

AMD mirrors: would add (0)

none

_{3 changed files · base 4187ade7b8 · head 4a77db5299 · Python record: build 93039 at 1e5d0ea888 · kernel record: table 43b4aae (build 93244), map 43b4aae · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 3 optional steps the selector would also run}

damienlaine · 2026-10-07T16:48:08Z

/amd-ci retry

github-actions · 2026-10-07T16:48:28Z

✅ No AMD CI build exists for the current PR head. Use /amd-ci run first.

damienlaine · 2026-10-07T17:05:44Z

/ci retry

github-actions · 2026-10-07T17:06:59Z

✅ Queued 3 failed job(s) for retry in Buildkite CI #93347.

damienlaine · 2026-10-07T18:34:46Z

@NickLucche @DarkLight1337 Andy approved this yesterday. It still needs a codeowner approval for entrypoints/speech_to_text and tests/models, the change in those paths is small (abort the engine request on websocket close, two tests for the max_model_len stop). The two remaining CI failures are unrelated (DeepSeek V4 dspark on ROCm, DFlashLaguna/MistralLarge3 init on h200-mig).

DarkLight1337 · 2026-10-08T04:37:32Z

/ci retry

github-actions · 2026-10-08T04:37:51Z

…x_model_len (vllm-project#45022) Signed-off-by: Damien Laine <damien.laine@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

damienlaine requested review from ApostaC, ProExpertProg, WoosukKwon, alexm-redhat, heheda12345, hmellor, houseroad, mgoin, njhill, orozery, patrickvonplaten, robertgshaw2-redhat, tlrmchlsmth, yewentao256, youkaichao and ywang96 as code owners June 9, 2026 13:55

mergify Bot added documentation Improvements or additions to documentation performance Performance-related issues v1 bug Something isn't working labels Jun 9, 2026

hmellor added the closed-as-slop Pull request determined to be low effort and agent generated label Jun 9, 2026

hmellor closed this Jun 9, 2026

hmellor removed the closed-as-slop Pull request determined to be low effort and agent generated label Jun 9, 2026

hmellor reopened this Jun 9, 2026

mergify Bot added the mistral Related to Mistral models label Jun 10, 2026

mergify Bot removed the needs-rebase label Oct 5, 2026

damienlaine mentioned this pull request Oct 5, 2026

[Bug]: Paused streaming sessions deadlock the scheduler when the KV cache is full #60137

Open

andylolu2 approved these changes Oct 6, 2026

View reviewed changes

This was referenced Oct 7, 2026

[Core][Model] Voxtral realtime: opt-in blank-run penalty to break self-sustained silence ruts #47615

Draft

[Bug]: Realtime STT (Voxtral realtime): self-sustained blank-token rut mutes live transcription for minutes over real speech #47614

Open

NickLucche added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 7, 2026

damienlaine added 3 commits October 7, 2026 17:34

[Bugfix][Model] Voxtral realtime: size profiling to the streaming chu…

8c9be92

…nk, stop the stream at max_model_len Signed-off-by: Damien Laine <damien.laine@gmail.com>

[Bugfix][Frontend] Realtime: abort the engine request on websocket di…

44efca9

…sconnect Signed-off-by: Damien Laine <damien.laine@gmail.com>

damienlaine force-pushed the voxtral-realtime-unbounded branch from 188b1cc to c87cc56 Compare October 7, 2026 15:34

DarkLight1337 approved these changes Oct 8, 2026

View reviewed changes

Merge branch 'main' into voxtral-realtime-unbounded

4a77db5

DarkLight1337 merged commit 2186b89 into vllm-project:main Oct 9, 2026
17 checks passed

This was referenced Oct 12, 2026

fix(container): backport vLLM #45022 Voxtral realtime boot fix into the vLLM runtime image ai-dynamo/dynamo#16190

Closed

[BUG]: Voxtral realtime startup profiling uses oversized audio with vLLM 0.31.0 ai-dynamo/dynamo#16243

Closed

criew mentioned this pull request Oct 13, 2026

feat(chat): Spracheingabe Stufe 2 — Live-Transkript per WebSocket criew/opaa#2615

Open

6 tasks

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

damienlaine commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update 2026-10-05

Fixes

Tests

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

mergify Bot commented Jun 9, 2026

Uh oh!

mergify Bot commented Jun 9, 2026

Uh oh!

hmellor commented Jun 9, 2026

Uh oh!

damienlaine commented Jun 10, 2026

Uh oh!

damienlaine commented Oct 5, 2026

Uh oh!

andylolu2 left a comment

Choose a reason for hiding this comment

Uh oh!

damienlaine commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

damienlaine commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

vllm-agent commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI selector (shadow): 9 test steps (34 jobs) instead of 64 (99 jobs)

Uh oh!

damienlaine commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

damienlaine commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

damienlaine commented Oct 7, 2026

Uh oh!

DarkLight1337 commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

damienlaine commented Jun 9, 2026 •

edited

Loading

vllm-agent commented Oct 7, 2026 •

edited

Loading