iframe-proxy

sfeng33 · 2026-09-24T21:00:15Z

Problem

VLLM_BATCH_INVARIANT=1 forces custom all-reduce off and runs every TP all-reduce through NCCL pinned to one channel. Custom all-reduce cannot simply be re-enabled because should_custom_ar() picks the backend by tensor size, and the tensor is [num_tokens, hidden]:

flowchart LR
    A["all-reduce [tokens, hidden]"] --> B{"bytes < 8 MiB?"}
    B -- "yes (decode, small prefill)" --> C["custom kernel<br/>fp32 accumulate, round once"]
    B -- "no (large prefill batch)" --> D["NCCL<br/>rounds to bf16 at every tree step"]

The same request is reduced by different summation orders depending on what else the scheduler batched with it. Measured on 4xH100, TP=4, bf16: NCCL and the custom kernel disagree in 33% of output elements. End to end (Qwen2.5-1.5B, bs=1 vs bs=24, custom all-reduce on): 24/24 prompts diverge, starting at position 0.

Fix

Under batch invariance, make the backend choice size-independent and keep the custom kernel on a fixed reduction order:

Stop forcing disable_custom_all_reduce (config/parallel.py).
should_custom_ar() ignores the size threshold; all_reduce() reduces inputs above max_size in max_size chunks. The 1-stage kernel sums each element independently in rank order 0..N-1, so chunking is bit-exact (custom_all_reduce.py).
Pin VLLM_CUSTOM_ALLREDUCE_ALGO=1stage in override_envs_for_invariance(). The 2-stage kernel rotates the rank order per slice, which is size-dependent (fp32: 20-30% of elements differ between sizes).
Gate AITER all-reduce and QuickReduce off under batch invariance, next to the existing FlashInfer gate. Without the forced disable they would become reachable on ROCm, and neither has a fixed reduction order (cuda_communicator.py).

Chunking is chosen over sizing the IPC buffer to max_num_batched_tokens x hidden at init: no extra memory, no model info needed in the communicator, and it works for any all-reduce shape (vision towers, MoE). Outside batch invariance nothing changes.

Latency inside a CUDA graph, bf16, TP=4 H100, per all-reduce:

Test plan

All on H100, TP=4 unless noted.

New test_custom_allreduce_chunked[2,4]: 3-chunk 17.6 MiB input matches the fp32 sequential reference bitwise, eager and inside a CUDA graph. New unit test for the size gate under batch invariance. Full tests/distributed/test_custom_all_reduce.py: 19 passed, 1 skipped (SM100-only).
tests/v1/determinism/test_batch_invariance.py with VLLM_TEST_TP_SIZE=4: 22/22 pass (all attention backends), custom all-reduce now active.
tests/v1/determinism/test_online_batch_invariance.py with VLLM_TP_SIZE=4: 3/3 pass.
VLM (Qwen3-VL-2B-Instruct, image inputs) bs=1 vs bsN at TP=4 with batch invariance: bitwise identical.
Standalone bs=1 vs bs=24 on Qwen2.5-1.5B with prefix caching off: 24/24 diverge before, 0/24 after.

sfeng33 · 2026-09-24T21:04:17Z

@@ -1139,6 +1139,8 @@ def _triton_alloc_fn(size: int, alignment: int, stream: int | None):

 def override_envs_for_invariance():
    os.environ["VLLM_ALLREDUCE_USE_SYMM_MEM"] = "0"
+    # Only the 1-stage kernel has a size- and rank-independent reduction order.
+    os.environ["VLLM_CUSTOM_ALLREDUCE_ALGO"] = "1stage"


Note: this pin can be removed once #57042 lands, since it gives the 2-stage kernel the same fixed rank order as 1-stage and makes both bit-identical at every size.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mosafariuk · 2026-09-27T08:00:29Z

Glad to see #50136 (the issue behind #50505) getting this treatment — chunking neatly sidesteps the memory trade-off my init-time sizing had to cap at 256 MiB, and the shape-generality argument is a real win for the vision/MoE paths.

Two pieces from #50505 that may be useful here: (1) its standalone determinism test suite (+228 lines, both-directions bitwise checks across sizes) has zero overlap with your diff and ports directly onto the chunked path — happy to rebase it onto this PR or file it as a follow-up, whichever you prefer; (2) on the 1-stage pin: agreed it drops out once #57042 lands — that branch is mergeable against current main with the A/B evidence in-thread.

If this lands first, I'll close #50505 in its favor.

mosafariuk · 2026-09-27T09:09:41Z

One more thing from a closer read of the diff, as a question: fused_allreduce_gemma_rms_norm() is called directly by MiniMax M3, Kimi K3 and DeepSeek V3.2 (bypassing the communicator), and its _can_use_flashinfer has no batch-invariance check — so those models still switch implementations at num_tokens <= max_token_num (~146 tokens at TP=4 / 36 at TP=8 for hidden 7168 bf16, per the size table). Pre-existing on main, not introduced here — but since the PR's goal is removing size-dependent dispatch under BI, should that path be guarded here too? #50505 carried a 5-line guard + a CPU test for exactly this; it drops in without touching your design if useful.

sfeng33 · 2026-09-28T19:17:37Z

Hey @mosafariuk , thanks for the careful read! MiniMax M3, Kimi K3 and DeepSeek V3.2 (along with GLM-5.2, which shares the deepseek_v32 code) aren't on the validated model list (https://github.com/vllm-project/vllm/blob/main/docs/features/batch_invariance.md#tested-models) for batch invariance yet. The DeepSeek models on that list (V3, V3.1, R1) go through deepseek_v2, which doesn't call fused_allreduce_gemma_rms_norm(). This PR only covers the all-reduce path through the communicator, so I'd leave the fused norm path out of scope here. We can handle it when those models are brought under batch invariance.

On the 1-stage pin: #57042 can merge after this PR, and then we can drop the pin.

yewentao256

Thanks for the work!

Please do the following:

e2e accuracy using lm_eval...
e2e benchmark using vllm bench serve...

When doing e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

"lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9212|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9212|±  |0.0074|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9151|±  |0.0077|
|     |       |strict-match    |     5|exact_match|↑  |0.9158|±  |0.0076|

vllm bench serve --model zai-org/GLM-5.2-FP8  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16

# Now

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.08      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              41.50     
Output token throughput (tok/s):         41.50     
Peak output token throughput (tok/s):    76.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          340041.81 
---------------Time to First Token----------------
Mean TTFT (ms):                          1669.15   
Median TTFT (ms):                        1669.02   
P99 TTFT (ms):                           2724.38   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.23      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              39.63     
Output token throughput (tok/s):         39.63     
Peak output token throughput (tok/s):    50.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          324671.20 
---------------Time to First Token----------------
Mean TTFT (ms):                          1668.69   
Median TTFT (ms):                        1649.00   
P99 TTFT (ms):                           3126.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

mosafariuk · 2026-09-28T19:59:04Z

That scoping makes sense — with those models outside the validated list, the communicator path is the right boundary for this PR; the guard's there whenever they're brought under BI. And good to hear on the sequencing — #57042 is mergeable against current main and ready whenever this lands. Good luck with the e2e runs!

sfeng33 · 2026-09-28T20:59:01Z

Thanks @yewentao256! Here are the e2e results. Setup: Qwen/Qwen2.5-32B-Instruct (on the batch-invariance validated list), TP=4 on H200, VLLM_BATCH_INVARIANT=1 on both sides. "main" is 7ba3df6, and "this PR" is the PR's 4 commits on top of the same commit. Server command:

VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen2.5-32B-Instruct -tp 4

On main, batch invariance forces NCCL (Using [] all-reduce backends). With this PR it uses custom all-reduce (Using ['CUSTOM'] all-reduce backends).

Full raw logs: https://gist.github.com/sfeng33/da1dff7421681534e63e1b23a052ef71

Accuracy (gsm8k)

# this PR
$ lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9257/v1/completions,model=Qwen/Qwen2.5-32B-Instruct,num_concurrent=256" --tasks gsm8k

local-completions ({'base_url': 'http://127.0.0.1:9257/v1/completions', 'model': 'Qwen/Qwen2.5-32B-Instruct', 'num_concurrent': 256}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7089|±  |0.0125|
|     |       |strict-match    |     5|exact_match|↑  |0.7521|±  |0.0119|


# main
$ lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=Qwen/Qwen2.5-32B-Instruct,num_concurrent=256" --tasks gsm8k

local-completions ({'base_url': 'http://127.0.0.1:9256/v1/completions', 'model': 'Qwen/Qwen2.5-32B-Instruct', 'num_concurrent': 256}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7036|±  |0.0126|
|     |       |strict-match    |     5|exact_match|↑  |0.7384|±  |0.0121|

Performance

Prefill-heavy (8192 in / 1 out), so each forward pass exceeds custom all-reduce's 8 MiB max_size and goes through the chunked path:

# this PR
$ vllm bench serve --model Qwen/Qwen2.5-32B-Instruct --dataset-name random --host 127.0.0.1 --port 9257 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16 --seed 0 

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  50.02     
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              2.56      
Output token throughput (tok/s):         2.56      
Peak output token throughput (tok/s):    3.00      
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          20967.25  
---------------Time to First Token----------------
Mean TTFT (ms):                          25433.67  
Median TTFT (ms):                        25431.58  
P99 TTFT (ms):                           49842.67  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
$ vllm bench serve --model Qwen/Qwen2.5-32B-Instruct --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16 --seed 0 

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  138.66    
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              0.92      
Output token throughput (tok/s):         0.92      
Peak output token throughput (tok/s):    2.00      
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          7563.09   
---------------Time to First Token----------------
Mean TTFT (ms):                          70431.99  
Median TTFT (ms):                        70435.08  
P99 TTFT (ms):                           138292.88 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

Decode-heavy (1024 in / 512 out):

# this PR
$ vllm bench serve --model Qwen/Qwen2.5-32B-Instruct --dataset-name random --host 127.0.0.1 --port 9257 --random-input-len 1024 --random-output-len 512 --request-rate inf --num-prompts 256 --num-warmups 16 --seed 0 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     256       
Failed requests:                         0         
Benchmark duration (s):                  22.09     
Total input tokens:                      262144    
Total generated tokens:                  131072    
Request throughput (req/s):              11.59     
Output token throughput (tok/s):         5933.12   
Peak output token throughput (tok/s):    9216.00   
Peak concurrent requests:                256.00    
Total token throughput (tok/s):          17799.36  
---------------Time to First Token----------------
Mean TTFT (ms):                          2377.64   
Median TTFT (ms):                        1153.58   
P99 TTFT (ms):                           7601.60   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          37.79     
Median TPOT (ms):                        40.31     
P99 TPOT (ms):                           41.27     
---------------Inter-token Latency----------------
Mean ITL (ms):                           37.79     
Median ITL (ms):                         28.33     
P99 ITL (ms):                            390.46    
==================================================

# main
$ vllm bench serve --model Qwen/Qwen2.5-32B-Instruct --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 1024 --random-output-len 512 --request-rate inf --num-prompts 256 --num-warmups 16 --seed 0 --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     256       
Failed requests:                         0         
Benchmark duration (s):                  49.90     
Total input tokens:                      262144    
Total generated tokens:                  131072    
Request throughput (req/s):              5.13      
Output token throughput (tok/s):         2626.47   
Peak output token throughput (tok/s):    4352.00   
Peak concurrent requests:                256.00    
Total token throughput (tok/s):          7879.41   
---------------Time to First Token----------------
Mean TTFT (ms):                          5536.50   
Median TTFT (ms):                        2200.10   
P99 TTFT (ms):                           18342.11  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          85.75     
Median TPOT (ms):                        92.15     
P99 TPOT (ms):                           95.25     
---------------Inter-token Latency----------------
Mean ITL (ms):                           85.76     
Median ITL (ms):                         62.51     
P99 ITL (ms):                            1083.65   
==================================================

yewentao256

Thanks for the work!

Please take a look at this AI generated comment:

Does re-enabling custom AR under batch invariance also need to account for the existing 16-byte eligibility check above?

The `max_size` gate is now ignored under BI, but `inp_size % 16 != 0` can still make the backend choice depend on the batched tensor size. For example, a bf16 `[N, 14]` tensor would use NCCL for `N=1` (28 bytes) but could use custom AR for `N=4` (112 bytes).

Previously this could not affect BI because custom AR was disabled entirely. With this PR enabling it, should we ensure the remaining eligibility conditions are also batch-size invariant?

sfeng33 · 2026-09-29T17:33:36Z

Thanks @yewentao256, you are right that the inp_size % 16 check depends on the total tensor size, so in principle it could route the same request to NCCL or to custom all-reduce depending on batch size. In practice this doesn't affect the validated models. Their all-reduces are on [num_tokens, hidden_size] activations, and every validated hidden size is a multiple of 8 elements. That's why I didn't change it in this PR.
But please let me know if you prefer to close the gap anyway.

yewentao256

OK, we can have a follow up for it. Please also update the doc (saying not supported yet) and we should be ready to go.
Thanks for the work!

should_custom_ar() falls back to NCCL above max_size, so the all-reduce backend depended on the number of tokens in the batch and the same request could be reduced with two different summation orders. Under batch invariance, select the custom kernel regardless of size, reduce inputs above max_size in chunks (the 1-stage kernel reduces each element independently, so results are unchanged), and pin the 1-stage kernel. Gate the AITER and QuickReduce backends off under batch invariance since they have no fixed reduction order. Fixes vllm-project#50136 Signed-off-by: Flora Feng <4florafeng@gmail.com>

should_custom_reduce_scatter() falls back to NCCL above 16 MiB, so with custom all-reduce enabled the sequence-parallel reduce-scatter would switch backends by token count. Decline it under batch invariance, which restores the previous always-NCCL behavior; a chunked custom reduce-scatter is a follow-up. Signed-off-by: Flora Feng <4florafeng@gmail.com>

should_custom_ar() admits weak-contiguous tensors such as a transposed matrix, but the chunk loop used view(-1), which raises on non-C-contiguous strides. Flatten input and output over their storage instead, matching how the kernel already treats the buffer. Signed-off-by: Flora Feng <4florafeng@gmail.com>

Tests build the communicator with __new__, so the reduce-scatter gate must not depend on __init__ having run. Signed-off-by: Flora Feng <4florafeng@gmail.com>

Custom all-reduce is no longer disabled under VLLM_BATCH_INVARIANT. Note the remaining gap: all-reduces whose byte size is not a multiple of 16 can still switch between NCCL and custom all-reduce by batch size. Signed-off-by: sfeng33 <4florafeng@gmail.com>

mergify · 2026-10-01T19:49:55Z

Documentation preview: https://vllm--58623.org.readthedocs.build/en/58623/

sfeng33 · 2026-10-01T19:51:40Z

/ci run

github-actions · 2026-10-01T19:52:03Z

Bumps vLLM 0.31.0 -> 0.32.0. ## Dependencies - `vllm` 0.32.0 wheel. torch stays 2.13.0, flashinfer stays 0.7.0.post1. - `transformers` 5.6.2 -> 5.19.0 (vLLM 0.32 requires `>=5.16.1,<5.20`; 5.19.0 is what vLLM's CI tests). The `transformers==5.6.2` override is removed. `tokenizers` 0.22.2 -> 0.23.3. - `nvidia-cutlass-dsl` override 4.7.1 -> 4.8.0 (exact pin in vLLM 0.32). `tokenspeed-mla` 0.2.15, `humming-kernels` 0.1.17, `instanttensor` 0.2.2, `xgrammar` 0.2.8 come from vLLM; `humming-kernels` and `instanttensor` are exempt from the 7-day cutoff. - `ifbench` is excluded from the workspace: it pins `setuptools<78`, vLLM 0.32 needs `>=78.1.1`. No config or test references it. ## Patches deleted (fixed upstream in 0.32) - `monkey_patch_shm_buffer_ownership`: vllm-project/vllm#61000. - `monkey_patch_tokenize_params_validation`: vllm-project/vllm#57035 (`max_tokens` is an upper bound, not reserved from the context). - `monkey_patch_nano_v3_reasoning_parser`: upstream `nemotron_v3` parser. The Nemotron 3 Nano / 3.5 auto-detection now picks `nemotron_v3`. Configs that set `reasoning_parser = "nano_v3"` must rename it. - DeepSeek V4 `layer_types` shim (`utils/transformers_compat.py` and its vLLM patch): transformers >= 5.15 knows the V4 layer types. ## Breakages fixed - `serving_tokens.py`: `GenerateResponseChoice` is gone and `GenerateResponse` is now a discriminated union (vllm-project/vllm#58588). Subclass `GenerateTokensChoice` / `GenerateTokensResponse` and check `isinstance` against `GenerateTokensResponse`. - `prefill_logprobs`: validate with `GenerateTokensResponse` (the union alias has no `model_validate_json`). - `_patch_lora_key_prefix`: re-copied from 0.32's `LoRAModel.from_local_checkpoint`; tensorizer support was removed upstream (vllm-project/vllm#60024). - NemotronH config: transformers 5.19 renames legacy layer types (`mamba` → `linear_attention`, `attention` → `full_attention`) by setting `config.layer_types`, which was a read-only property. `layer_types` is now an `attribute_map` alias of `layers_block_type` (as upstream), and the model and conversion code use the new names. Hub configs with legacy `layers_block_type` or `hybrid_override_pattern` load as before. - Qwen3.5 tests: `from_config` through the auto class now swaps a composite config for its text config; the tests build VLMs with `Qwen3_5ForCausalLM`, as the trainer does. - `monkey_patch_kv_xfer_finished_tolerate_freed`: synced the copied method with 0.32 (`_set_kv_fetch_stage(req, KV_FETCH_COMPLETED_WAITING)`). Still needed on 0.32 (fixes land in 0.33 or are not merged): DeepSeek V4 C128 boundary, DeepSeek V4 `attn_sink` loading, GPT-OSS weight loading, KV-transfer freed assert, MiniMax think-end, routed experts with P/D connectors. Every remaining patch target still exists in 0.32; targets whose source changed were checked against the new source. ## Validation CPU: `pytest tests/unit -m "not gpu"` 854 passed; ruff clean. Import/attribute check of every `vllm` import and monkey-patch target in `src/` and `tests/` against 0.32: no missing names; patched methods whose source changed were reviewed. GPU (H200, max 2 nodes; 0.31 baselines use main's lock at the same base commit, same nodes): | Check | 0.32 | 0.31 | |---|---|---| | `tests/unit -m gpu` (NemotronH, Qwen3.5, Mamba CP, DSv4 kernels, …) | pass after the fixes above¹ | – | | Integration: reverse_text, reverse_text_moe, reverse_text_lora, reverse_text_rl_opd, reverse_text_rl_sft, reverse_text_sft, reverse_text_sft_lora, alphabet_sort | pass² | – | | Qwen3-30B-A3B RL, bf16, NCCL (15 steps, median step / mismatch KL) | 15.3 s / 0.028 | 14.9 s / 0.025 | | Qwen3-30B-A3B RL, online fp8 (`fp8_per_block`), NCCL | 15.4 s / 0.061 | 16.4 s / 0.063 | | Qwen3-30B-A3B RL, bf16, filesystem broadcast (trainer step) | 156 s / 0.030 | 155 s / 0.020 | | GLM-4.5-Air RL, router replay, 1 train + 1 infer node (10 steps, median step / mismatch KL / reward) | 23.4 s / 0.0028 / 0.49→0.65 | 24.7 s / 0.0036 / 0.44→0.66 | | Qwen3-30B-A3B P/D (NIXL 1P1D, vllm-router 0.2.3) vs colocated, router replay + sampling masks, batch-invariant: identical tokens, rows, masks, logprobs | c1 10/10, c10 20/20, uint16 c1 10/10 / c10 20/20, decode preemption 20/20 | same as 0.31 results | | GLM-4.5-Air (TP4) P/D vs prefill server queried directly (same node), batch-invariant: identical tokens and rows | 0/10 (decode diverges); 10/10 with `disable_custom_all_reduce = true`³ | 10/10 | | Long multi-turn prompt (14.4k prompt + `max_tokens` 4096, `max_model_len` 16384), plain and router-replay server | 200, prompt not truncated; `/generate` 15.9k + 4096 → 200, capped at `max_model_len`; 16.4k prompt → 400 | same | | DeepSeek-V4-Flash TP8+EP serving smoke | loads, coherent completion | – | ¹ The DSv4 `cudnn_flashmla` kernel tests fail the same way on main when vLLM's DSv4 test loads first (FlashMLA symbol clash); with #4139 applied they pass on 0.32 (123 passed). ² Each integration module's `test_dashboard` needs `playwright install --with-deps`, which fails on our nodes; all other tests pass. ³ 0.32 uses the custom all-reduce under `VLLM_BATCH_INVARIANT=1` (vllm-project/vllm#58623). Two TP>1 servers then no longer decode bit-identically, so P/D only matches colocated serving bit-for-bit with `disable_custom_all_reduce = true`. prime-rl does not set batch invariance; RL is unaffected (GLM row above). Not covered: Qwen3.5/Qwen3.8 PLE reload (no checkpoint on disk); GLM-4.5-Air disaggregated RL (needs 3 nodes). Router-replay requests need `top_k > 0` and `temperature > 0` (sampling-mask replay), on 0.31 and 0.32 alike.  --- > [!NOTE] > **Medium Risk** > Touches core inference dependencies and removes several runtime patches; misalignment with vLLM 0.32 APIs or Nemotron parser changes could break rollouts or serving until configs are updated. > > **Overview** > **Upgrades the inference stack from vLLM 0.31.0 to 0.32.0**, with matching pins: `transformers` 5.19.0, `nvidia-cutlass-dsl` 4.8.0, and direct wheel URLs. `ifbench` is dropped from the uv workspace because its `setuptools<78` pin conflicts with vLLM 0.32. > > Several **vLLM workarounds are removed** now that upstream fixed them: SHM buffer ownership on dequeue, relaxed `TokenizeParams` prompt-length checks, the custom `nano_v3` reasoning parser, and the DeepSeek V4 `layer_types` transformers shim (`transformers_compat.py`). **Nemotron 3 Nano / 3.5** auto-detection now uses the built-in **`nemotron_v3`** parser instead of `nano_v3`. > > **API follow-ups for 0.32:** token-in/token-out serving and `prefill_logprobs` use `GenerateTokensChoice` / `GenerateTokensResponse` instead of the old `GenerateResponse*` types. The LoRA checkpoint patch is re-synced from 0.32 (tensorizer path removed). The KV-transfer “freed request” patch adds `_set_kv_fetch_stage(..., KV_FETCH_COMPLETED_WAITING)` to match upstream. > > **Trainer Nemotron-H** aligns layer-type names with newer transformers (`linear_attention`, `full_attention`) via `attribute_map`, while still accepting legacy `mamba` / `attention` in `layers_block_type`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f92627e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

…lm-project#58623) Signed-off-by: Flora Feng <4florafeng@gmail.com> Signed-off-by: sfeng33 <4florafeng@gmail.com>

mergify Bot added the nvidia label Sep 24, 2026

github-project-automation Bot added this to NVIDIA Sep 24, 2026

sfeng33 commented Sep 24, 2026

View reviewed changes

sfeng33 marked this pull request as ready for review September 24, 2026 21:20

sfeng33 requested review from ProExpertProg, WoosukKwon, houseroad, mgoin, robertgshaw2-redhat, tlrmchlsmth, yewentao256 and youkaichao as code owners September 24, 2026 21:20

claude Bot reviewed Sep 24, 2026

View reviewed changes

mergify Bot added the needs-rebase label Sep 25, 2026

sfeng33 force-pushed the bi-custom-allreduce branch from 9d06984 to 8838120 Compare September 25, 2026 18:39

vllm-project deleted a comment from mergify Bot Sep 25, 2026

mergify Bot removed the needs-rebase label Sep 25, 2026

yewentao256 reviewed Sep 28, 2026

View reviewed changes

yewentao256 reviewed Sep 29, 2026

View reviewed changes

yewentao256 approved these changes Sep 30, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Sep 30, 2026

sfeng33 added 3 commits October 1, 2026 19:48

sfeng33 added 2 commits October 1, 2026 19:48

Default batch_invariant on CustomAllreduce

5dd1513

Tests build the communicator with __new__, so the reduce-scatter gate must not depend on __init__ having run. Signed-off-by: Flora Feng <4florafeng@gmail.com>

sfeng33 force-pushed the bi-custom-allreduce branch from 8838120 to 1b89712 Compare October 1, 2026 19:49

mergify Bot added the documentation Improvements or additions to documentation label Oct 1, 2026

sfeng33 enabled auto-merge (squash) October 2, 2026 00:06

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 2, 2026

sfeng33 merged commit 785b990 into vllm-project:main Oct 2, 2026
201 checks passed

sfeng33 deleted the bi-custom-allreduce branch October 2, 2026 13:30

mosafariuk mentioned this pull request Oct 6, 2026

[Core][Distributed] Size custom all-reduce buffers at init so batch-invariant mode can use them #50505

Closed

mawong-amd mentioned this pull request Oct 8, 2026

[Feature] Support batch invariance on ROCm #52231

Merged

1ucayu mentioned this pull request Oct 17, 2026

[RFC]: Logprobs/Logits Semantics and Determinism Across the vLLM Ecosystem #42259

Open

faresobeid mentioned this pull request Oct 18, 2026

chore: bump vLLM to 0.32.0 PrimeIntellect-ai/prime-rl#4140

Merged

mosafariuk mentioned this pull request Oct 19, 2026

[Bugfix] Pad unaligned custom all-reduce inputs under batch invariance #61035

Open

Size	custom 1-stage	NCCL (pinned, current BI path)
24 KiB (decode)	5 us	39 us
8 MiB	81 us	693 us

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

sfeng33 commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Test plan

Uh oh!

sfeng33 Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mosafariuk commented Sep 27, 2026

Uh oh!

mosafariuk commented Sep 27, 2026

Uh oh!

sfeng33 commented Sep 28, 2026

Uh oh!

yewentao256 left a comment

Choose a reason for hiding this comment

Uh oh!

mosafariuk commented Sep 28, 2026

Uh oh!

sfeng33 commented Sep 28, 2026

Accuracy (gsm8k)

Performance

Uh oh!

yewentao256 left a comment

Choose a reason for hiding this comment

Uh oh!

sfeng33 commented Sep 29, 2026

Uh oh!

yewentao256 left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Oct 1, 2026

Uh oh!

sfeng33 commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sfeng33 commented Sep 24, 2026 •

edited

Loading