iframe-proxy

ilmarkov · 2026-09-28T14:29:28Z

Purpose

Fixes silent MoE corruption when running CUTLASS NVFP4 or MXFP4 MoE with expert parallelism.

CutlassExpertsFp4 and CutlassExpertsMxfp4 accepted expert_map in apply() but never used it. With EP enabled, topk_ids stayed in global expert space while weights are local, so tokens were to the wrong experts.
This showed up as degenerate outputs (e.g. ~0% GSM8K on RedHatAI/GLM-5.3-MXFP4 with DP=8 / EP=8 see issue #58729 ).

PR remaps global topk_ids local ids via expert_map before CUTLASS FP4/MXFP4 grouped GEMM.

Test Plan

gsm8k validation

server:

vllm serve RedHatAI/GLM-5.3-MXFP4 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --data-parallel-size 8 \
  --served-model-name RedHatAI/GLM-5.3-MXFP4 \
  --trust-remote-code \
  --enable-cumem-allocator \
  --enable-ep-weight-filter \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm47 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --numa-bind \
  --gpu-memory-utilization 0.87 \
  --block-size 64 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --max-cudagraph-capture-size 64 \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'

client:

lm_eval --model local-chat-completions \
  --model_args "model=RedHatAI/GLM-5.3-MXFP4,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=32,max_retries=5,tokenized_requests=False,timeout=600" \
  --tasks gsm8k \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --gen_kwargs reasoning_effort=low \
  --batch_size 1

result:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9507|±  |0.0060|
|     |       |strict-match    |     5|exact_match|↑  |0.9484|±  |0.0061|

(Result before was 0% score)

pytests:

pytest tests/kernels/moe/test_cutlass_moe.py::test_remap_topk_ids_to_local -v
pytest tests/kernels/moe/test_cutlass_moe.py::test_cutlass_fp4_moe_EP -v
pytest tests/kernels/moe/test_ocp_mx_moe.py::test_make_mxfp4_moe_kernel_validates_cutlass_expert_parallel -v
pytest tests/kernels/moe/test_ocp_mx_moe.py::test_make_mxfp4_moe_kernel_rejects_cutlass_unsupported_expert_parallel -v

New remapping kernel overhead is ~1.6 us, i.e. ~1% of Gemm kernel.

Changes:

Move the existing Triton remap_topk_to_local kernel from gpt_oss_triton_kernels_moe.py into fused_moe/utils.py and reuse it from both GPT-OSS Triton and CUTLASS paths.
Add out_dtype support (int32 for CUTLASS, int64 default for GPT-OSS) in the remap_topk_to_local kernel.
Fix misleading CutlassExpertsFp8 comment: FP8 EP already applies expert_map in moe_permute, not in the CUTLASS kernel itself.
Relax _supports_parallel_config for CutlassExpertsFp4 / CutlassExpertsMxfp4: allow EP with allgather_reducescatter; still block FlashInfer NVLink / DeepEP HT backends (same as FP8).
make_mxfp4_moe_kernel validation so compressed-tensors CUTLASS bypass fails fast on unsupported parallel configs (adapted from Add guard for Cutlass MXFP4xMXFP4 DP/EP kernel #59033 ).
Supersedes Add guard for Cutlass MXFP4xMXFP4 DP/EP kernel #59033 (guard-only PR that rejected all EP)
Fixes [Bug]: GLM MXFP4 returns 0 GSM8K on GB200 (DEP8+EP8, vLLM 0.30.0) #58729

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: Markov Ilya <markovilya197@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Markov Ilya <markovilya197@gmail.com>

mergify · 2026-10-08T21:10:21Z

Resolve test_cutlass_moe.py import conflict: keep remap_topk_to_local and kMxfp4Dynamic from main. Signed-off-by: Markov Ilya <markovilya197@gmail.com>

github-actions · 2026-10-09T14:32:33Z

✅ @ilmarkov, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

bnellnm · 2026-10-09T14:52:11Z

/ci run

github-actions · 2026-10-09T14:52:32Z

❌ This PR is 4 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

ilmarkov · 2026-10-12T09:43:44Z

/ci run

github-actions · 2026-10-12T09:44:05Z

❌ This PR is 2 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

ilmarkov · 2026-10-12T10:12:47Z

/ci run

github-actions · 2026-10-12T10:13:07Z

✅ Triggered Buildkite CI #94172 for commit 5a0c6946a195.

vllm-agent · 2026-10-12T13:41:13Z

CI selector (shadow): 34 test steps (74 jobs) instead of 70 (95 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

Selector would run (34)

amd-fp8-moe-kernels-mi355
async-engine-inputs-utils-worker
basic-models-test-other-cpu
basic-models-tests-extra-initialization ×14
batch-invariance-b200
cpu-kernel-tests ×2
cpu-multimodal-config
cpu-params-env-tokenizers-parser
distributed-dp-tests-4-gpus
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-pooling
entrypoints-integration-responses-api
fusion-e2e-tp2-ar-rms-dynamo-partition-amd
fusion-e2e-tp2-ar-rms-inductor-partition-amd
fusion-e2e-tp2-b200
fusion-e2e-tp2-quick-h100
kernels-b200 ×3
kernels-core-operation-test ×3
kernels-moe-test ×5
kernels-quantization-test ×6
kernels-root-misc-test-b200
kimi-k3-prefix-cache-4xb200 ×2
kv-offload-large
lm-eval-small-models
lora ×4
lora-tp-distributed ×4
model-executor
pytorch-compilation-unit-tests-h100
quantization ×4
quantized-moe-test-b200
v1-executor-worker
v1-kv-connectors ×4
v1-others-cpu

Would skip (today's rules run them) (49)

amd-qwen3-next-mtp-async-eplb-accuracy
ascend-npu-test
basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-tests-initialization
basic-models-tests-other
batch-invariance-h100
benchmarks-cli-test
cpu-language-generation-and-pooling-model-tests ×3
distributed-compile-unit-tests-2xh100
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-speech_to_text
fusion-e2e-quick-h100
kernels-deepgemm-test-h100
kernels-fla-ops-test-b200
kernels-fusedmoe-layer-test-2-b200s
kernels-fusedmoe-layer-test-2-h100s
kernels-mhc-test-b200
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
metrics-tracing-2-gpus
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
multi-modal-processor-cpu ×4
pytorch-compilation-dynamic-shapes
pytorch-compilation-passes-unit-tests
pytorch-compilation-unit-tests
pytorch-fullgraph-cudagraph-l4-compatibility
pytorch-fullgraph-test
regression
rl-entrypoints-tests
samplers-multimodal-beam-search
samplers-test
v1-core
v1-kv-offload
v1-logits-oracle
v1-metrics-lmeval
v1-sample
v1-spec-decode

Would add (today's rules do not run them) (13)

basic-models-tests-extra-initialization ×14 (Python record)
cpu-kernel-tests ×2 (code map)
cpu-multimodal-config (code map)
cpu-params-env-tokenizers-parser (code map)
distributed-dp-tests-4-gpus (code map)
kernels-core-operation-test ×3 (code map)
kernels-quantization-test ×6 (code map)
kimi-k3-prefix-cache-4xb200 ×2 (Python record)
kv-offload-large (Python record)
lm-eval-small-models (Python record)
lora ×4 (code map)
quantization ×4 (code map)
v1-others-cpu (code map)

AMD mirrors: would skip (64)

basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-tests-initialization
basic-models-tests-other
batch-invariance-h100
benchmarks-cli-test
deepseek-v4-kernel-test-b200
distributed-compile-unit-tests-2xh100
distributed-model-tests-2-gpus ×3
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-speech_to_text
fault-tolerance-e2e-2xh100
fusion-and-compile-unit-tests-2xb200
fusion-e2e-quick-h100
fusion-e2e-tp2-ar-rms-config-sweep-h100
fusion-e2e-tp2-b200
fusion-e2e-tp2-quick-h100
kernels-b200 ×3
kernels-fla-ops-test-b200
kernels-fusedmoe-layer-test-2-b200s
kernels-fusedmoe-layer-test-2-h100s
kernels-mhc-test-b200
kernels-root-misc-test-b200
language-models-tests-extra-standard ×2
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
metrics-tracing-2-gpus
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
openai-api-correctness
pipeline-context-parallelism-4-gpus
platform-tests
pytorch-compilation-dynamic-shapes
pytorch-compilation-passes-unit-tests
pytorch-compilation-unit-tests
pytorch-fullgraph-cudagraph-l4-compatibility
pytorch-fullgraph-test
regression
rl-entrypoints-tests
samplers-multimodal-beam-search
samplers-test
spec-decode-draft-model ×4
spec-decode-eagle-1-deepseek-qwen
spec-decode-eagle-2-llama3-qwen-vl-other
spec-decode-mtp-deepseek-mimo
spec-decode-mtp-gemma4
spec-decode-mtp-qwen3-5
spec-decode-ngram-suffix
spec-decode-speculators
v1-core
v1-kv-offload
v1-logits-oracle
v1-metrics-lmeval
v1-sample

AMD mirrors: would add (7)

distributed-dp-tests-4-gpus (code map)
kernels-core-operation-test ×3 (code map)
kernels-quantization-test ×6 (code map)
kv-offload-large (Python record)
lora ×4 (code map)
quantization ×4 (code map)
quantized-models-test (code map)

_{6 changed files · base 2534a96fcf · head 5a0c6946a1 · Python record: build 93039 at 1e5d0ea888 · kernel record: table 74f4404 (build 94125), map 74f4404 · not counted: 11 build steps, 5 A100 steps the generator no longer emits, 30 optional steps the selector would also run}

…arallel (vllm-project#59042) Signed-off-by: Markov Ilya <markovilya197@gmail.com>

Kimi-K3 with --quantization-config '{"moe": {"weight": "int4_per_group_32"}}' fails at startup on gfx942 since vllm-project#59042: ValueError: AiterMxfp4Experts does not support this MoE deployment since ... quantization scheme ... The gfx942 requant (vllm-project#51274) converts the MXFP4 experts to packed int4 at load and builds the kernel with make_mxfp4_moe_kernel(experts_cls=AiterMxfp4Experts). vllm-project#59042 added a fail-closed experts check to make_mxfp4_moe_kernel that only tries MXFP4 quant keys, which AiterMxfp4Experts accepts on gfx950 only, so the check rejects the int4 experts on gfx942. Stock decode is affected. Accept (kInt4Static32, None) in AiterMxfp4Experts on gfx942 only, and try it as the last MXFP4 scheme candidate so the check validates the int4 weights as what they are. gfx950 is unchanged: its MXFP4 candidates still match first. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com>

Kimi-K3 with --quantization-config '{"moe": {"weight": "int4_per_group_32"}}' fails at startup on gfx942 since vllm-project#59042: ValueError: AiterMxfp4Experts does not support this MoE deployment since ... quantization scheme ... The gfx942 requant (vllm-project#51274) converts the MXFP4 experts to packed int4 at load and builds the kernel with make_mxfp4_moe_kernel(experts_cls=AiterMxfp4Experts). vllm-project#59042 added a fail-closed experts check to make_mxfp4_moe_kernel that only tries MXFP4 quant keys, which AiterMxfp4Experts accepts on gfx950 only, so the check rejects the int4 experts on gfx942. Stock decode is affected. Accept (kInt4Static32, None) in AiterMxfp4Experts on gfx942 only, and try it as the last MXFP4 scheme candidate so the check validates the int4 weights as what they are. gfx950 is unchanged: its MXFP4 candidates still match first. Tests pin AiterMxfp4Experts' quant schemes per arch and run the experts check for every AITER MXFP4 backend on the archs it is deployed on, faking the arch so any ROCm CI machine covers all of them. The gfx942 int4 cases fail on main. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com>

Support expert_map remapping in cutlass moe mxfp4

1c7267b

Signed-off-by: Markov Ilya <markovilya197@gmail.com>

ilmarkov requested review from AndreasKaratzas, WoosukKwon, mgoin, pavanimajety, tlrmchlsmth, yewentao256 and zyongye as code owners September 28, 2026 14:29

claude Bot reviewed Sep 28, 2026

View reviewed changes

mergify Bot added gpt-oss Related to GPT-OSS models nvidia labels Sep 28, 2026

github-project-automation Bot added this to gpt-oss Issues & Enhancements and NVIDIA Sep 28, 2026

mergify Bot added the bug Something isn't working label Sep 28, 2026

github-project-automation Bot moved this to To Triage in gpt-oss Issues & Enhancements Sep 28, 2026

ilmarkov mentioned this pull request Sep 28, 2026

Add guard for Cutlass MXFP4xMXFP4 DP/EP kernel #59033

Closed

4 tasks

ilmarkov added 2 commits September 28, 2026 15:19

Improve validation for MXFP4 Cutlass

8e9ebeb

Signed-off-by: Markov Ilya <markovilya197@gmail.com>

Add tests

bad9f43

Signed-off-by: Markov Ilya <markovilya197@gmail.com>

ilmarkov changed the title ~~[Bugfix][MoE] Apply expert_map remapping for CUTLASS NVFP4/MXFP4 expert parallel~~ [Bugfix][MoE] Apply expert_map remapping for CUTLASS FP4 MOE expert parallel Sep 28, 2026

robertgshaw2-redhat reviewed Sep 28, 2026

View reviewed changes

Comment thread vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py

freyfwt reviewed Sep 29, 2026

View reviewed changes

Comment thread vllm/model_executor/layers/fused_moe/utils.py

Address comments

bfbf4d8

Signed-off-by: Markov Ilya <markovilya197@gmail.com>

bnellnm approved these changes Oct 8, 2026

View reviewed changes

mergify Bot added the needs-rebase label Oct 8, 2026

Merge branch 'main' into imarkov/cutlass-mxfp4-ep-support

e23027a

Resolve test_cutlass_moe.py import conflict: keep remap_topk_to_local and kMxfp4Dynamic from main. Signed-off-by: Markov Ilya <markovilya197@gmail.com>

ilmarkov requested a review from wzhao18 as a code owner October 9, 2026 13:24

mergify Bot removed the needs-rebase label Oct 9, 2026

bnellnm added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 9, 2026

Merge branch 'main' into imarkov/cutlass-mxfp4-ep-support

aa896e7

Merge branch 'main' into imarkov/cutlass-mxfp4-ep-support

5a0c694

robertgshaw2-redhat merged commit b3ee560 into vllm-project:main Oct 12, 2026
202 checks passed

github-project-automation Bot moved this from Ready to Done in gpt-oss Issues & Enhancements Oct 12, 2026

github-project-automation Bot moved this from Ready to Done in NVIDIA Oct 12, 2026

michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026

[Bugfix][MoE] Apply expert_map remapping for CUTLASS FP4 MOE expert p…

b57ac31

…arallel (vllm-project#59042) Signed-off-by: Markov Ilya <markovilya197@gmail.com>

xaguilar-amd mentioned this pull request Oct 22, 2026

[Bugfix][ROCm][Kimi-K3] Validate gfx942's int4 expert requant as int4 #63131

Merged

5 of 6 tasks

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

[Bugfix][MoE] Apply expert_map remapping for CUTLASS FP4 MOE expert parallel - #59042

[Bugfix][MoE] Apply expert_map remapping for CUTLASS FP4 MOE expert parallel#59042
robertgshaw2-redhat merged 7 commits into
vllm-project:mainfrom
neuralmagic:imarkov/cutlass-mxfp4-ep-support

ilmarkov commented Sep 28, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

bnellnm commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

ilmarkov commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

ilmarkov commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

vllm-agent commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

ilmarkov commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

bnellnm commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

ilmarkov commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

ilmarkov commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

vllm-agent commented Oct 12, 2026

CI selector (shadow): 34 test steps (74 jobs) instead of 70 (95 jobs)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ilmarkov commented Sep 28, 2026 •

edited

Loading