Purpose

Skip padded work in block-FP8 DeepGemmExperts on SM90 and SM100-family GPUs. Both grouped GEMMs use DeepGEMM's prefix-sum scheduler, and fused SiLU/SwiGLU activation quantization consumes the same live expert endpoints. This avoids visiting unused padded GEMM tiles and quantizing the full allocated workspace when only a fraction contains live rows.

The shared permutation helper produces expert endpoints for both architectures. SM100 dispatch uses packed UE8M0 activation scales; SM90 dispatch uses the column-major FP32 activation-scale path, including plain and power-of-two-rounded scales. The dispatch remains limited to block-FP8 weights and supported fused gated activations. mk_alignment_scope and DeepGEMM's default ensure_zero_padding=True remain in place. Other paths keep their existing behavior.

Relationship to existing work

This PR now includes the SM90 work from #59220 alongside the SM100 work. The latest commit restores the exact combined source tree tested before those paths were split; #59220 is being closed as superseded. #59044 enables prefix sums for FP4 weights on SM10.x but leaves FP8 callers unchanged. The shared permutation and packed-activation helpers from that PR are included here for review against main and should be reconciled when it lands. #54109 changes the padded activation-format contract and targets the FP4 DeepGEMM caller; it does not wire both FP8 GEMMs and fused activation to these endpoints. No issue is closed.

Validation

The GPU runs used the combined implementation before the split. The final commit restores that same source tree; local pre-commit, mypy, and git diff --check passed after restoration. Validation overlaid three changed source files onto a pinned vLLM image rather than rebuilding current main. DeepGEMM binaries and headers were unchanged.

SM100 on B200

Two independent single-B200 runs used Pirate (b200-02) and Pirate2 (dell-b200-02). The full-expert graph regression compared the existing FP8 scheduler with prefix sums during CUDA-graph replay, including changing routing, one live row, empty routing, GPU token-count metadata, and poisoned workspaces. Packed activation tests checked live values and scales, untouched gaps, clamping, empty experts, and changing endpoints. 75 tests passed on each cluster (32.18 s and 39.48 s), before the SM90 extension. A later B200 run of the combined source passed 123 tests covering the packed and column-major paths, before two direct GEMM tests were added.

The full-expert benchmark covered permutation, both GEMMs, activation quantization, and unpermutation/reduction, using 32 experts, FP8 E4M3 weights with 128×128 scales, intermediate size 2,048, hidden size 7,168, top-k 8, and BF16 inputs/outputs. CUPTI kernel-span timings of captured CUDA graphs used cold L2 and 100 measured samples per arm across four alternating rounds; allocation, compilation, and capture were excluded. In the heavily padded 512-capacity / 16-live-token case, latency improved from 472.176 to 347.456 µs on Pirate and 469.101 to 344.942 µs on Pirate2 (26.4–26.5%). At 4,096 / 4,096 tokens, it improved from 1,570.834 to 1,509.025 µs and 1,590.503 to 1,513.257 µs (3.9–4.9%). All benchmark outputs matched bit for bit.

Paired full GSM8K evaluation used RedHatAI/Qwen3-30B-A3B-FP8-block at revision 6b2399be90eff484848cbbabd684930e01400d54: all 1,319 questions, five-shot, greedy, seed 42, at most 256 output tokens and 64 sequences, model length 8,192, TP1/EP1, BF16 KV cache, moe_backend=deep_gemm, and full/piecewise CUDA graphs. Pirate baseline/optimized accuracy was 1,176/1,319 (89.158%) and 1,182/1,319 (89.613%); Pirate2 was 1,178/1,319 (89.310%) and 1,176/1,319 (89.158%). All runs passed the 85% threshold. Generated token sequences matched on 810 and 805 questions, respectively. These results do not establish bitwise model equivalence or an accuracy improvement.

SM90 on H200

The SM90 regression covers both projections leaving unused output capacity untouched, changing routing and endpoints during CUDA-graph replay, empty routing, and both plain and rounded FP32 column-major activation scales. 125 tests passed for each scale mode on one Waldorf H200. These runs used the combined implementation. The pinned DeepGEMM prefix-sum scheduler skips unused whole tiles; padding within a partially filled tile remains. All measured full-expert outputs matched bit for bit.

Using the same full-expert benchmark shape and timing method, the 512-capacity / 16-live-token case improved from 694.415 to 438.144 µs with plain FP32 scales and 694.735 to 438.735 µs with rounded FP32 scales (about 37%). At 4,096 / 4,096 tokens, it improved from 2,874.813 to 2,646.141 µs and 2,935.535 to 2,739.246 µs (6.7–8.0%). A ten-shape activation-only comparison found no material regression in the ordinary dense path. Expert metadata added roughly 0.3–1.8 µs for some fully occupied small shapes.

Paired full GSM8K used the same pinned model and revision, 1,319 questions, five-shot greedy decoding, seed 42, at most 256 output tokens and 256 sequences, model length 8,192, TP1/EP1, BF16 KV cache, moe_backend=deep_gemm, full/piecewise CUDA graphs, and rounded FP32 activation scales. Baseline accuracy was 1,180/1,319 (89.462%); optimized accuracy was 1,181/1,319 (89.538%). Both exceeded 85%, and 1,317 generated token sequences matched exactly. This was an accuracy check, not a model-throughput benchmark or evidence of improved accuracy.

Commands and scope

GPU commands were run with the pinned container's /tmp/check/.venv/bin/python, with extracted tests in /tmp/bundle and source overlays under /tmp/overlay:

# B200: tests, benchmark, and paired model evaluation for each variant
PYTHONPATH=/tmp/overlay /tmp/check/.venv/bin/python -m pytest /tmp/bundle/test_fp8.py /tmp/bundle/test_activation.py -q --disable-warnings --junitxml=/tmp/bundle/pytest.xml
PYTHONPATH=/tmp/overlay /tmp/check/.venv/bin/python /tmp/bundle/benchmark.py
FP8_VARIANT=dense PYTHONPATH=/tmp/eval_dense /tmp/check/.venv/bin/python /tmp/bundle/model_eval.py
FP8_VARIANT=prefix PYTHONPATH=/tmp/eval_prefix /tmp/check/.venv/bin/python /tmp/bundle/model_eval.py

# H200: repeat tests and benchmark once for each scale mode
VLLM_USE_DEEP_GEMM_E8M0=0 PYTHONPATH=/tmp/overlay /tmp/check/.venv/bin/python -m pytest /tmp/bundle/test_fp8.py /tmp/bundle/test_activation.py -q
VLLM_USE_DEEP_GEMM_E8M0=1 PYTHONPATH=/tmp/overlay /tmp/check/.venv/bin/python -m pytest /tmp/bundle/test_fp8.py /tmp/bundle/test_activation.py -q
VLLM_USE_DEEP_GEMM_E8M0=0 PYTHONPATH=/tmp/overlay /tmp/check/.venv/bin/python /tmp/bundle/benchmark.py
VLLM_USE_DEEP_GEMM_E8M0=1 PYTHONPATH=/tmp/overlay /tmp/check/.venv/bin/python /tmp/bundle/benchmark.py

test_fp8.py extracts the relevant tests and helpers from tests/kernels/moe/test_deepgemm.py and tests/kernels/moe/utils.py to avoid unrelated test-package dependencies; test_activation.py copies the activation suite. Local checks on the restored combined tree:

.venv/bin/pre-commit run --files tests/kernels/moe/test_deepgemm.py tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py vllm/model_executor/layers/fused_moe/deep_gemm_utils.py vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py vllm/model_executor/layers/quantization/utils/fp8_utils.py
.venv/bin/pre-commit run mypy-3.12 --hook-stage manual --files tests/kernels/moe/test_deepgemm.py vllm/model_executor/layers/fused_moe/deep_gemm_utils.py vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py vllm/model_executor/layers/quantization/utils/fp8_utils.py
git diff --check

The pinned runtime was docker.io/vllm/vllm-openai@sha256:f29125bcc6d276ed38a67c9dfd4e51ccbaba09ad92847a0913c0df38d9c05c71, vLLM 0.29.1rc1.dev422+gd05da62e9; source changes were based on 953f90d25f. B200 and H200 were tested. SM103, SM120 fallback execution, and distributed DeepEP V2 were not tested. Synthetic expert latency is not end-to-end serving throughput. Both clusters' model and benchmark artifacts are preserved under local repros/block-fp8-padding-20260928/ and repros/block-fp8-sm90-20260929/; H200 artifacts are also on Waldorf PVC synthid-bench-results under /block-fp8-sm90-padding-20260929/.

AI assistance: Codex helped implement, validate, combine, and prepare this change. The human submitter must review every changed line and independently verify relevant tests before merge.