iframe-proxy

ahmed-bsod · 2026-09-25T02:29:44Z

Purpose

On ROCm's DeepSeek-V4.1's path each mHC seam runs today as a chain of separate aiter kernels, with the RMSNorm as one more standalone pass over the collapse. Aiter ships a fused kernel for the whole seam now ROCm/aiter#5824, a split-K main kernel (post-mix, bf16 residual write, gate projection, square sums) plus a per-token reduce (gates including the Sinkhorn loop, RMSNorm applied in place to the collapse): two launches per seam. This PR wires vLLM to it on gfx950.

Test Plan

image used: vllm/vllm-openai-rocm:nightly-rocm100

serve command:

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_USE_BREAKABLE_CUDAGRAPH=1
export VLLM_USE_RUST_FRONTEND=1
export AITER_TRITON_LOG_LEVEL=ERROR
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export OMP_NUM_THREADS=1
vllm serve /data/models/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --moe-backend aiter \
  --gpu-memory-utilization 0.80 \
  --language-model-only \
  --max-model-len 360448 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 16384 \
  --no-async-scheduling \
  --no-enable-prefix-caching \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'

bench command:

vllm bench serve \
    --model="$MODEL_PATH" \
    --backend=vllm \
    --tokenizer="$MODEL_PATH" \
    --dataset-name=random \
    --random-input-len=32768 \
    --random-output-len=32 \
    --random-range-ratio 0 \
    --num-prompts=64 \
    --max-concurrency=32 \
    --ignore-eos \
    --temperature=0 \
    --save-result --result-dir . \
    --metric-percentiles="50,90,99" \
    --percentile-metrics="ttft,tpot,itl,e2el"  2>&1 | tee -a "$LOG_FILE"

vllm bench serve \
    --model="$MODEL_PATH" \
    --backend=vllm \
    --tokenizer="$MODEL_PATH" \
    --dataset-name=random \
    --random-input-len=350000 \
    --random-output-len=350 \
    --random-range-ratio 0 \
    --num-prompts=128 \
    --max-concurrency=32 \
    --ignore-eos \
    --temperature=0 \
    --save-result --result-dir . \
    --metric-percentiles="50,90,99" \
    --percentile-metrics="ttft,tpot,itl,e2el"  2>&1 | tee -a "$LOG_FILE"

lm-eval command:

MODEL_PATH="${1:-/data/models/DeepSeek-V4.1-Flash}"
lm_eval --model local-chat-completions --apply_chat_template \
    --tasks /app/scripts/infx/gsm8k.yaml \
    --output_path eval_infx_out --log_samples \
    --model_args "model=${MODEL_PATH},base_url=http://0.0.0.0:8000/v1/chat/completions,api_key=EMPTY,eos_string=</s>,max_retries=5,num_concurrent=16,timeout=1800,tokenized_requests=False,max_length=32768" \
    --gen_kwargs "max_tokens=8192,temperature=0,top_p=1,thinking=false"

Test Result

For ISL 32k/OSL 32/CONC 32 throughput improves by about 4.7%

For ISL 350k/OSL 350/CONC 32 throughput improves by about 2.94%

lmeval seems to be within range

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

github-actions · 2026-09-25T02:30:13Z

Fangzhou-Ai · 2026-09-28T19:59:31Z

we need an UT for this new kernel.

mergify · 2026-09-28T23:09:33Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ahmed-bsod.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

github-actions · 2026-09-28T23:26:38Z

✅ @ahmed-bsod, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

mergify · 2026-09-28T23:34:01Z

Hi @ahmed-bsod, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

ahmed-bsod · 2026-09-29T00:09:47Z

/ci run

github-actions · 2026-09-29T00:10:07Z

❌ This PR is 5 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

…on kernel Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

ahmed-bsod · 2026-09-29T00:11:03Z

/ci run

github-actions · 2026-09-29T00:11:23Z

shen-shanshan

Reviewed by @Fangzhou-Ai

vllm/vllm-openai-rocm:nightly-rocm100-36768d1bfd39094681cdbc8cb37d4b31c0729c89 is the first published ROCm nightly build whose underlying vLLM commit (36768d1bfd39094681cdbc8cb37d4b31c0729c89) includes all three PRs this recipe was waiting on: vllm-project/vllm#58655, #53492 and #58208. Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…Triton kernel (vllm-project#58655) Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

mergify Bot added deepseek Related to DeepSeek models DSv4.1 Related to DeepSeek-V4.1 models labels Sep 25, 2026

ahmed-bsod changed the title ~~fuse mhc~~ [ROCm][DSv4.1][Perf] Run the delayed mHC seams through aiter's fused Triton kernel Sep 28, 2026

mergify Bot added the rocm Related to AMD ROCm label Sep 28, 2026

github-project-automation Bot added this to AMD Sep 28, 2026

github-project-automation Bot moved this to Todo in AMD Sep 28, 2026

ahmed-bsod force-pushed the ahmed/fused-mhc branch 2 times, most recently from e0d9ce3 to c2acf44 Compare September 28, 2026 16:28

mergify Bot added the needs-rebase label Sep 28, 2026

ahmed-bsod force-pushed the ahmed/fused-mhc branch from 84d009a to e97404b Compare September 28, 2026 23:20

ahmed-bsod marked this pull request as ready for review September 28, 2026 23:20

ahmed-bsod requested review from AndreasKaratzas, WoosukKwon, dllehr-amd, hongxiayang, mgoin, tjtanaa, tlrmchlsmth, yewentao256 and zyongye as code owners September 28, 2026 23:20

claude Bot reviewed Sep 28, 2026

View reviewed changes

Fangzhou-Ai added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 28, 2026

mergify Bot removed the needs-rebase label Sep 28, 2026

Fangzhou-Ai mentioned this pull request Sep 28, 2026

[AMD][MI355X] DSv4.1-Flash vLLM: enable sparse-MLA Gluon kernel / DSv4.1-Flash vLLM：启用 sparse-MLA Gluon kernel SemiAnalysisAI/InferenceX#3555

Closed

3 of 10 tasks

ahmed-bsod added 3 commits September 28, 2026 20:10

Replace existing unfused mhc path with aiter's fused mhc+rmsnorm trit…

13007b8

…on kernel Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

Add unit test

290947d

Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

Apply ruff format

980c3bf

Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

ahmed-bsod force-pushed the ahmed/fused-mhc branch from f96efd8 to 980c3bf Compare September 29, 2026 00:10

Fangzhou-Ai approved these changes Sep 29, 2026

View reviewed changes

shen-shanshan approved these changes Sep 29, 2026

View reviewed changes

shen-shanshan merged commit 0af3441 into vllm-project:main Sep 29, 2026
208 checks passed

cpersson-amd mentioned this pull request Oct 2, 2026

[ROCm][DSv4.1][Perf] Overlap the mHC gate projection with attention and FFN #59802

Open

6 tasks done

michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026

[ROCm][DSv4.1][Perf] Run the delayed mHC seams through aiter's fused …

9616d23

…Triton kernel (vllm-project#58655) Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>

zkawfanx mentioned this pull request Oct 22, 2026

[Start here] Classify the pinned DSv4.1 timeline and prepare chronological backport batches zkawfanx/vllm#18

Open

3 of 5 tasks

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

[ROCm][DSv4.1][Perf] Run the delayed mHC seams through aiter's fused Triton kernel - #58655

[ROCm][DSv4.1][Perf] Run the delayed mHC seams through aiter's fused Triton kernel#58655
shen-shanshan merged 3 commits into
vllm-project:mainfrom
ahmed-bsod:ahmed/fused-mhc

ahmed-bsod commented Sep 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

Fangzhou-Ai commented Sep 28, 2026

Uh oh!

mergify Bot commented Sep 28, 2026

Uh oh!

claude Bot left a comment

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

mergify Bot commented Sep 28, 2026

Uh oh!

ahmed-bsod commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

ahmed-bsod commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

shen-shanshan left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

ahmed-bsod commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

Fangzhou-Ai commented Sep 28, 2026

Uh oh!

mergify Bot commented Sep 28, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

mergify Bot commented Sep 28, 2026

Uh oh!

ahmed-bsod commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

ahmed-bsod commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

shen-shanshan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ahmed-bsod commented Sep 25, 2026 •

edited

Loading