[XPU][Kernel] Tune Triton W8A8 block-FP8 GEMM for Intel B70 by pmanczak · Pull Request #56063 · vllm-project/vllm · GitHub
Skip to content

[XPU][Kernel] Tune Triton W8A8 block-FP8 GEMM for Intel B70 - #56063

Merged
hmellor merged 1 commit into
vllm-project:mainfrom
pmanczak:pmanczak/xpu-w8a8-block-fp8-tuning
Oct 2, 2026
Merged

hmellor merged 1 commit into
vllm-project:mainfrom
pmanczak:pmanczak/xpu-w8a8-block-fp8-tuning

Conversation

@pmanczak

@pmanczak pmanczak commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Follows the pattern established in #53065 (XPU Triton fused MoE tuning): port
benchmarks/kernels/benchmark_w8a8_block_fp8.py to Intel XPU and ship tuned
Triton kernel configs for the W8A8 blockwise FP8 GEMM (block_shape=[128,128])
on Intel(R) Arc(TM) Pro B70 Graphics.

Changes

  • benchmarks/kernels/benchmark_w8a8_block_fp8.py: XPU support (platform
    assert, device_type, guard torch.cuda.init(), and a curated
    get_xpu_tuning_space() — 96 configs vs. 1280, since per-config Triton
    JIT compile time on B70 is ~1-8s vs. sub-100ms on CUDA; only used when
    current_platform.is_xpu())
  • 26 new tuned config JSONs under
    vllm/model_executor/layers/quantization/utils/configs/ for
    device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,dtype=fp8_w8a8,block_shape=[128,128]
    (comparable to MI300X's 28 and H20/H100/H200's 23-26 for this kernel),
    covering dense-linear shapes from DeepSeek-V3, DeepSeek-V2-Lite
    (TP=1), Llama-3.1-8B (TP=1, packed QKV/gate-up), and Qwen3-8B's two
    shapes not already covered by Llama's

Results

Intel Arc Pro B70, fp8. Baseline is the kernel's built-in default config
(BLOCK_SIZE_M=64, GROUP_SIZE_M=32, num_warps=4, num_stages=2, used when no
tuned JSON is found); tuned uses the generated JSON. Per-batch sweeps (10
sizes, matching the tuned grid) for two DeepSeek-V2-Lite (tp=1) shapes;
geometric-mean speedups are 2.73x (N=576,K=2048, kv_a_proj_with_mqa /
kv_b_proj) and 2.00x (N=22016,K=2048, gate_up_proj):

batch N=576,K=2048 N=22016,K=2048
1 5.08x 2.67x
4 2.62x 1.93x
16 5.17x 3.28x
32 3.21x 2.99x
64 3.21x 2.22x
128 4.18x 1.50x
512 2.89x 1.69x
1024 1.31x 1.25x
2048 1.02x 1.69x
4096 2.02x 1.69x

Numerical validation: every tuned config was checked against an FP32
reference GEMM using deterministic FP8 block-quantized inputs (relative L2
error < 2e-2). DeepSeek-V2-Lite passed 60/60 batch-size x shape
combinations; Llama-3.1-8B passed 60/60; Qwen3-8B's 2 newly-tuned shapes
passed 40/40 (its other 2 shapes reuse Llama-3.1-8B's already-validated
configs, see below).

End-to-end

Served yiliu4/DeepSeek-V2-Lite-Chat-Block-wise-FP8-Test-Only (block-FP8
checkpoint) on a single B70 card. Both rows use explicit
--linear-backend triton, comparing Triton's built-in per-shape heuristic
(baseline) with the tuned shape-specific JSONs. Workload: 512 input tokens,
128 output tokens, 32 prompts, inf request rate; 1 warm-up run discarded, 3
measured runs per arm (seeds 2-4):

model shape TP baseline tuned speedup
DeepSeek-V2-Lite-Chat (block-FP8) all dense linears (7 shapes) 1 109.63 tok/s 140.88 tok/s 1.285x
Llama-3.1-8B-Instruct (block-FP8) all runtime linears (4 shapes) 1 36.99 tok/s 84.20 tok/s 2.276x
Hunyuan-MT-7B (block-FP8) all runtime linears (4 shapes) 1 37.04 tok/s 84.55 tok/s 2.282x
Qwen3-8B (block-FP8) all runtime linears (4 shapes, 2 newly tuned) 1 36.29 tok/s 82.89 tok/s 2.284x

For reference, the default (oneDNN, auto) path is substantially faster
than tuned Triton on all four models (roughly 2-5.7x), confirming the
"Note on scope" above: this PR's value is the tuning pipeline plus configs
for whenever Triton is the selected/needed path, not a claim that Triton
should replace oneDNN on B70.

Test plan

  • Tune on XPU with
    benchmark_w8a8_block_fp8.py tune --tp-size <tp> --batch-size <bs>.
  • Verify every generated winner against an fp32 reference
    (relative L2 error < 2e-2).
  • Confirm serving with --linear-backend triton loads the exact JSON
    (fp8_utils.py logs Using configuration from ... for W8A8 Block FP8 kernel.).
  • Compare the kernel's default config and tuned config with identical
    inputs at the tuned batch-size grid.
  • Compare heuristic and tuned Triton E2E with identical workloads and fresh
    engines.
  • Keep CUDA/ROCm behavior unchanged; XPU pruning is guarded by
    current_platform.is_xpu().

@mergify mergify Bot added performance Performance-related issues quantization intel-gpu Related to Intel GPU labels Sep 9, 2026
@pmanczak
pmanczak force-pushed the pmanczak/xpu-w8a8-block-fp8-tuning branch 3 times, most recently from a63d038 to e88e49c Compare September 10, 2026 06:22
@pmanczak
pmanczak marked this pull request as ready for review September 10, 2026 06:34

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@pmanczak
pmanczak force-pushed the pmanczak/xpu-w8a8-block-fp8-tuning branch from e88e49c to 5516c92 Compare September 10, 2026 06:37
Comment thread benchmarks/kernels/benchmark_w8a8_block_fp8.py
@yma11

yma11 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

@jikunshang I have an open about the need of triton config tuning on BMG, please comment.

@pmanczak

pmanczak commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

❌ This PR is 1272 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

Port the W8A8 block-FP8 tuning benchmark to XPU and add 26 tuned
configurations for Intel Arc Pro B70. The configs cover DeepSeek-V3,
DeepSeek-V2-Lite, Llama-3.1-8B, and Qwen3-8B linear shapes.

All generated configurations were numerically validated against an FP32
reference before inclusion.

Signed-off-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com>
@pmanczak
pmanczak force-pushed the pmanczak/xpu-w8a8-block-fp8-tuning branch from 7bccde9 to 4dff424 Compare October 1, 2026 10:59
@pmanczak

pmanczak commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #92394 for commit 4dff42446ce6.

@pmanczak

pmanczak commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

✅ Queued 99 failed job(s) for retry in Buildkite CI #92394.

@pmanczak

pmanczak commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

✅ Queued 99 failed job(s) for retry in Buildkite CI #92394.

@pmanczak

pmanczak commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

✅ Queued 99 failed job(s) for retry in Buildkite CI #92394.

@pmanczak

pmanczak commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

@hmellor
hmellor merged commit bdfb026 into vllm-project:main Oct 2, 2026
198 of 199 checks passed
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
…ject#56063)

Signed-off-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com>
Co-authored-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

intel-gpu Related to Intel GPU performance Performance-related issues quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants