{{ message }}
Repository navigation
[XPU][Kernel] Tune Triton W8A8 block-FP8 GEMM for Intel B70 - #56063
Merged
hmellor merged 1 commit intoOct 2, 2026
Merged
Conversation
pmanczak
force-pushed
the
pmanczak/xpu-w8a8-block-fp8-tuning
branch
3 times, most recently
from
September 10, 2026 06:22
a63d038 to
e88e49c
Compare
pmanczak
marked this pull request as ready for review
September 10, 2026 06:34
pmanczak
requested review from
mgoin,
pavanimajety,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
zyongye
as code owners
September 10, 2026 06:34
pmanczak
force-pushed
the
pmanczak/xpu-w8a8-block-fp8-tuning
branch
from
September 10, 2026 06:37
e88e49c to
5516c92
Compare
yma11
reviewed
Sep 11, 2026
Contributor
|
@jikunshang I have an open about the need of triton config tuning on BMG, please comment. |
pmanczak
force-pushed
the
pmanczak/xpu-w8a8-block-fp8-tuning
branch
from
September 15, 2026 07:56
5516c92 to
fe35dbe
Compare
1 task done
jikunshang
approved these changes
Sep 18, 2026
pmanczak
force-pushed
the
pmanczak/xpu-w8a8-block-fp8-tuning
branch
from
October 1, 2026 09:49
fe35dbe to
7bccde9
Compare
Contributor
Author
|
/ci run |
|
❌ This PR is 1272 commits behind upstream |
Port the W8A8 block-FP8 tuning benchmark to XPU and add 26 tuned configurations for Intel Arc Pro B70. The configs cover DeepSeek-V3, DeepSeek-V2-Lite, Llama-3.1-8B, and Qwen3-8B linear shapes. All generated configurations were numerically validated against an FP32 reference before inclusion. Signed-off-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com>
pmanczak
force-pushed
the
pmanczak/xpu-w8a8-block-fp8-tuning
branch
from
October 1, 2026 10:59
7bccde9 to
4dff424
Compare
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #92394 for commit |
Contributor
Author
|
/ci retry |
|
✅ Queued 99 failed job(s) for retry in Buildkite CI #92394. |
Contributor
Author
|
/ci retry |
|
✅ Queued 99 failed job(s) for retry in Buildkite CI #92394. |
Contributor
Author
|
/ci retry |
|
✅ Queued 99 failed job(s) for retry in Buildkite CI #92394. |
Contributor
Author
|
/ci retry |
michael-gpu
pushed a commit
to practical-inference/vllm-ampere-extended
that referenced
this pull request
Oct 20, 2026
…ject#56063) Signed-off-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com> Co-authored-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Purpose
Follows the pattern established in #53065 (XPU Triton fused MoE tuning): port
benchmarks/kernels/benchmark_w8a8_block_fp8.pyto Intel XPU and ship tunedTriton kernel configs for the W8A8 blockwise FP8 GEMM (
block_shape=[128,128])on Intel(R) Arc(TM) Pro B70 Graphics.
Changes
benchmarks/kernels/benchmark_w8a8_block_fp8.py: XPU support (platformassert,
device_type, guardtorch.cuda.init(), and a curatedget_xpu_tuning_space()— 96 configs vs. 1280, since per-config TritonJIT compile time on B70 is ~1-8s vs. sub-100ms on CUDA; only used when
current_platform.is_xpu())vllm/model_executor/layers/quantization/utils/configs/fordevice_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,dtype=fp8_w8a8,block_shape=[128,128](comparable to MI300X's 28 and H20/H100/H200's 23-26 for this kernel),
covering dense-linear shapes from DeepSeek-V3, DeepSeek-V2-Lite
(TP=1), Llama-3.1-8B (TP=1, packed QKV/gate-up), and Qwen3-8B's two
shapes not already covered by Llama's
Results
Intel Arc Pro B70, fp8. Baseline is the kernel's built-in default config
(
BLOCK_SIZE_M=64, GROUP_SIZE_M=32, num_warps=4, num_stages=2, used when notuned JSON is found); tuned uses the generated JSON. Per-batch sweeps (10
sizes, matching the tuned grid) for two DeepSeek-V2-Lite (tp=1) shapes;
geometric-mean speedups are 2.73x (
N=576,K=2048, kv_a_proj_with_mqa /kv_b_proj) and 2.00x (
N=22016,K=2048, gate_up_proj):Numerical validation: every tuned config was checked against an FP32
reference GEMM using deterministic FP8 block-quantized inputs (relative L2
error < 2e-2). DeepSeek-V2-Lite passed 60/60 batch-size x shape
combinations; Llama-3.1-8B passed 60/60; Qwen3-8B's 2 newly-tuned shapes
passed 40/40 (its other 2 shapes reuse Llama-3.1-8B's already-validated
configs, see below).
End-to-end
Served
yiliu4/DeepSeek-V2-Lite-Chat-Block-wise-FP8-Test-Only(block-FP8checkpoint) on a single B70 card. Both rows use explicit
--linear-backend triton, comparing Triton's built-in per-shape heuristic(baseline) with the tuned shape-specific JSONs. Workload: 512 input tokens,
128 output tokens, 32 prompts, inf request rate; 1 warm-up run discarded, 3
measured runs per arm (seeds 2-4):
DeepSeek-V2-Lite-Chat(block-FP8)Llama-3.1-8B-Instruct(block-FP8)Hunyuan-MT-7B(block-FP8)Qwen3-8B(block-FP8)For reference, the default (oneDNN,
auto) path is substantially fasterthan tuned Triton on all four models (roughly 2-5.7x), confirming the
"Note on scope" above: this PR's value is the tuning pipeline plus configs
for whenever Triton is the selected/needed path, not a claim that Triton
should replace oneDNN on B70.
Test plan
benchmark_w8a8_block_fp8.py tune --tp-size <tp> --batch-size <bs>.(relative L2 error < 2e-2).
--linear-backend tritonloads the exact JSON(
fp8_utils.pylogsUsing configuration from ... for W8A8 Block FP8 kernel.).inputs at the tuned batch-size grid.
engines.
current_platform.is_xpu().