Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

Initial attempt · Queued · Run 39383016006 / attempt 1 · 2026-10-20 06:54 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Wait for the smoke run covering the minimum concurrency of each of the four topologies and the four gsm8k evals.

Initial attempt · Queued · Run 39383016006 / attempt 2 · 2026-10-20 07:13 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Attempt 1 was cancelled while queued by the scheduler (preemption, no repair used), so the cancelled jobs were rerun as attempt 2 on the same run and head; wait for the smoke results.

Initial attempt · Queued · Run 39383016006 / attempt 3 · 2026-10-20 07:15 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Attempts 1 and 2 were cancelled while queued by the scheduler (preemption, no repair used), so the cancelled jobs were rerun as attempt 3 on the same run and head; wait for the smoke results.

Initial attempt · Queued · Run 39383016006 / attempt 4 · 2026-10-20 11:10 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Run attempts 1 to 3 were cancelled while queued by the scheduler (preemption, no repair used); attempt 4 has been queued at background priority since 07:21 UTC with no job started, so keep waiting for the smoke results.

Initial attempt · Running · Run 39383016006 / attempt 5 · 2026-10-20 12:44 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Attempt 5 c120 and c960 smoke benchmarks started on gb300-nv after earlier scheduler preemptions (no repair used); wait for all four benchmarks and four gsm8k evals.

resumed · The session ended before finishing; session 2 continues it in a new run of its planner.

Initial attempt · Running · Run 39383016006 / attempt 6 · 2026-10-20 13:46 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Attempt 6 c960 benchmark and c480 gsm8k eval have run since about 13:20 UTC with the other smoke jobs queued; wait for all four benchmarks and four evals.

Initial attempt · Queued · Run 39383016006 / attempt 7 · 2026-10-20 14:04 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Next: Attempt 6 c480 gsm8k eval passed; the scheduler preempted the rest (no repair used), including c960 in warmup with 0 request errors at 14:00 UTC, and queued attempt 7, so wait for it.

Initial attempt · Running · Run 39383016006 / attempt 8 · 2026-10-21 01:00 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb · Mean latency
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Note: All rows: Δ N/A: dataset/scenario/statistic mismatch.
Note: AgentX c960 PTP8x2/DTP16x1 0fa29f: 77 request errors.
Note: AgentX c2400 PTP8x4/DTP16x1 93f978: 21 request errors.

Note: gsm8k/em_strict · c2400: failed.
Note: gsm8k/em_strict · c2400: Δ N/A: eval score unavailable.

Next: The c2400 gsm8k eval server never started because decode TP5 on one node hung silently while loading the DSpark draft weights until the 8-minute post-load barrier, although the identical 4P1D setup loaded in the c2400 benchmark job; treat it as a transient node stall and rerun the failed eval once the running c1440 benchmark finishes.

resumed · The session ended before finishing; session 3 continues it in a new run of its planner.

resumed · The session ended before finishing; session 4 continues it in a new run of its planner.

Infrastructure retry 1 · Queued · Run 39383016006 / attempt 9 · 2026-10-21 01:56 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb · Mean latency
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Note: All rows: Δ N/A: dataset/scenario/statistic mismatch.
Note: AgentX c960 PTP8x2/DTP16x1 0fa29f: 77 request errors.
Note: AgentX c2400 PTP8x4/DTP16x1 93f978: 21 request errors.

Next: Attempt 8's c2400 gsm8k eval failed when decode TP5 hung silently loading DSpark draft weights until the post-load barrier, a node stall since the identical 4P1D setup loaded in the c2400 benchmark job, and the c1440 benchmark was preempted by the scheduler at 01:51 UTC; both were rerun as attempt 9 on the same run and head.

Infrastructure retry 1 · Queued · Run 39383016006 / attempt 10 · 2026-10-21 03:06 UTC
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae · 12a46f9d18cb · Mean latency
Change: Pin the v0.5.22-cu130 release (commit baa9befa, descends from c9a8fba9; all 37 recipe SGLang args remain) and move the recipe Dynamo wheel 1.5.0.dev20260914 to 1.6.0.dev20261014, since sgl-project/sglang#41818 resolves --dp-size N --enable-dp-attention to attn_dp_size=N, dp_size=1, which only Dynamo 1.6 sglang_dp_layout reads (else the router sees one DP rank); drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, undefined in v0.5.22 and inert under the default unified cache; deps: torch 2.14.1, flashinfer 0.7.1, sglang-kernel 0.4.9, deep-gemm 0.2.2, Mooncake 0.3.13.post1; router/Mooncake metadata labels stay as they key frozen points.

Note: All rows: Δ N/A: dataset/scenario/statistic mismatch.
Note: AgentX c960 PTP8x2/DTP16x1 0fa29f: 77 request errors.
Note: AgentX c2400 PTP8x4/DTP16x1 93f978: 21 request errors.

Next: Attempt 10 (rerun of the c2400 gsm8k eval after a node stall and of the scheduler-preempted c1440 benchmark) has been queued at background priority since 01:59 UTC with no job started; keep waiting.

unexpected-error · Cleanup pending. Stop and confirm owned runs before closing.

unexpected-error · Repairs: unknown · Runs: 39383016006
All owned runs ended. PR closed; branch deleted for retry.