Repository navigation
config(musespark): tune GB300 SGLang AgentX settings per concurrency / config(musespark): 按并发调整 GB300 SGLang AgentX 参数 - #4342
Conversation
…/ config(musespark): 按并发调整 GB300 SGLang AgentX 参数 Replace the Muse Spark Open GB300 SGLang AgentX zip sweep (TP4/EP4 at concurrency 1-32) with 10 per-concurrency points: GPU-resident KV at concurrency 1, 5, 8, 10, 15 and DRAM HiCache at 16, 20, 24, 34, 40. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shyeh25
added
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39408192975 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39408192975 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because the recipe and master config have designated code owners and this replaces the whole sweep with hand-tuned points, a human look would still be worthwhile.
What was reviewed:
- Each conc-list entry in nvidia-master.yaml (1, 5, 8, 10, 15 and 16, 20, 24, 34, 40) has a matching
override_tp4_c{conc}block in agentic.yaml, and HiCache is enabled only on the DRAM-offload points. - The new perf-changelog entry is appended at the end and leaves existing bytes unchanged. Its
pr-linkstill ends inpull/XXXand needs the real number. Its text says write_through is used at concurrency 16-40, but c24 and c34 don't sethicache-write-policy. The file is append-only, so it's worth fixing that before merge. - c5 and c8 cap max-running-requests below CONC (2 and 3), while c24 and c34 set it above (48 and 96). This looks deliberate but isn't explained in the recipe.
Extended reasoning...
The change edits the single-node Muse Spark GB300 FP4 SGLang AgentX srt-slurm recipe (agentic.yaml), two conc-lists in nvidia-master.yaml, and appends a perf-changelog.yaml entry. It is benchmark configuration only and touches no security-sensitive surface. CODEOWNERS assigns both nvidia-master.yaml and the gb300 recipe tree to specific owners, which rules out auto-approval. The recipe change replaces a zipped sweep with ten hand-tuned override blocks of about 170 lines, so the per-point tuning choices are a judgment call for the owners. The changelog entry still has a placeholder pr-link.
…rrency 24 and 34 Both points already ran with write_through (the SGLang default); this states it in the recipe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shyeh25
added
the
full-sweep-fail-fast
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#45513 has been merged. This PR's AgentX config ( This PR has no authorized |
|
/use 39408192975 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
|
@nvpohanh staged run 39408192975. This run remains available across future |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — PR head ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#45513 is MERGED (2026-10-21T05:22:39Z). On sglang ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — the sign-off contains all 18 items of the current ✅ Check 6 (Upstream image / engine order): PASS — ✅ Check 7 (Deprecated models): PASS — on 2026-10-21, MODELS.md lists ✅ Check 8 (No architecture hacks): PASS — the diff has no ✅ Check 9 (Chat template): PASS — the DFlash spec-decode config is benchmarked with AgentX ✅ Check 10 (No engine patches): PASS — the diff is YAML-only. The recipe has no setup scripts, patches, ✅ Check 11 (Golden AL): PASS — ➖ Check 12 (Append-only): N/A — the new ✅ Check 13 (Draft precision): PASS — the draft is ✅ Check 14 (Pareto coverage): PASS — one curve: musespark / agentic-coding / gb300_sglang / fp4, from run 39408192975 (head ✅ Check 15 (PR description): PASS — the body's claims match the recipe at ✅ Check 16 (NVIDIA overview): PASS — the Framework Recipes table of the NVIDIA overview has a Muse-Spark-Open row with an SGLang checkmark linking to the SGLang cookbook page. ➖ Check 17 (Metrics publication): N/A — this is a direct SGLang deployment with no Dynamo TRT-LLM serving path. Assessed commit: |
Resolve conflicts: take main's refactored recipes and re-apply this PR's non-image edits; take main's perf-changelog.yaml and re-append this PR's entry. 中文：合并 origin/main；配方采用 main 重构后的版本并重新应用本 PR 的非镜像改动，perf-changelog.yaml 采用 main 版本并在末尾重新追加本 PR 的条目。
Summary
Replace the Muse Spark Open GB300 SGLang AgentX zip sweep (TP4/EP4 at concurrency 1–32) with 10 per-concurrency points, each with its own
override_tp4_c{conc}section:Per point, the recipe sets request admission and decode graph batch limits, memory fraction, prefill-decode interval and chunked prefill; concurrency 16–34 use shortest-prefill-first, concurrency 10, 15 and 40 run 4 tokenizer workers with incremental streaming output, and concurrency 34 and 40 set
SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS=4. The base raises the SWA/full ratio to 0.3, frees out-of-window SWA slots, and removes the explicitfa4draft attention backend so SGLang uses the image default (trtllm_mha). Concurrency 5, 10 and 15 keepfa4. Concurrency 40 setsspeculative-draft-attention-backend: trtllm_mhaandspeculative-draft-kv-cache-dtype: fp8_e4m3explicitly, which restates that default. As a result, the draft KV cache changes from BF16 (forced byfa4) to FP8fp8_e4m3at concurrency 1, 8, 16, 20, 24, 34 and 40, inheriting the target's KV cache dtype. It stays BF16 at concurrency 5, 10 and 15. The draft weights are unchanged: the image, target checkpoint and DFlash2 draft checkpoint (RadixArk/Muse-Spark-Open-DFlash2, served as shipped in BF16 with 8 draft tokens) are the same as before.AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
将 Muse Spark Open GB300 SGLang AgentX 的 zip sweep（TP4/EP4、并发 1–32）替换为 10 个按并发配置的点，每个点有独立的
override_tp4_c{conc}段：每个点分别设置请求准入与 decode graph 批大小上限、显存占用比例、prefill-decode interval 和 chunked prefill；并发 16–34 使用 shortest-prefill-first，并发 10、15、40 使用 4 个 tokenizer worker 及增量流式输出，并发 34 和 40 设置
SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS=4。基础配置将 SWA/full 比例提高到 0.3，释放窗口外的 SWA 槽位，并移除显式指定的fa4草稿注意力后端，改用镜像默认值（trtllm_mha）。并发 5、10、15 仍使用fa4；并发 40 显式设置speculative-draft-attention-backend: trtllm_mha和speculative-draft-kv-cache-dtype: fp8_e4m3，与该默认值一致。因此，并发 1、8、16、20、24、34、40 的草稿 KV cache 从 BF16（由fa4强制）变为 FP8fp8_e4m3，与目标模型的 KV cache dtype 一致；并发 5、10、15 仍为 BF16。草稿模型权重不变：镜像、目标模型检查点和 DFlash2 草稿检查点（RadixArk/Muse-Spark-Open-DFlash2，按发布的 BF16 精度提供，8 个草稿 token）均与之前相同。AI 模型使用说明：运行环境提供的模型标识为 claude-opus-5-5。工作内容包括执行调优（编排代理及其委派的任务代理均使用 claude-opus-5-5）、根据实测结果生成配方、主配置和性能变更记录的改动、在本地检查，以及起草本 PR 说明。
🤖 Generated with Claude Code