Repository navigation
[NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep with a GPU-only KV cache / [NVIDIA][AgentX] Kimi-K3 B300 TP8 仅使用 GPU KV 缓存的扫描 - #4361
Oseltamivir wants to merge 2 commits into
Conversation
No-offload control for the 3 TB and 1.5 TB SimpleCPUOffloadConnector runs: same points and prefix-cache granularity, no host DRAM tier. Drops the Mooncake RDMA store and its setup script. 3 TB 和 1.5 TB SimpleCPUOffloadConnector 运行的无卸载对照：相同的点和前缀缓存 粒度，不使用主机 DRAM 层。移除 Mooncake RDMA 存储及其安装脚本。
将 perf-changelog 条目链接到 #4361。
Oseltamivir
added
the
full-sweep-fail-fast
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Run TP8 x DCP8 with a GPU-only KV cache: drop the Mooncake RDMA store and its kimik3-b300-mooncake.sh setup script, with no host DRAM offload tier." | ||
| - "Drop prefix-match-unit 128 so prefix-cache granularity matches the SimpleCPUOffloadConnector DRAM variants." | ||
| - "Sweep concurrency 1, 4, 16, 24, 32, 40, 48, 56 and 64: drop conc 2, 8 and 70 and add conc 64 (max-num-seqs 128)." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD |
There was a problem hiding this comment.
🔴 The changelog check will fail on this PR, and the merge-prep step that fills in the PR number will reject it. The new entry's pr-link ends in pull/TBD, but validate_added_pr_link only accepts the real PR URL or an XXX placeholder (PR_LINK_PLACEHOLDERS in infx/workflows/validate_perf_changelog.py:22). Fix: set pr-link to https://github.com/SemiAnalysisAI/InferenceX/pull/XXX (or the real PR number). The merge tooling then rewrites the placeholder to the final link.
Why this was flagged
On a PR run, validate_added_pr_link (infx/workflows/validate_perf_changelog.py:139-150) checks every appended entry. It raises new PR entry must use 'https://github.com/SemiAnalysisAI/InferenceX/pull/<n>' or an XXX placeholder; found '.../pull/TBD' because PR_LINK_PLACEHOLDERS (line 22) holds only XXX and https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. On a main-branch run, CANONICAL_PR_LINK (pull/\d+) also rejects it. prepare_perf_changelog_merge.py:105-110 raises appended entry ... has unexpected pr-link for any link other than a placeholder or the expected URL. So the changelog gate fails and the benchmark for kimik3-fp4-b300-vllm-agentic-dspark is never triggered. No other entry in the file uses TBD.
Verification: The diff appends an entry whose last line is pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD (perf-changelog.yaml:10504). In infx/workflows/validate_perf_changelog.py, PR_LINK_PLACEHOLDERS (lines 22-25) contains only "XXX" and ".../pull/XXX". Result: the changelog check fails and the merge tooling cannot finalize the entry until pr-link is changed to .../pull/XXX or the real number.
| # No prefix-match-unit: keeps prefix-cache granularity equal to the DRAM | ||
| # offload variants, which cannot use sub-block KDA hits. |
There was a problem hiding this comment.
🟡 nit (optional): Future readers of this recipe get a comment about "the DRAM offload variants", which do not exist anywhere in this tree. It explains a removed line by pointing at sibling PRs (#4348/#4349), which is history and cross-PR context. AGENTS.md:65 forbids that kind of comment. Fix: delete the comment, or reduce it to a self-contained technical reason for leaving prefix-match-unit unset. The comparison with the DRAM runs already lives in the perf-changelog.yaml entry.
Why this was flagged
The diff adds a comment at agentic.yaml:41-42 about an absent prefix-match-unit key. Its justification is parity with "the DRAM offload variants". After this change no DRAM variant of kimik3-fp4-b300-vllm-agentic-dspark exists in the repo: nvidia-master.yaml now has only kv-offloading: none. The variants it refers to live in the unmerged sibling PRs #4348 and #4349. AGENTS.md:65 ('No history... what it replaced') and AGENTS.md:66 ('Only technical explanation') rule out comments like this. On the base branch the recipe set prefix-match-unit: 128 with no such comment. Once the sibling PRs merge or close, a maintainer has no in-tree way to find out what the comment means.
Verification: The diff adds this at agentic.yaml:41-42 about "the DRAM offload variants". No DRAM offload variant of this benchmark exists in the tree after the change; the recipe sets KV_OFFLOADING: none and nvidia-master.yaml:1518 has a single arm with kv-offloading: none. AGENTS.md "Code comments" forbids history and "what it replaced". Nothing breaks at runtime; the only harm is a misleading comment.
This branch has not been deployed
1 participant
No-offload control for #4348 (3 TB) and #4349 (1.5 TB). Those sweeps served almost no prompt tokens from the host DRAM tier: ≤0.6% extra prefill avoided at conc ≤ 48 and ~8–13% at conc 56–64, while eager write-through copies every computed block to DRAM. This sweep runs the same 9 points with no offload, so the three runs show what the DRAM tier actually buys.
kv-offloading: none: nokv-transfer-config, nodram-utilization. Drops the Mooncake RDMA store and the now-unusedkimik3-b300-mooncake.shsetup script.prefix-match-unitstays off, matching [NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep conc 1-64 on the full 3 TB DRAM tier / [NVIDIA][AgentX] Kimi-K3 B300 TP8 在完整 3 TB DRAM 层上扫描 conc 1-64 #4348/[NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep conc 1-64 on a 1.5 TB DRAM tier / [NVIDIA][AgentX] Kimi-K3 B300 TP8 在 1.5 TB DRAM 层上扫描 conc 1-64 #4349's prefix-cache granularity so the only difference is the DRAM tier.[1, 4, 16, 24, 32, 40, 48, 56, 64](conc 70 → 64 withmax-num-seqs128), throughput only (no-evals: true).#4348, #4349 and this PR all edit the same config entry; only one of them can merge as is.
中文
#4348（3 TB）和 #4349（1.5 TB）的无卸载对照。那两次扫描几乎没有从主机 DRAM 层提供提示 token：conc ≤ 48 时避免的额外 prefill ≤0.6%，conc 56–64 时约 8–13%，而 eager 直写模式会把每个计算完成的块都复制到 DRAM。本次扫描在不卸载的情况下运行相同的 9 个点，三组结果可以说明 DRAM 层的实际收益。
kv-offloading: none：不使用kv-transfer-config和dram-utilization。移除 Mooncake RDMA 存储及不再使用的kimik3-b300-mooncake.sh安装脚本。prefix-match-unit，与 [NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep conc 1-64 on the full 3 TB DRAM tier / [NVIDIA][AgentX] Kimi-K3 B300 TP8 在完整 3 TB DRAM 层上扫描 conc 1-64 #4348/[NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep conc 1-64 on a 1.5 TB DRAM tier / [NVIDIA][AgentX] Kimi-K3 B300 TP8 在 1.5 TB DRAM 层上扫描 conc 1-64 #4349 的前缀缓存粒度一致，使唯一差异为 DRAM 层。[1, 4, 16, 24, 32, 40, 48, 56, 64]（conc 70 → 64，max-num-seqs128），仅测吞吐（no-evals: true）。#4348、#4349 和本 PR 修改同一配置条目，按现状只能合并其中一个。
AI model disclosure
🤖 Generated with Claude Code