Repository navigation
[NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep conc 1-64 on the full 3 TB DRAM tier / [NVIDIA][AgentX] Kimi-K3 B300 TP8 在完整 3 TB DRAM 层上扫描 conc 1-64 - #4348
Oseltamivir wants to merge 9 commits into
Conversation
Raise dram-utilization to 1.0 so every point's Mooncake segment takes the 3 TB AgentX ceiling (2999 GB, 374GB per rank), and replace conc 70 with 64. 将 dram-utilization 提高到 1.0，使每个点的 Mooncake 段使用完整的 3 TB AgentX 上限（2999 GB，每个 rank 374GB），并将 conc 70 替换为 64。
将 perf-changelog 条目链接到 #4348。
Oseltamivir
added
the
full-sweep-fail-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Besides the inline findings, I checked that the conc 70 to 64 rename is consistent. max-num-seqs 128 is 2x CONC 64, as the recipe's admission comment says. The unchanged graph capture list still covers 1..128. No references to c70 or CONC 70 are left in the kimik3 recipes.
Extended reasoning...
This note records checks made beyond the inline findings. They covered the conc-list change in nvidia-master.yaml and the override_c64 variant in agentic.yaml, and found that the two files agree.
| mode: embedded | ||
| metadata_server: P2PHANDSHAKE | ||
| global_segment_size: 281GB | ||
| global_segment_size: 374GB |
There was a problem hiding this comment.
🔴 If Mooncake reads GB as GiB, operators get about 3.21 TB of pinned host memory per node. That is above the 3 TB AgentX ceiling and the recorded 2999 GB. global_segment_size: 374GB comes from 2999 / 8 in decimal GB. Mooncake's size parser treats GB as 1024**3, so 8 ranks pin 2992 GiB, more than the 2794 GiB cap (MAX_AGENTIC_AVAILABLE_CPU_DRAM_MIB = 2861022). Fix: give the per-rank segment in Mooncake's own units so 8 × segment stays within TOTAL_CPU_DRAM_GB (e.g. 349GB, about 2999e9/8/2^30). Otherwise, state the binary unit in the recipe comment.
Why this was flagged
This depends on the installed mooncake-transfer-engine-cuda13 0.3.11.post1 parsing global_segment_size with binary multipliers (gb -> 1024**3). The repo's vllm-mooncake.sh only pip-installs Mooncake and does no conversion. At agentic.yaml:35 every TP8 point registers 8 × 374 GiB = 2992 GiB ≈ 3213 GB of RDMA-pinned DRAM, plus 8 × 4GB local_buffer_size. generate.py:428-439 clamps the budget to MAX_AGENTIC_AVAILABLE_CPU_DRAM_MIB = 2861022 MiB (2794 GiB) and records allocated_cpu_dram_gb 2999. So the recorded number understates the real tier by about 214 GB, and the run goes past the AgentX ceiling the config comment claims. On a ~4.17 TB DSXE node this leaves about 0.95 TB, not the ~1.2 TB the PR's risk note assumes, for the ~1.5 TB checkpoint's page cache and the vLLM processes. That raises the host-OOM risk at every conc point. On the base branch, 281 GiB × 8 = 2248 GiB (~2414 GB) was already larger than the recorded 2249, which is pre-existing, but it stayed under the 2794 GiB cap. single_node.py:138 only compares the TOTAL_CPU_DRAM_GB strings, so nothing checks the segment size.
Verification: agentic.yaml:35 changes to global_segment_size: 374GB, which is 2999/8 worked out in decimal GB. Under that reading, 8 × 374 GiB = 2992 GiB ≈ 3213 GB is pinned and RDMA-registered. That is above the 2794 GiB (3 TB) AgentX ceiling. The base value 281GB stayed under the 3 TB ceiling, so going over the ceiling is new in this change.
| description: | ||
| - "Raise dram-utilization from 0.75 to 1.0 so the TP8 Mooncake DRAM tier takes the full 3 TB AgentX ceiling (total-cpu-dram-gb 2249 -> 2999, global_segment_size 281GB -> 374GB per rank) at every point." | ||
| - "Sweep concurrency 1 to 64: replace the conc 70 point with conc 64 (max-num-seqs 128)." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD |
There was a problem hiding this comment.
🔴 The perf-changelog check on this PR will fail, because the new entry's pr-link placeholder pull/TBD is not one the validator accepts. validate_added_pr_link only allows the PR's own URL or an XXX placeholder (PR_LINK_PLACEHOLDERS). Fix: set the pr-link to https://github.com/SemiAnalysisAI/InferenceX/pull/<this PR number> or to the https://github.com/SemiAnalysisAI/InferenceX/pull/XXX placeholder.
Why this was flagged
The new block at inferencex-e2e/perf-changelog.yaml:10332 ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD. On PR runs, compare_entries passes every appended entry to validate_added_pr_link (inferencex-e2e/infx/workflows/validate_perf_changelog.py:192-193). That function raises "new PR entry must use ... or an XXX placeholder" whenever the link is neither in PR_LINK_PLACEHOLDERS (lines 22-25: only XXX and .../pull/XXX) nor the PR's own URL (lines 146-150). The non-PR branch also fails, because CANONICAL_PR_LINK (line 21) requires pull/\d+. The base branch has no such entry, so its changelog validation passes. CONTRIBUTING.md:206 also documents pull/XXX as the placeholder.
Verification: Trigger: the PR is merged through the supported merge path (infx.workflows.merge_with_reuse). The new entry at inferencex-e2e/perf-changelog.yaml:10332 has pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD. validate_added_pr_link (validate_perf_changelog.py:146-150) rejects it at merge_with_reuse.py:631. The PR-time check never calls compare_entries.
Mooncake parses GB as GiB, so 374GB pinned 3.21 TB and RDMA registration failed with ENOMEM on every rank; 349GB is 2998 GB. Mark the entry no-evals. Mooncake 将 GB 解析为 GiB，374GB 实际锁定 3.21 TB，导致每个 rank 的 RDMA 注册 因 ENOMEM 失败；349GB 约为 2998 GB。将该条目标记为 no-evals。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39475571624 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39475571624 |
从 Kimi-K3 B300 扫描中移除 conc 2 和 8。
Single-node offload does not need Mooncake's RDMA store, whose segment registration on the DSXE EFA rail failed with ENOMEM on every rank. Drop the now-unused kimik3-b300-mooncake.sh setup script. 单节点卸载不需要 Mooncake 的 RDMA 存储；其段在 DSXE EFA 网卡上注册时每个 rank 都因 ENOMEM 失败。删除不再使用的 kimik3-b300-mooncake.sh 安装脚本。
# Conflicts: # inferencex-e2e/perf-changelog.yaml
# Conflicts: # inferencex-e2e/perf-changelog.yaml
…ctor SimpleCPUOffloadConnector asserts CPU hits are scheduler-block aligned; 128-token KDA partial hits killed the engine at c48. SimpleCPUOffloadConnector 要求 CPU 命中按调度块对齐；128 token 的 KDA 部分命中 导致 c48 时引擎崩溃。
# Conflicts: # inferencex-e2e/perf-changelog.yaml
1 participant
Runs the Kimi-K3 B300 DSXE TP8 (DCP8, DSpark, host DRAM KV tier) AgentX sweep at conc 1, 4, 16, 24, 32, 40, 48, 56 and 64, with every point's KV offload getting the full 3 TB AgentX DRAM ceiling.
dram-utilization0.75 → 1.0:total-cpu-dram-gb2249 → 2999 (theMAX_AGENTIC_AVAILABLE_CPU_DRAM_MIBcap; DSXE nodes report ~4.17 TB).SimpleCPUOffloadConnector(kv-offload-backend: vllm-simple) withcpu_bytes_to_use_per_rank374875000000 (2999 GB / 8). Single-node offload needs no RDMA; Mooncake's segment registration on the DSXE EFA rail failed withCannot allocate memoryon every rank at both 349 GiB and 174 GiB per rank. The now-unusedkimik3-b300-mooncake.shsetup script is removed.prefix-match-unit: 128, matching the MI355X Kimi-K3SimpleCPUOffloadConnectorrecipe. With it, Kimi-K3's KDA layers allow 128-token prefix hits, but the connector asserts CPU hits are whole scheduler blocks; c48 died onnum_external_tokens=21760 not aligned to group 0 block_size=12288. GPU prefix-cache matching becomes block-granular.no-evals: true.[1, 4, 16, 24, 32, 40, 48, 56, 64]: conc 2 and 8 are dropped, and the conc 70 point becomes conc 64 (max-num-seqs128, same graph capture list andgpu-memory-utilization0.9).Risk: ~3 TB of pinned host memory leaves ~1.2 TB of the node for the ~1.5 TB checkpoint's page cache and the vLLM processes. Watch for host OOM or a slower weight load.
中文
在 Kimi-K3 B300 DSXE TP8（DCP8、DSpark、主机 DRAM KV 层）上运行 conc 1、4、16、24、32、40、48、56 和 64 的 AgentX 扫描，每个点的 KV 卸载都使用完整的 3 TB AgentX DRAM 上限。
dram-utilization0.75 → 1.0：total-cpu-dram-gb2249 → 2999（即MAX_AGENTIC_AVAILABLE_CPU_DRAM_MIB上限；DSXE 节点约有 4.17 TB）。SimpleCPUOffloadConnector（kv-offload-backend: vllm-simple），cpu_bytes_to_use_per_rank为 374875000000（2999 GB / 8）。单节点卸载不需要 RDMA；Mooncake 在 DSXE EFA 网卡上注册段时，每个 rank 349 GiB 和 174 GiB 均因Cannot allocate memory失败。删除不再使用的kimik3-b300-mooncake.sh安装脚本。prefix-match-unit: 128，与使用SimpleCPUOffloadConnector的 MI355X Kimi-K3 配方一致。设置该值时 Kimi-K3 的 KDA 层允许 128 token 的前缀命中，但该 connector 要求 CPU 命中为完整调度块；c48 因num_external_tokens=21760 not aligned to group 0 block_size=12288崩溃。GPU 前缀缓存匹配改为按块粒度。no-evals: true。[1, 4, 16, 24, 32, 40, 48, 56, 64]：移除 conc 2 和 8，conc 70 点改为 conc 64（max-num-seqs128，CUDA graph 捕获列表和gpu-memory-utilization0.9 不变）。风险： 约 3 TB 的锁页主机内存只给节点留下约 1.2 TB，用于约 1.5 TB 检查点的页缓存和 vLLM 进程。需关注主机 OOM 或权重加载变慢。
AI model disclosure
🤖 Generated with Claude Code