Repository navigation
[NVIDIA][AgentX] Kimi-K3 B300 TP8 sweep with lazy DRAM KV offload / [NVIDIA][AgentX] Kimi-K3 B300 TP8 惰性 DRAM KV 卸载扫描 - #4362
Oseltamivir wants to merge 2 commits into
Conversation
Same as the 1.5 TB eager run except lazy_offload: true, which copies cached blocks from the GPU evictable queue before eviction instead of every computed block. 与 1.5 TB eager 运行相同，但设置 lazy_offload: true：在驱逐前从 GPU 可驱逐队列 复制已缓存块，而不是复制每个计算完成的块。
将 perf-changelog 条目链接到 #4362。
|
Thanks for the contribution!
中文感谢你的贡献！
|
Oseltamivir
added
full-sweep-fail-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Besides the inline finding, I checked three other things and they hold up. The recipe's override_cN variants (1, 4, 16, 24, 32, 40, 48, 56, 64) match the new conc-list. The per-rank cpu_bytes_to_use_per_rank equals 1499 GB divided by 8. Nothing else in the repo still references kimik3-b300-mooncake.sh, and the { name: vllm-simple } with no version follows existing entries.
Extended reasoning...
This change swaps the Kimi-K3 B300 single-node AgentX recipe from Mooncake to SimpleCPUOffloadConnector in lazy mode. It also deletes the old setup script and its INSTALLERS entry. It is config only and touches no security-sensitive code. The note records the recipe-to-master consistency checks that were made beyond the inline changelog finding.
| - "Lower dram-utilization from 0.75 to 0.5 (total-cpu-dram-gb 2249 -> 1499, cpu_bytes_to_use_per_rank 187375000000)." | ||
| - "Drop prefix-match-unit 128: SimpleCPUOffloadConnector asserts CPU cache hits are scheduler-block aligned, and 128-token KDA partial hits killed the engine." | ||
| - "Sweep concurrency 1, 4, 16, 24, 32, 40, 48, 56 and 64: drop conc 2, 8 and 70 and add conc 64 (max-num-seqs 128)." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD |
There was a problem hiding this comment.
🟡 nit (optional): If this merges as is, the permanent changelog record links to pull/TBD, not this PR. That link is dead. The file is append-only, so it cannot be corrected after merge. Fix: before merge, set pr-link to this PR's real number.
Why this was flagged
The new block at inferencex-e2e/perf-changelog.yaml:10505 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD. AGENTS.md:140 calls the file append-only and byte-sensitive and says to keep every existing byte, so a merged placeholder cannot be fixed by a later PR. The schema at infx/matrix/validation.py:1069 types pr_link only as str, so nothing rejects the placeholder. Anyone tracing the kimik3-fp4-b300-vllm-agentic-dspark result change back to its PR from this entry reaches a dead link. Earlier pull/XXX entries (lines 5598, 9727, 10290) show this has slipped through before, which is precedent, not a safeguard.
Verification: nit. Triggered when this PR merges with the block as written. The new entry at inferencex-e2e/perf-changelog.yaml:10505 has pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD, which is a dead link, and the pydantic schema (infx/matrix/validation.py:1069, pr_link: str) does not reject it. A plain GitHub merge lands the dead link on main.
This branch has not been deployed
1 participant
Lazy-offload variant of #4349 (1.5 TB, eager). In #4348/#4349 the DRAM tier served almost nothing even though 1.5 TB holds each run's entire reusable KV ~3.7× over: GPU + CPU hits stayed 15–39 points below the theoretical ~96% at conc 48–64. The leading, unverified explanation is that eager mode rarely lands Kimi-K3's KDA state for a block boundary in DRAM (mamba "align" mode frees the previous state block each step, and the eager store skips freed blocks), so CPU hits that need that state never match. Lazy mode copies cached blocks from the GPU evictable queue just before eviction, which includes freed-but-still-hashed state blocks.
lazy_offload: truein theSimpleCPUOffloadConnectorconfig (cpu_bytes_to_use_per_rank187375000000, 1499 GB total).[1, 4, 16, 24, 32, 40, 48, 56, 64], noprefix-match-unit, throughput only (no-evals: true); drops Mooncake and the unusedkimik3-b300-mooncake.shlike the other variants.#4348, #4349, #4361 and this PR all edit the same config entry; only one can merge as is.
中文
#4349（1.5 TB，eager）的惰性卸载版本。在 #4348/#4349 中，尽管 1.5 TB 足以容纳每次运行全部可复用 KV 约 3.7 倍，DRAM 层几乎没有提供命中：conc 48–64 时 GPU + CPU 命中率仍比理论值约 96% 低 15–39 个百分点。目前主要但未经验证的解释是：eager 模式很少把 Kimi-K3 在块边界处的 KDA 状态写入 DRAM（mamba "align" 模式每一步都会释放上一个状态块，而 eager 存储会跳过已释放的块），因此需要该状态的 CPU 命中无法匹配。惰性模式在驱逐前从 GPU 可驱逐队列复制已缓存块，其中包括已释放但仍带哈希的状态块。
SimpleCPUOffloadConnector配置中lazy_offload: true（cpu_bytes_to_use_per_rank187375000000，总计 1499 GB）。[1, 4, 16, 24, 32, 40, 48, 56, 64]，不设prefix-match-unit，仅测吞吐（no-evals: true）；与其他版本一样移除 Mooncake 及不再使用的kimik3-b300-mooncake.sh。#4348、#4349、#4361 与本 PR 修改同一配置条目，按现状只能合并其中一个。
AI model disclosure
🤖 Generated with Claude Code