…und endpoints (vllm-project#55844)

Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

…#271)

## Summary

<!--
Briefly describe what this PR changes and why.
-->
Enabling Mooncake currently sets its config path only in the driver,
while CPU KV events can alter the GPU parser's block-size and hash
state. Validate the optional config before Ray startup, pass supported
settings through the Ray job environment, and ignore CPU Stored/Removed
events before they affect GPU routing state.

This lets existing agent inference use a shared CPU Store through
`MooncakeStoreConnector`. Store lookup/save/load stays in the connector;
routing continues to score GPU residency.

## Changes

<!-- List the key changes. -->

- Validate the optional Mooncake JSON before Ray startup and explicitly
pass the resolved path and supported settings through the Ray job
environment, with forwarded-key and mismatch diagnostics.
- Generate connector settings once, optionally save decode KV, and
preserve a pre-start snapshot of the submitted settings in result
metadata.
- Exclude CPU/SSD/unknown-medium Stored/Removed events from GPU routing
state with per-node/medium diagnostics, and warn once per malformed
recognized-event batch.
- Add embedded and standalone CPU configuration examples, a path-based
opt-in CLI and a short setup/run README.
- Cover configuration propagation, metadata snapshots, CLI validation
and event isolation with targeted tests.

## Compatibility and diagnostics

Supply `--mooncake-config-path /path/to/store.json` to enable Mooncake;
omit the
path to disable it. `--mooncake-save-decode-cache` is a `store_true`
option and
requires the config path and GPU device. The former `--enable-mooncake`
and
`--no-mooncake-save-decode-cache` options are removed. This entrypoint's
Store
integration requires the vLLM backend.

Config validation returns a resolved path through the Ray job
environment; the
caller explicitly records that path without rewriting the driver's
global
environment. Startup logs list the explicitly forwarded variable names,
and an
already-initialized Ray job reports the requested/active values of
mismatching
variables. This does not automatically forward `MC_TCP_*` settings.

Connector settings are generated once in `init_config` and copied from
the
submitted configuration before engine startup. Result metadata reuses
this
snapshot, including the original path and enablement, even if arguments
or
configuration objects are subsequently changed. It records submitted
settings,
not proof of successful Store reads/writes or the engine's resolved
defaults.

The vLLM KV-event parser admits GPU Stored/Removed events,
case-insensitively,
to the router's cache view. Missing/None media retain legacy GPU
behavior.
CPU and SSD events are excluded and logged at INFO; all other non-None
media,
including empty strings, are excluded with a WARNING. Excluded events
cannot
learn GPU block size, alter GPU hash mappings or populate CPU/SSD
residency
counts. This also applies to native CPU offload without Mooncake.
Backend Store
operations and separately collected vLLM external-hit metrics remain
available.

Filtered event/block counts are cumulative for the parser's lifetime and
keyed
by node and medium. Each key logs its first event, then at most once
every
30 seconds when more events arrive. These totals describe received
events,
not CPU/SSD occupancy. Unrecognized media are excluded from GPU state
and
require explicit support before entering the routing cache view.

Malformed recognized events produce one WARNING summary per batch,
including
the node, rejected-event count and first rejected event's type/error
reason.
Valid events in the same batch continue to be processed.

## Validation

- 336 CPU tests passed for the agent-aware router and inference
configuration on the rebased source; 15 special-sanity tests also
passed.
- Full pre-commit passes using upstream's configuration: ruff,
ruff-format, mypy and compileall.
- Native GPU PUT/GET readback and complete agent runs were exercised in
the recorded dependency environments below.

## Experimental results

Common setup: Qwen3-8B, mini-swe-agent, six GPUs with three TP2
replicas, a maximum context length of 16,384, and 64 samples × 8
rollouts = 512 attempts. GPU prefix caching and rl-insight are enabled.
The main tables use standalone mode and select the latest available
result for each configuration.

Wall time runs from the first driver-log timestamp to
`generate_sequences summary`. Throughput is total backend-generated
tokens divided by this full wall time and six GPUs; it includes startup,
tools, sandboxes and evaluation. No tail time is removed.

Mean TTFT is the sum of the three replicas' cumulative TTFT sums divided
by their total sample count. Hottest replica TTFT is the maximum
per-replica mean. TTFT includes engine queueing, external KV loading and
prefill. P50/P95 TTFT and model request E2E are interpolated from merged
histogram buckets and are estimates, especially in wide buckets. These
are raw backend metrics without reward/success filtering.

Load balance is the maximum/minimum cumulative `local_compute` prompt
tokens across replicas. Prefill volume is the sum of
`vllm:prompt_tokens_by_source_total{source="local_compute"}` and
excludes cached/transferred prompt tokens. Evictions/replica counts GPU
BlockRemoved events divided by three; it is reported as block removals
rather than a proven count of capacity-only evictions.

The reported GPU, external and combined hit rates are arithmetic means
of per-replica ratios. GPU hit uses GPU prefix-query tokens; external
hit uses external query tokens after GPU misses; combined hit uses `(GPU
hits + external hits) / GPU queries` on each replica. Parentheses show
the per-replica GPU-hit range. The two tier hit rates cannot be added
directly.

Total turns uses the last cumulative router dispatch counters rather
than completed backend requests. Per-request routing lines were not
emitted in these runs; the cumulative counters count the same dispatch
events and avoid Ray log deduplication. The 128-concurrent KV-aware 0.7
/ 128 GiB run and the 16-concurrent KV-aware 0.7 / disabled run each
have a one-request tail difference from the backend completion counter.

All main-table Δ values compare with Mooncake disabled: hit-rate changes
use percentage points (pp), turn counts show absolute and relative
changes, and other metrics use relative percentages. Deltas use
unrounded values. Runs differ in time window, some sandbox settings and
application revisions; these are observed comparisons with the recorded
sandbox/KV-load failures, rather than strict same-source, same-window
causal controls.

### Sticky: concurrency 128, disabled / 64 GiB / 128 GiB

| Metric | Mooncake disabled | 64 GiB | Δ vs disabled | 128 GiB | Δ vs
disabled |
| --- | --- | --- | --- | --- | --- |
| Wall time (min) | 313.15 | 325.87 | +4.1% | 312.85 | −0.1% |
| Mean TTFT (s) | 37.33 | 56.51 | +51.4% | 97.49 | +161.1% |
| Hottest replica TTFT (s) | 42.23 | 58.29 | +38.0% | 99.65 | +135.9% |
| Load balance (prefill tokens, max/min) | 1.21× | 1.32× | +9.5% | 1.05×
| −13.2% |
| Prefill volume (M tokens) | 136.62 | 105.19 | −23.0% | 8.70 | −93.6% |
| Evictions/replica (GPU block removals) | 2,801,619 | 2,906,756 | +3.8%
| 2,650,206 | −5.4% |
| gpu_hit (replica mean and range) | 26.92% (26.5–27.6) | 26.71%
(26.0–28.0) | −0.21 pp | 22.29% (20.7–25.4) | −4.64 pp |
| Total turns (router dispatches) | 21,918 | 23,115 | +1,197 (+5.5%) |
21,041 | −877 (−4.0%) |
| Throughput (Tokens/s/GPU) | 24.89 | 24.66 | −0.9% | 24.60 | −1.2% |
| P50 TTFT (s, estimated) | 39.94 | 58.53 | +46.6% | 110.15 | +175.8% |
| P95 TTFT (s, estimated) | 75.99 | 135.60 | +78.4% | 155.02 | +104.0% |
| P50 model request E2E (s, estimated) | 66.94 | 79.97 | +19.5% | 106.13
| +58.5% |
| External KV hit rate (replica mean) | — | 27.06% | — | 93.53% | — |
| Combined KV hit rate (replica mean) | 26.92% | 46.54% | +19.62 pp |
94.97% | +68.05 pp |

### KV-aware 0.7: concurrency 128, disabled / 64 GiB / 128 GiB

| Metric | Mooncake disabled | 64 GiB | Δ vs disabled | 128 GiB | Δ vs
disabled |
| --- | --- | --- | --- | --- | --- |
| Wall time (min) | 308.17 | 203.73 | −33.9% | 244.02 | −20.8% |
| Mean TTFT (s) | 37.31 | 38.68 | +3.7% | 40.15 | +7.6% |
| Hottest replica TTFT (s) | 53.92 | 56.45 | +4.7% | 45.80 | −15.1% |
| Load balance (prefill tokens, max/min) | 1.19× | 1.34× | +12.5% |
1.17× | −1.7% |
| Prefill volume (M tokens) | 126.94 | 55.69 | −56.1% | 24.22 | −80.9% |
| Evictions/replica (GPU block removals) | 2,547,509 | 1,806,758 |
−29.1% | 1,466,678 | −42.4% |
| gpu_hit (replica mean and range) | 34.60% (33.0–36.8) | 48.29%
(41.1–61.1) | +13.69 pp | 56.73% (53.9–59.6) | +22.13 pp |
| Total turns (router dispatches) | 22,760 | 23,543 | +783 (+3.4%) |
23,438 | +678 (+3.0%) |
| Throughput (Tokens/s/GPU) | 26.16 | 41.14 | +57.3% | 31.93 | +22.1% |
| P50 TTFT (s, estimated) | 27.68 | 15.25 | −44.9% | 6.04 | −78.2% |
| P95 TTFT (s, estimated) | 132.13 | 149.50 | +13.1% | 379.81 | +187.4%
|
| P50 model request E2E (s, estimated) | 55.76 | 22.15 | −60.3% | 11.83
| −78.8% |
| External KV hit rate (replica mean) | — | 44.39% | — | 72.19% | — |
| Combined KV hit rate (replica mean) | 34.60% | 70.52% | +35.93 pp |
88.00% | +53.40 pp |

The 0.7 / 128 GiB run includes about 80 minutes waiting for the final
session; the full duration is retained.

### KV-aware 0.9: concurrency 128, disabled / 64 GiB / 128 GiB

| Metric | Mooncake disabled | 64 GiB | Δ vs disabled | 128 GiB | Δ vs
disabled |
| --- | --- | --- | --- | --- | --- |
| Wall time (min) | 284.30 | 204.43 | −28.1% | 151.23 | −46.8% |
| Mean TTFT (s) | 34.98 | 42.21 | +20.7% | 29.13 | −16.7% |
| Hottest replica TTFT (s) | 42.93 | 58.42 | +36.1% | 49.84 | +16.1% |
| Load balance (prefill tokens, max/min) | 1.22× | 1.14× | −6.3% | 1.75×
| +43.0% |
| Prefill volume (M tokens) | 132.65 | 63.73 | −52.0% | 19.57 | −85.2% |
| Evictions/replica (GPU block removals) | 2,639,836 | 1,906,797 |
−27.8% | 1,101,687 | −58.3% |
| gpu_hit (replica mean and range) | 34.09% (32.0–36.8) | 49.44%
(42.6–53.7) | +15.35 pp | 67.36% (59.4–76.8) | +33.28 pp |
| Total turns (router dispatches) | 23,735 | 24,325 | +590 (+2.5%) |
22,869 | −866 (−3.6%) |
| Throughput (Tokens/s/GPU) | 26.95 | 42.01 | +55.9% | 51.80 | +92.2% |
| P50 TTFT (s, estimated) | 27.92 | 17.64 | −36.8% | 0.32 | −98.9% |
| P95 TTFT (s, estimated) | 124.43 | 153.83 | +23.6% | 189.75 | +52.5% |
| P50 model request E2E (s, estimated) | 55.90 | 27.17 | −51.4% | 4.67 |
−91.6% |
| External KV hit rate (replica mean) | — | 39.42% | — | 68.20% | — |
| Combined KV hit rate (replica mean) | 34.09% | 69.16% | +35.08 pp |
89.23% | +55.14 pp |

### Deployment mode: standalone / embedded

Mooncake is enabled in both modes, with a total pool of 128 GiB and
concurrency 128. Δ compares embedded with standalone. Application/verl
revisions also differ between these historical runs.

**Sticky**

| Metric | standalone | embedded | Δ |
| --- | --- | --- | --- |
| Wall time (min) | 312.85 | 192.18 | −38.6% |
| Mean TTFT (s) | 97.49 | 37.40 | −61.6% |
| Hottest replica TTFT (s) | 99.65 | 38.11 | −61.7% |
| Load balance (prefill tokens, max/min) | 1.05× | 1.16× | +10.9% |
| Prefill volume (M tokens) | 8.70 | 12.35 | +41.9% |
| Evictions/replica (GPU block removals) | 2,650,206 | 3,096,148 |
+16.8% |
| gpu_hit (replica mean and range) | 22.29% (20.7–25.4) | 21.28%
(21.0–21.4) | −1.00 pp |
| Total turns (router dispatches) | 21,041 | 23,116 | +2,075 (+9.9%) |
| Throughput (Tokens/s/GPU) | 24.60 | 42.46 | +72.6% |
| P50 TTFT (s, estimated) | 110.15 | 48.79 | −55.7% |
| P95 TTFT (s, estimated) | 155.02 | 76.88 | −50.4% |
| P50 model request E2E (s, estimated) | 106.13 | 43.94 | −58.6% |
| External KV hit rate (replica mean) | 93.53% | 92.05% | −1.48 pp |
| Combined KV hit rate (replica mean) | 94.97% | 93.74% | −1.23 pp |

**KV-aware 0.7**

| Metric | standalone | embedded | Δ |
| --- | --- | --- | --- |
| Wall time (min) | 244.02 | 170.05 | −30.3% |
| Mean TTFT (s) | 40.15 | 24.44 | −39.1% |
| Hottest replica TTFT (s) | 45.80 | 32.01 | −30.1% |
| Load balance (prefill tokens, max/min) | 1.17× | 1.71× | +46.2% |
| Prefill volume (M tokens) | 24.22 | 17.13 | −29.3% |
| Evictions/replica (GPU block removals) | 1,466,678 | 2,279,805 |
+55.4% |
| gpu_hit (replica mean and range) | 56.73% (53.9–59.6) | 43.60%
(39.4–47.1) | −13.13 pp |
| Total turns (router dispatches) | 23,438 | 24,513 | +1,075 (+4.6%) |
| Throughput (Tokens/s/GPU) | 31.93 | 49.31 | +54.4% |
| P50 TTFT (s, estimated) | 6.04 | 12.48 | +106.7% |
| P95 TTFT (s, estimated) | 379.81 | 116.40 | −69.4% |
| P50 model request E2E (s, estimated) | 11.83 | 16.46 | +39.2% |
| External KV hit rate (replica mean) | 72.19% | 85.56% | +13.38 pp |
| Combined KV hit rate (replica mean) | 88.00% | 91.77% | +3.77 pp |

**KV-aware 0.9**

| Metric | standalone | embedded | Δ |
| --- | --- | --- | --- |
| Wall time (min) | 151.23 | 158.53 | +4.8% |
| Mean TTFT (s) | 29.13 | 21.06 | −27.7% |
| Hottest replica TTFT (s) | 49.84 | 33.83 | −32.1% |
| Load balance (prefill tokens, max/min) | 1.75× | 1.91× | +9.2% |
| Prefill volume (M tokens) | 19.57 | 16.43 | −16.1% |
| Evictions/replica (GPU block removals) | 1,101,687 | 1,876,828 |
+70.4% |
| gpu_hit (replica mean and range) | 67.36% (59.4–76.8) | 52.40%
(46.8–60.3) | −14.96 pp |
| Total turns (router dispatches) | 22,869 | 24,471 | +1,602 (+7.0%) |
| Throughput (Tokens/s/GPU) | 51.80 | 53.53 | +3.3% |
| P50 TTFT (s, estimated) | 0.32 | 4.96 | +1471.6% |
| P95 TTFT (s, estimated) | 189.75 | 99.12 | −47.8% |
| P50 model request E2E (s, estimated) | 4.67 | 11.23 | +140.3% |
| External KV hit rate (replica mean) | 68.20% | 82.92% | +14.71 pp |
| Combined KV hit rate (replica mean) | 89.23% | 91.60% | +2.37 pp |

### Concurrency: 16 / 128

Both use standalone mode. Sticky enabled columns both use 128 GiB.
KV-aware 0.7 enabled columns use 32 GiB at concurrency 16 and 128 GiB at
concurrency 128, so that comparison also changes capacity. No completed
16-concurrent KV-aware 0.9 run is available.

**Sticky (both enabled columns use 128 GiB)**

| Metric | 16 concurrent · disabled | 128 concurrent · disabled | 16
concurrent · enabled | 128 concurrent · enabled |
| --- | --- | --- | --- | --- |
| Wall time (min) | 246.80 | 313.15 | 351.02 | 312.85 |
| Mean TTFT (s) | 0.17 | 37.33 | 0.17 | 97.49 |
| Hottest replica TTFT (s) | 0.18 | 42.23 | 0.17 | 99.65 |
| Load balance (prefill tokens, max/min) | 2.16× | 1.21× | 3.41× | 1.05×
|
| Prefill volume (M tokens) | 7.66 | 136.62 | 7.17 | 8.70 |
| Evictions/replica (GPU block removals) | 212,905 | 2,801,619 | 213,921
| 2,650,206 |
| gpu_hit (replica mean and range) | 96.18% (96.0–96.4) | 26.92%
(26.5–27.6) | 96.33% (96.1–96.7) | 22.29% (20.7–25.4) |
| Total turns (router dispatches) | 23,653 | 21,918 | 24,335 | 21,041 |
| Throughput (Tokens/s/GPU) | 40.41 | 24.89 | 30.34 | 24.60 |
| P50 TTFT (s, estimated) | 0.08 | 39.94 | 0.08 | 110.15 |
| P95 TTFT (s, estimated) | 0.59 | 75.99 | 0.52 | 155.02 |
| P50 model request E2E (s, estimated) | 0.88 | 66.94 | 0.88 | 106.13 |
| External KV hit rate (replica mean) | — | — | 4.41% | 93.53% |
| Combined KV hit rate (replica mean) | 96.18% | 26.92% | 96.50% |
94.97% |

**KV-aware 0.7 (16 concurrent enabled: 32 GiB; 128 concurrent enabled:
128 GiB)**

| Metric | 16 concurrent · disabled | 128 concurrent · disabled | 16
concurrent · enabled | 128 concurrent · enabled |
| --- | --- | --- | --- | --- |
| Wall time (min) | 179.72 | 308.17 | 314.12 | 244.02 |
| Mean TTFT (s) | 0.18 | 37.31 | 0.17 | 40.15 |
| Hottest replica TTFT (s) | 0.19 | 53.92 | 0.18 | 45.80 |
| Load balance (prefill tokens, max/min) | 1.51× | 1.19× | 2.07× | 1.17×
|
| Prefill volume (M tokens) | 7.59 | 126.94 | 7.23 | 24.22 |
| Evictions/replica (GPU block removals) | 204,731 | 2,547,509 | 209,245
| 1,466,678 |
| gpu_hit (replica mean and range) | 96.36% (96.2–96.5) | 34.60%
(33.0–36.8) | 96.11% (96.0–96.2) | 56.73% (53.9–59.6) |
| Total turns (router dispatches) | 24,190 | 22,760 | 22,775 | 23,438 |
| Throughput (Tokens/s/GPU) | 50.42 | 26.16 | 35.27 | 31.93 |
| P50 TTFT (s, estimated) | 0.08 | 27.68 | 0.08 | 6.04 |
| P95 TTFT (s, estimated) | 0.63 | 132.13 | 0.55 | 379.81 |
| P50 model request E2E (s, estimated) | 0.97 | 55.76 | 0.89 | 11.83 |
| External KV hit rate (replica mean) | — | — | 4.22% | 72.19% |
| Combined KV hit rate (replica mean) | 96.36% | 34.60% | 96.27% |
88.00% |
| Peak GPU KV usage (%) | 35.58 | 100.00 | 34.40 | 100.00 |
| Threshold fallbacks | 0 | 19,536 | 0 | 16,389 |

At concurrency 16, peak GPU KV usage was about 35% and the 0.7 threshold
did not trigger a fallback. At concurrency 128, usage reached 100% and
triggered fallbacks.

## Experimental environment

The following environment was used for the latest Store validation runs.


| Component | Tested version / preparation |
| ------------ |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
| vLLM | `0.29.0`, base `98dff2a81d747d1dba01a47f939f48c3526d4206`, with
KV-event endpoint discovery support from
[#55844](vllm-project/vllm#55844) (`90531daf`)
and Store scheduling runtime backports from
[#54643](vllm-project/vllm#54643) (`34b1e9f7`) /
[#54853](vllm-project/vllm#54853) (`0b066293`) |
| Mooncake | `mooncake-transfer-engine-cuda13
0.3.13.post1+tcp.pump.74d26b56`, a local build of base
`719735896c86b56fabec6cf3e825fb2ea640597a` with the runtime pump
correction in
[74d26b56](kvcache-ai/Mooncake@74d26b5)
from [#4063](kvcache-ai/Mooncake#4063) |
| Torch / Ray | `2.13.0+cu130` / `2.56.1` |
| Yuanrong SDK | `openyuanrong-sandbox==0.10.2rc9` for full agent runs |


The current Router requires #55844 for port-0 KV-event binding and
discovery through `GET /kv_event_sources`; this capability is already
present in the installed runtime. Separately, #54643/#54853 address
Store scheduling/save state and missing current block tables.

The Mooncake pump correction ensures that already accepted TCP work is
still pumped after a queue-full rejection. Only that correction was
backported, not the entire PR's queued-byte admission feature.

## Deployment

Standalone storage separates pool capacity/lifetime from model workers;
a single owner can concentrate copy/TCP load. Embedded storage lets
workers contribute memory and can copy locally when accessing their own
process segment, but does not guarantee self-placement. Same-host memory
in another process still uses TCP in this setup. Both modes use a
master; embedded is not limited to a single node.

Size the pool for reusable KV that outlives GPU residency. Aggregate GPU
KV capacity is a starting reference, not a mandatory minimum: replica
duplication, reuse distance and retained generations matter. Compare
64/128 GiB under fixed conditions, monitoring allocation/eviction,
successful GET volume and TTFT. Include per-rank private buffers and
other host memory in the budget; increasing capacity did not
monotonically improve full-agent throughput.

### Recorded worker/owner TCP settings

```text
MC_TCP_MAX_QUEUED_TRANSFERS_PER_PEER=49152
MC_TCP_MAX_PENDING_ADMISSIONS_PER_PEER=1024
MC_TCP_LANES_PER_PEER=4
MC_TCP_ADMISSION_TIMEOUT_MS=1000
```

The first two values bound transfer work items and pending admission,
not agent sessions. Increasing them consumes more buffers and can
increase latency. Set them in the actual owner and worker environments
before initialization; the UniAgent entry point does not automatically
forward all `MC_TCP_*` settings. For an existing Ray cluster, use the
node launch environment or explicit Ray Jobs `runtime_env.env_vars`;
driver-shell exports alone are insufficient.

## Checklist

- [x] The PR is focused and linked to an issue or explains why no issue
is needed.
- [x] The title follows the format above and names the layer that owns
the change.
- [x] Tests cover the behavior, or the Validation section explains why
they are not practical.
- [x] User-facing API, config, and workflow changes include
documentation or runnable examples.
- [x] Compatibility impact and migration steps are documented.
- [x] Logs, fixtures, and examples contain no credentials or private
data.
- [x] `pre-commit run --all-files --show-diff-on-failure` passes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added Mooncake Store setup examples for embedded and standalone
CPU-cache configurations.
* Added Mooncake options to the inference example, including
configuration validation and optional result reporting.
* **Bug Fixes**
* CPU KV-cache events no longer affect GPU cache tracking, and malformed
block-hash data is rejected.
* **Documentation**
* Added guidance on Mooncake setup, replica compatibility, worker
configuration, and cache-load failure behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->