iframe-proxy

xhx1022 · 2026-04-11T08:55:12Z

Background

The previous routed_experts_capturer relied on SharedMemory + fcntl.flock + /tmp lock files for cross-process transport. The worker was forced into a cpu().numpy() on every step, and the scheduler performed a flock + shm read for every finished request. Consequences: synchronous CUDA calls on the hot path, frequent NCCL timeouts on cross-node EP, an outright AssertionError under SP + modular-kernel, and routed-experts data being dropped or misaligned under abort / preemption / async scheduling.

This PR reworks both the transport layer and the correctness boundaries from the ground up.

1. Drop shm / flock, route through `ModelRunnerOutput`

2. New `topk_ids` Shape Under Sequence Parallelism

The old capture() recognized only two layouts: n == total (naive) and n == token_num_per_dp (modular). SP shards along dim=0 across the TP group with ceil-div padding, which introduces a third shape n == ceil(token_num_per_dp / tp_size); the old code fell through to the else branch and asserted.

This PR adds a dedicated SP branch: get_tp_group().all_gather(topk_ids, dim=0) reconstructs the full per-DP-rank tensor; the downstream assignment device_buffer[:token_num_per_dp, layer_id, :] = topk_ids[start_loc:end_loc, :] naturally discards the padding rows; int32/int64 inputs are implicitly cast to int32 by __setitem__.

3. Async D2H

The old path contained two forced synchronizations: device_buffer[...].cpu().numpy() and slot_mapping_attn[...].cpu().numpy().

init_routed_experts_capturer now pre-allocates three buffers:

routed_experts_cpu (pinned CPU): the D2H landing buffer for routing data.
routed_experts_slot_mapping_cpu (pinned CPU): the D2H landing buffer for slot_mapping.
routed_experts_slot_mapping_device (private GPU copy): a per-step snapshot of slot_mapping. We cannot fire D2H directly from block_table.slot_mapping, because the next step's _prepare_inputs rewrites it in place and would tear the data pending on the copy stream.
Sync path: two copy_(..., non_blocking=True) calls are enqueued; the existing event.synchronize() inside _to_list already covers them, so there is zero additional synchronization.
Async path: at the end of execute_model a RoutedExpertsTensors (post-clone()) is handed to AsyncGPUModelRunnerOutput; to_cpu_nonblocking() runs on async_output_copy_stream followed by async_copy_ready_event.record(), and get_output() waits on the event before calling tolists(). RoutedExpertsTensors is deliberately modeled after LogprobsTensors (a NamedTuple with the same public API, sharing the copy stream and the ready event), so no extra stream / event overhead is introduced.

4. Data Integrity Under Abort / Preemption / Async Scheduling

4.1 Problems

The old _send_abort_outputs_to_client did not carry routed_experts.
Once kv_cache_manager.free(request) runs, the reclaimed blocks may be reassigned to other requests, which then overwrite the corresponding rows in the slot buffer.
Async off-by-one race: under step_with_batch_queue, step N+1's schedule() executes before step N's update_from_output (which contains store_batch). If X is preempted in step N+1, the captured slot buffer is missing the routing entries for the new tokens generated in step N; by the time step N's output finally lands, X's blocks have already been freed, so _get_routed_experts returns zero rows and falls back to the cached T-1 rows, while the client expects T rows → AssertionError.

Root cause: the slot buffer is subject to free / reuse, which makes a pre-free snapshot mandatory; but the async pipeline places the capture moment before the previous step's store_batch, so any single snapshot is unavoidably one row short.

4.2 Solution: Pre-free Capture + Post-output Refresh

RoutedExpertsCache: a dict[req_id, Entry] where Entry = (data, pending_block_ids, pending_num_tokens). The pending_* fields are a note left for the scheduler: "after the next update_from_output completes store_batch, re-read using this block list."
Pre-free capture: _preempt_request / finish_requests call cache.capture(req_id, data, block_ids, num_tokens) before releasing blocks. Internally this is take-the-best (the existing entry is replaced only when data.shape[0] > existing.length), to prevent a later, partial preemption from overwriting a more complete snapshot.
update_from_output ordering: routed_experts_mgr.store_batch(...) runs first, followed by cache.refresh_pending(); this guarantees that the routing of tokens generated in the final step of a finishing request is written before we re-read.
Take-the-best in the stopped branch: _get_routed_experts(request) + cache.get_best(req_id, current) compare lengths, return the more complete one, and pop the cache entry.
Slot clipping: max_slots = len(block_ids) * block_size; num_tokens = min(request.num_tokens - 1, max_slots) — prevents out-of-range reads when a preempted request has not yet been reallocated and avoids reading uncovered slots during partial re-prefill.

5. Multi-node TP16 + DP + EP

Worker hot path: all D2H is non-blocking and no longer forces CUDA-stream synchronization; cross-node EP collectives no longer time out waiting for the next kernel to be dispatched.
Scheduler hot path: update_from_output performs a single numpy fancy-index per step (on the order of a few MB); collective_rpc cadence remains steady and the main NCCL-timeout trigger is eliminated.
Cross-node transport: the dependency on /dev/shm + fcntl.flock is gone; routing data piggybacks on ModelRunnerOutput → EngineCore, so multi-node deployments incur no additional cost.

SP × TP=16 optimization (inside the capturer):

Shape detection for n == ceil(token_num_per_dp / tp_size).
Per-layer staging into _sp_staging(max_shard, L, top_k), avoiding an all_gather inside each layer's forward.
flush_sp() at the end of execute_model performs a single all-gather over the staged buffer, collapsing L per-layer NCCL ops into one step-level collective (measurably reducing EP latency on gpt-oss-120B with 36 MoE layers × TP16).
Safe / fast dual modes: safe (per-layer all-gather, identical to HEAD) is the default; VLLM_RE_FAST_FLUSH_SP=1 enables fast mode.

DP+EP per-rank semantics: the naive branch uses cumsum(num_tokens_dp) to compute [start_loc, end_loc); the modular branch takes the entire local-rank slice; the SP branch slices after all_gather. Together the three branches eliminate the "routing from another rank landing in my slot" misalignment.

6. MTP / Speculative Decoding

Capacity: the worker buffer is pre-allocated against max_num_batched_tokens, which already accounts for num_spec_tokens at the scheduler level; routed_experts_by_slot is sized as num_blocks * block_size and aligned with physical blocks, so speculative tokens do not claim additional slots.
Draft / target separation: capture is an MoE-layer hook and fires only during the target forward. If downstream code binds a draft MoE to the same capturer, overlapping layer ids will overwrite each other and must be avoided.
Rejected-draft isolation: writes from rejected drafts are constrained by each request's own block_ids + offset, so they are physically disjoint across requests. Within the same request, the next step advances num_computed_tokens by num_accepted_tokens, so rejected positions are overwritten by the next forward; _get_routed_experts further clips to request.num_tokens - 1 (accepted tokens only), which naturally excludes rejected slots.
Async spec-decode: reuses the copy stream and ready event of AsyncGPUModelRunnerOutput; RoutedExpertsTensors is enqueued via clone() and never contends with the drafter's default stream. The three-stage flow of pre-free capture + refresh_pending is oblivious to the target / spec distinction.

TODO

Not yet supported: PP, DCP, PCP, hierarchical KV cache, PD disaggregation, and the flashinfer backend.

Experiment Result

Close: #38079

gemini-code-assist

Code Review

This pull request refactors the routed experts capture mechanism, replacing the shared-memory-based implementation with a device-buffer-based approach managed by a new RoutedExpertsManager. It introduces support for returning routed experts in the disaggregated serving protocol and ensures that routing data is captured for aborted requests before KV-cache blocks are freed. Review feedback identifies a critical race condition in GPUModelRunner where shared buffers and metadata could be overwritten during concurrent batch execution, as well as a performance bottleneck caused by a blocking GPU-CPU synchronization when handling slot mappings.

gemini-code-assist · 2026-04-11T09:01:01Z

            if self.routed_experts_initialized:
-                capturer = RoutedExpertsCapturer.get_instance()
-                if capturer is not None:
-                    capturer.save_captured_experts(indices=self.slot_mapping)  # noqa
-                else:
-                    logger.error("RoutedExpertsCapturer not initialized.")
+                # Non-blocking D2H on the default stream into pinned memory.
+                # clear_buffer() in the next step is also on the default
+                # stream, so it is automatically ordered after this copy.
+                # For async scheduling, the copy stream's wait_stream(default)
+                # ensures async_copy_ready_event covers this copy too.
+                buf = self.routed_experts_capturer.get_device_buffer()
+                total = scheduler_output.total_num_scheduled_tokens
+                self.routed_experts_cpu[:total].copy_(buf[:total], non_blocking=True)
+                self._re_total = total
+                self._re_slot_mapping = self.routed_experts_slot_mapping


There is a critical race condition here when using v1 with batch_queue_size > 1 (e.g., in pipeline parallelism or async scheduling).

Shared Buffer Overwrite: self.routed_experts_cpu is a single shared buffer on the GPUModelRunner. If execute_model(N+1) is called before sample_tokens(N) (which happens when the batch queue is used), the copy_ for batch N+1 will overwrite the data for batch N before the scheduler has a chance to consume it.

Stale Metadata: Similarly, self.routed_experts_slot_mapping is a shared attribute. execute_model(N+1) will overwrite it with the slot mapping of the new batch, causing sample_tokens(N) to associate batch N's routing data with batch N+1's slots.

Device Buffer Clearing: execute_model(N+1) calls self.routed_experts_capturer.clear_buffer() at the start. While stream ordering ensures the copy_ for batch N starts before the zero_ for batch N+1, any delay in the D2H transfer could lead to reading zeroed or corrupted data if the next batch's forward pass starts filling the buffer.

Recommendation: Move the routed experts capture logic into execute_model (before it returns) and use per-batch CPU tensors instead of a shared buffer. You should use buf[:total].to("cpu", non_blocking=True) to get a unique tensor for each batch and store it in the ExecuteModelState (or a similar per-batch container) to be retrieved later in sample_tokens.

gemini-code-assist · 2026-04-11T09:01:01Z

+            self.routed_experts_slot_mapping = (
+                slot_mapping_attn[:num_tokens].cpu().numpy()
+            )


This .cpu().numpy() call is a blocking GPU-CPU synchronization point in the middle of the model execution path. In the v1 engine, especially with async scheduling, we should avoid all blocking syncs to maintain high throughput and low latency.

Recommendation: Use a non-blocking transfer for the slot mapping metadata (e.g., slot_mapping_attn[:num_tokens].to("cpu", non_blocking=True)) and defer the conversion to a numpy array until get_output() is called, where synchronization is already handled by the async_copy_ready_event.

mergify · 2026-04-13T02:48:07Z

…nsfer and HTTP support (vllm-project#39568) Signed-off-by: xhx1022 <1737006628@qq.com> Signed-off-by: arlenxu <arlenxu@tencent.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: arlenxu <arlenxu@tencent.com> Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>

…nsfer and HTTP support (vllm-project#39568) Signed-off-by: xhx1022 <1737006628@qq.com> Signed-off-by: arlenxu <arlenxu@tencent.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: arlenxu <arlenxu@tencent.com> Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com> Signed-off-by: Liuweixiong0118 <lwx34158427@gmail.com>

vLLM 0.20.2's routed-experts capture overruns its shared-memory buffer on hybrid-attention MoE models with multiple KV-cache groups (vllm-project/vllm#40692); vLLM 0.22.0 ships the rewritten capture (vllm-project/vllm#39568). flashinfer moves to 0.6.11.post2, matching vLLM 0.22.x's own exact pins; torch stays 2.11.0+cu128 via the cu129 vLLM wheel index. vLLM >= 0.22 also natively defines start/finish_weight_update on GPUWorker (vllm-project/vllm#39212) and asserts that worker extensions may not shadow Worker attrs, which killed every WorkerProc at EngineCore init with the driver seeing only "Engine core initialization failed. ... Failed core proc(s): {}". Rename NewInferenceWorkerWrap's two colliding methods with a skyrl_ prefix and update the collective_rpc method strings to match; client-facing RemoteInferenceClient method names are unchanged. Validated end-to-end on Qwen3-30B-A3B GRPO with R3 enabled (non-colocated 2-node, new inference stack): healthy rewards and rollout_train_logprobs_abs_diff_mean ~= 0.01. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: James Braza <jamesbraza@gmail.com>

vLLM 0.20.2's routed-experts capture overruns its shared-memory buffer on hybrid-attention MoE models with multiple KV-cache groups (vllm-project/vllm#40692); vLLM 0.22.0 ships the rewritten capture (vllm-project/vllm#39568). flashinfer moves to 0.6.11.post2, matching vLLM 0.22.x's own exact pins; torch stays 2.11.0+cu128 via the cu129 vLLM wheel index. vLLM >= 0.22 also natively defines start/finish_weight_update on GPUWorker (vllm-project/vllm#39212) and asserts that worker extensions may not shadow Worker attrs, which killed every WorkerProc at EngineCore init with the driver seeing only "Engine core initialization failed. ... Failed core proc(s): {}". Rename NewInferenceWorkerWrap's two colliding methods with a skyrl_ prefix and update the collective_rpc method strings to match; client-facing RemoteInferenceClient method names are unchanged. The same API (mandatory since vLLM 0.21.0) requires wrapping /update_weights with /start_weight_update and /finish_weight_update, and its non-packed IPC receive expects bare rebuild_cuda_tensor args in the handle dict. Make RemoteInferenceClient.update_named_weights issue the wrapping calls itself (each call carries the full weight set) and send bare handle args from the GPU weight-sync test's IpcTrainer. test_update_weights_flow[no_pd] and test_update_weights_ipc pass end-to-end on 0.22.1; the pd_1P1D param remains blocked by a pre-existing regression on main (NovaSky-AI#1759 switched the dependency to nixl-cu12, which ships module nixl_cu12, while vLLM imports nixl). Validated end-to-end on Qwen3-30B-A3B GRPO with R3 enabled (non-colocated 2-node, new inference stack): healthy rewards and rollout_train_logprobs_abs_diff_mean ~= 0.01. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: James Braza <jamesbraza@gmail.com>

### What this PR does / why we need it? Upstream PR [vllm-project/vllm#39568](vllm-project/vllm#39568) is a complete rewrite of the routed-experts capture/transport pipeline. It supersedes both: - The original 0.20.2 design — `RoutedExpertsCapturer.get_instance()` singleton, `save_captured_experts(indices=...)`, shared-memory + `fcntl.flock` cross-process transport. - The intermediate PR #39917 design — module-level `get_global_experts_capturer()`, `init_routed_experts_capturer_with_shared_cache()`, `issue_routing_d2h_copy()`, `extract_routed_experts_for_current_batch()`. This API existed in main for only a few days and was never in a stable release; it has been **fully removed**. After the upgrade to vLLM 0515, vllm-ascend faces two API surfaces that are incompatible at the source level: | Aspect | 0.20.2 | main | |---|---|---| | Capturer access | `RoutedExpertsCapturer.get_instance()` (singleton) | `runner.routed_experts_capturer` (per-runner instance, no global) | | Per-step `clear_buffer` | via singleton | via runner attribute | | Per-step D2H + ship | `capturer.save_captured_experts(indices=cpu_slot_mapping)` (sync, shm write) | runner-managed pinned `routed_experts_cpu` D2H + `RoutedExpertsLists` on `ModelRunnerOutput.routed_experts` | | Output channel | shm/flock to scheduler | `ModelRunnerOutput.routed_experts: RoutedExpertsLists` (NamedTuple, msgpack + zmq IPC) | | `slot_mapping` source | `slot_mapping.cpu().numpy()` saved to `self.cpu_slot_mapping` | private device snapshot `routed_experts_slot_mapping_device`, then pinned `routed_experts_slot_mapping_cpu` | | Layer hook injection | `select_experts` calls singleton from inside `apply()` | `module.router.set_capture_fn(...)` from `_bind_routed_experts_capturer` | ## Strategy Overview 1. **Keep the 0.20.2 path intact.** It already works end-to-end. All 0.20.2-specific call sites stay byte-identical. 2. **Add a parallel main path** gated by `vllm_version_is("0.20.2") == False`. Reuse upstream `GPUModelRunner.init_routed_experts_capturer()` (inherited) for buffer allocation; override only `_bind_routed_experts_capturer` because Ascend's `select_experts` does not go through upstream `BaseRouter`. 3. **Async scheduling: piggyback on upstream `AsyncGPUModelRunnerOutput`.** vllm-ascend already constructs that wrapper directly, so adding the `routed_experts=` kwarg is enough — the wrapper handles `to_cpu_nonblocking()` on its copy stream and `tolists()` finalization in `get_output()` for free. 4. **No new compat module, no monkey patches.** Branching is inline at each call site; total surface is one new method (`_bind_routed_experts_capturer`) plus three branched call sites in `model_runner_v1.py` and one in `fused_moe.py`. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.20.2 - vLLM main: vllm-project/vllm@ce29c26 --------- Signed-off-by: wangli <wangli858794774@gmail.com>

### What this PR does / why we need it? Upstream PR [vllm-project/vllm#39568](vllm-project/vllm#39568) is a complete rewrite of the routed-experts capture/transport pipeline. It supersedes both: - The original 0.20.2 design — `RoutedExpertsCapturer.get_instance()` singleton, `save_captured_experts(indices=...)`, shared-memory + `fcntl.flock` cross-process transport. - The intermediate PR #39917 design — module-level `get_global_experts_capturer()`, `init_routed_experts_capturer_with_shared_cache()`, `issue_routing_d2h_copy()`, `extract_routed_experts_for_current_batch()`. This API existed in main for only a few days and was never in a stable release; it has been **fully removed**. After the upgrade to vLLM 0515, vllm-ascend faces two API surfaces that are incompatible at the source level: | Aspect | 0.20.2 | main | |---|---|---| | Capturer access | `RoutedExpertsCapturer.get_instance()` (singleton) | `runner.routed_experts_capturer` (per-runner instance, no global) | | Per-step `clear_buffer` | via singleton | via runner attribute | | Per-step D2H + ship | `capturer.save_captured_experts(indices=cpu_slot_mapping)` (sync, shm write) | runner-managed pinned `routed_experts_cpu` D2H + `RoutedExpertsLists` on `ModelRunnerOutput.routed_experts` | | Output channel | shm/flock to scheduler | `ModelRunnerOutput.routed_experts: RoutedExpertsLists` (NamedTuple, msgpack + zmq IPC) | | `slot_mapping` source | `slot_mapping.cpu().numpy()` saved to `self.cpu_slot_mapping` | private device snapshot `routed_experts_slot_mapping_device`, then pinned `routed_experts_slot_mapping_cpu` | | Layer hook injection | `select_experts` calls singleton from inside `apply()` | `module.router.set_capture_fn(...)` from `_bind_routed_experts_capturer` | ## Strategy Overview 1. **Keep the 0.20.2 path intact.** It already works end-to-end. All 0.20.2-specific call sites stay byte-identical. 2. **Add a parallel main path** gated by `vllm_version_is("0.20.2") == False`. Reuse upstream `GPUModelRunner.init_routed_experts_capturer()` (inherited) for buffer allocation; override only `_bind_routed_experts_capturer` because Ascend's `select_experts` does not go through upstream `BaseRouter`. 3. **Async scheduling: piggyback on upstream `AsyncGPUModelRunnerOutput`.** vllm-ascend already constructs that wrapper directly, so adding the `routed_experts=` kwarg is enough — the wrapper handles `to_cpu_nonblocking()` on its copy stream and `tolists()` finalization in `get_output()` for free. 4. **No new compat module, no monkey patches.** Branching is inline at each call site; total surface is one new method (`_bind_routed_experts_capturer`) plus three branched call sites in `model_runner_v1.py` and one in `fused_moe.py`. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.20.2 - vLLM main: vllm-project/vllm@ce29c26 --------- Signed-off-by: wangli <wangli858794774@gmail.com> Signed-off-by: xqchen7 <chenxueqing7@huawei.com>

Replace the scheduler assertion with a fail-closed VLLM_ALLOW_ROUTED_EXPERTS_DCP gate while keeping PCP unsupported. Reconstruct rank-independent routing slots from scheduler block snapshots, verify the selected worker's PAD-bearing local map, and never store through a padding slot. Origin: reimplementation of mlfoundations/vllm@17c7c70 on the upstream vllm-project#39568 routed-expert architecture. Adaptation: scale scheduler routing blocks by DCP ownership and add recurring CPU coverage for both owners, configured interleave, a physical-block boundary, prefill/decode, opt-in, and PCP rejection. Validation: six focused scheduler/capture tests pass. Live TP8/DCP2 H100 runs also captured and collated routed-expert rows through the real vLLM-to-Harbor-to-MarinSkyRL consumer path, including the bounded 131072-token harvest. DCP remains explicit opt-in and PCP remains rejected.

…nsfer and HTTP support (vllm-project#39568) Signed-off-by: xhx1022 <1737006628@qq.com> Signed-off-by: arlenxu <arlenxu@tencent.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: arlenxu <arlenxu@tencent.com> Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>

xhx1022 requested review from ApostaC, WoosukKwon, alexm-redhat, heheda12345, mgoin, njhill, orozery, pavanimajety, robertgshaw2-redhat and ywang96 as code owners April 11, 2026 08:55

mergify Bot added frontend v1 labels Apr 11, 2026

xhx1022 changed the title ~~refactor(R3): eliminate shared memory IPC, transfer data via ModelRunnerOutput，support inference/v1/generate~~ [WIP] refactor(R3): eliminate shared memory IPC, transfer data via ModelRunnerOutput，support inference/v1/generate Apr 11, 2026

gemini-code-assist Bot reviewed Apr 11, 2026

View reviewed changes

xhx1022 changed the title ~~[WIP] refactor(R3): eliminate shared memory IPC, transfer data via ModelRunnerOutput，support inference/v1/generate~~ [WIP] [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support Apr 11, 2026

mergify Bot added the needs-rebase label Apr 13, 2026

xhx1022 force-pushed the r3_rfc branch from 0ef06e6 to 652700f Compare April 13, 2026 12:24

mergify Bot removed the needs-rebase label Apr 13, 2026

xhx1022 force-pushed the r3_rfc branch from d1e9260 to 5dfb465 Compare April 15, 2026 05:14

xhx1022 changed the title ~~[WIP] [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support~~ [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support Apr 16, 2026

xhx1022 force-pushed the r3_rfc branch 2 times, most recently from 71f4a0a to 38fe03f Compare April 17, 2026 14:08

youkaichao approved these changes Apr 18, 2026

View reviewed changes

xhx1022 force-pushed the r3_rfc branch 2 times, most recently from 7ee324c to 546e8f2 Compare April 18, 2026 10:40

xhx1022 requested review from DarkLight1337, hmellor and russellb as code owners April 18, 2026 10:40

mergify Bot added the performance Performance-related issues label Apr 18, 2026

S1ro1 mentioned this pull request May 21, 2026

[R3]: Move to new vLLM routed experts format PrimeIntellect-ai/prime-rl#2487

Merged

aoshen02 mentioned this pull request May 27, 2026

[Feature]Routing replay (R3) for vLLM rollout (vLLM v0.21.0) vllm-project/vime#34

Closed

CalvinXKY mentioned this pull request May 27, 2026

[Feature] Routing replay (R3) for vLLM rollout (vLLM 0.22+, /inference/v1/generate) vllm-project/vime#49

Merged

FurtherAI mentioned this pull request May 29, 2026

Enable default MoE expert replay for Megatron train/inference parity OpenPipe/ART#701

Merged

aoshen02 mentioned this pull request May 31, 2026

routed_experts all-zero with FlashInfer (TRTLLM/CUTLASS) MoE — wire FlashInfer routing_replay_out into the capturer #44116

Closed

h-aleix mentioned this pull request Jun 3, 2026

[Bug]: RoutedExpertsCapturer host buffer undersized for hybrid models with multiple KV cache groups #40692

Closed

3 tasks done

jamesbraza mentioned this pull request Jun 10, 2026

Rollout Routing Replay (R3) for the fsdp backend, using vllm==0.22 NovaSky-AI/SkyRL#1772

Open

aoshen02 mentioned this pull request Jul 11, 2026

[RFC] Training-Inference Consistency for RL #48305

Open

1 of 13 tasks

yonromai mentioned this pull request Jul 15, 2026

infra - one vllm fork marin-community/marin#7097

Closed

aoshen02 mentioned this pull request Aug 2, 2026

[MRV2] Enable routed-experts capture #50721

Merged

zhao-stack mentioned this pull request Sep 11, 2026

[CI] Reduce repeated work in vLLM interface analysis vllm-project/vllm-ascend#16364

Merged

Module	Change
Worker	`RoutedExpertsCapturer` is reduced to a GPU `int32` buffer plus `capture / clear / get_device_buffer`; all shm / flock / singleton / tempfile code is removed.
Scheduler	A new `RoutedExpertsManager` indexes `routed_experts_by_slot(num_slots, num_layers, top_k)` by slot, where `slot = block_id * block_size + offset`. This aligns with the KV-cache block pool, so prefix-caching reuse is obtained naturally.
Transport	GPU buffer → pinned CPU (non-blocking D2H) → `ModelRunnerOutput.routed_experts` → `Scheduler.update_from_output` → `store_batch`. Reuses the existing IPC path (same-node `ShmRingBuffer`, cross-node zmq) with zero extra cross-process sync.
Dtype	Transport uses `torch.int32` (NCCL-friendly); the scheduler-side large buffer uses `np.uint8` when `num_experts ≤ 256`, otherwise `np.uint16`.

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

[RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support - #39568

[RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support#39568
Isotr0py merged 25 commits into
vllm-project:mainfrom
xhx1022:r3_rfc

xhx1022 commented Apr 11, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 11, 2026

Uh oh!

gemini-code-assist Bot Apr 11, 2026

Uh oh!

mergify Bot commented Apr 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

xhx1022 commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

1. Drop shm / flock, route through ModelRunnerOutput

2. New topk_ids Shape Under Sequence Parallelism

3. Async D2H

4. Data Integrity Under Abort / Preemption / Async Scheduling

4.1 Problems

4.2 Solution: Pre-free Capture + Post-output Refresh

5. Multi-node TP16 + DP + EP

6. MTP / Speculative Decoding

TODO

Experiment Result

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Apr 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

xhx1022 commented Apr 11, 2026 •

edited

Loading

1. Drop shm / flock, route through `ModelRunnerOutput`

2. New `topk_ids` Shape Under Sequence Parallelism