[Core] Don't issue a blocking collective RPC during engine handshake by okorzh-amd · Pull Request #57226 · vllm-project/vllm · GitHub
Skip to content

[Core] Don't issue a blocking collective RPC during engine handshake - #57226

Merged
AndreasKaratzas merged 4 commits into
vllm-project:mainfrom
okorzh-amd:okorzh/no-blocking-rpc-in-engine-handshake
Sep 17, 2026
Merged

AndreasKaratzas merged 4 commits into
vllm-project:mainfrom
okorzh-amd:okorzh/no-blocking-rpc-in-engine-handshake

Conversation

@okorzh-amd

Copy link
Copy Markdown
Contributor

Purpose

EngineCoreProc._make_ready_response() calls Executor.supports_draft_weight_updates(),
which is a collective_rpc issued with no timeout (v1/executor/abstract.py). That
response is built inside process_input_sockets() before ready_event.set(), while
EngineCoreProc.__init__ waits on the same event with a fixed timeout
(v1/engine/core.py):

while not ready_event.wait(timeout=10):
    if not input_thread.is_alive():
        raise RuntimeError("Input socket thread died during startup")
    assert addresses.coordinator_input is not None
    logger.info("Waiting for READY message from DP Coordinator...")

A slow engine→worker round trip therefore stalls the handshake. With
data_parallel_size=1, addresses.coordinator_input is None, so the first expiry
raises a bare AssertionError and kills an engine whose workers are all alive and idle.
The front end then reports Failed core proc(s): {} — empty, because nothing actually
failed. At DP>1 the identical stall is benign: the loop logs and retries.

Observed on 8-GPU ROCm runs. The input thread's own traceback shows where it sits:

core.py     process_input_sockets      -> ready_response = self._make_ready_response()
core.py     _make_ready_response       -> self.model_executor.supports_draft_weight_updates()
abstract.py supports_draft_weight_updates -> self.collective_rpc(...)      # blocked, no deadline

The round trip is avoidable. Worker.supports_draft_weight_updates() short-circuits on
self.weight_transfer_engine is not None, and that engine is only ever constructed under
if vllm_config.weight_transfer_config is not None (v1/worker/gpu_worker.py). With no
weight-transfer config every worker returns False, so the RPC cannot change the answer.
This PR consults the executor only when weight transfer is configured. Behaviour with
weight transfer configured is unchanged.

Scope note: this removes the call that was observed blocking. It does not change the
timeout=10 wait or the bare assert, which remain a trap for anything else that lands on
the pre-ready path. Those are worth addressing separately; the ordering of the DP-coordinator
subscription relative to the executor call makes that a larger change than it looks.

Why this is not duplicating an existing PR

The closest open PR is #48871 ("Surface engine startup failure diagnostics where startup
actually waits"). It is complementary, not overlapping: #48871 changes
v1/engine/core_client.py and v1/engine/utils.py to improve reporting on the front-end
side, where startup waits. This PR changes v1/engine/core.py to remove the engine-side
cause of one class of those stalls. Neither subsumes the other, and with both applied a
stall that still occurs would be reported better by #48871.

Note both PRs add tests to tests/v1/engine/test_engine_core_client.py, so whichever lands
second will need a trivial rebase there.

Also reviewed and distinct: #37757 (startup error logging UX), #51018 (DP world-group port
binding), #55137 (Mooncake bootstrap registration hang).

Test Plan

Unit tests added to tests/v1/engine/test_engine_core_client.py:

pytest tests/v1/engine/test_engine_core_client.py \
  -k "ready_response_skips_draft_rpc or ready_response_queries_executor" -q
  • test_ready_response_skips_draft_rpc_without_weight_transfer — asserts the executor is
    not consulted and the reported value is False. Fails before this change.
  • test_ready_response_queries_executor_with_weight_transfer — asserts the executor is
    consulted and its value is reported. Passes before and after, pinning the path this change
    must not alter.

End-to-end: Kimi-K3, TP8, MI355X (gfx950), ROCm 10, v0.29.1rc1.dev187, with a probe
wrapping Executor.supports_draft_weight_updates and _make_ready_response.

Test Result

Unit tests, run against the installed package in both an unpatched and a patched image:

test before after
..._skips_draft_rpc_without_weight_transfer FAILED passed
..._queries_executor_with_weight_transfer passed passed
before:  1 failed, 1 passed
after:   2 passed

End-to-end on 8×MI355X, after the change:

[RPCPROBE] _make_ready_response built in 0.00s; supports_draft_weight_updates=False
[RPCPROBE] supports_draft_weight_updates() CALLED   ->  0 occurrences
INFO:     Application startup complete.

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

EngineCoreProc._make_ready_response() calls
Executor.supports_draft_weight_updates(), which is a collective_rpc issued
with no timeout. That response is built inside process_input_sockets()
*before* ready_event.set(), while EngineCoreProc.__init__ waits on the same
event with a fixed timeout:

    while not ready_event.wait(timeout=10):
        if not input_thread.is_alive():
            raise RuntimeError("Input socket thread died during startup")
        assert addresses.coordinator_input is not None

So a slow engine->worker round trip during startup stalls the handshake. With
data_parallel_size=1 addresses.coordinator_input is None, and the first
timeout raises a bare AssertionError, killing an engine whose workers are all
alive and idle. The log then reports "Failed core proc(s): {}" because nothing
actually failed. At DP>1 the same stall is benign: the loop logs and retries.

The round trip is avoidable. Worker.supports_draft_weight_updates() short
circuits on `self.weight_transfer_engine is not None`, and that engine is only
constructed under `if vllm_config.weight_transfer_config is not None`. When no
weight transfer is configured every worker returns False, so the RPC cannot
change the answer. Consult the executor only when weight transfer is
configured; behaviour with weight transfer is unchanged.

Observed on 8-GPU ROCm runs where the engine aborted at startup roughly one
run in three; after this change _make_ready_response() completes in 0.00s and
the RPC is not issued, with the reported value unchanged (False).

Signed-off-by: okorzh <okorzh@amd.com>
@okorzh-amd
okorzh-amd requested a review from njhill as a code owner September 16, 2026 19:00

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #89424 for commit b4355eeeff51.

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci retry

@github-actions

Copy link
Copy Markdown

✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/89424

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

@AndreasKaratzas
AndreasKaratzas merged commit 2307a1d into vllm-project:main Sep 17, 2026
129 checks passed
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
…llm-project#57226)

Signed-off-by: okorzh <okorzh@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants