{{ message }}
Repository navigation
[Core] Don't issue a blocking collective RPC during engine handshake - #57226
Merged
AndreasKaratzas merged 4 commits intoSep 17, 2026
Merged
AndreasKaratzas merged 4 commits into
AndreasKaratzas merged 4 commits into
Conversation
EngineCoreProc._make_ready_response() calls
Executor.supports_draft_weight_updates(), which is a collective_rpc issued
with no timeout. That response is built inside process_input_sockets()
*before* ready_event.set(), while EngineCoreProc.__init__ waits on the same
event with a fixed timeout:
while not ready_event.wait(timeout=10):
if not input_thread.is_alive():
raise RuntimeError("Input socket thread died during startup")
assert addresses.coordinator_input is not None
So a slow engine->worker round trip during startup stalls the handshake. With
data_parallel_size=1 addresses.coordinator_input is None, and the first
timeout raises a bare AssertionError, killing an engine whose workers are all
alive and idle. The log then reports "Failed core proc(s): {}" because nothing
actually failed. At DP>1 the same stall is benign: the loop logs and retries.
The round trip is avoidable. Worker.supports_draft_weight_updates() short
circuits on `self.weight_transfer_engine is not None`, and that engine is only
constructed under `if vllm_config.weight_transfer_config is not None`. When no
weight transfer is configured every worker returns False, so the RPC cannot
change the answer. Consult the executor only when weight transfer is
configured; behaviour with weight transfer is unchanged.
Observed on 8-GPU ROCm runs where the engine aborted at startup roughly one
run in three; after this change _make_ready_response() completes in 0.00s and
the RPC is not issued, with the reported value unchanged (False).
Signed-off-by: okorzh <okorzh@amd.com>
AndreasKaratzas
approved these changes
Sep 16, 2026
Member
|
/ci run |
|
✅ Triggered Buildkite CI #89424 for commit |
Member
|
/ci retry |
|
✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/89424 |
Member
|
/ci run |
michael-gpu
pushed a commit
to practical-inference/vllm-ampere-extended
that referenced
this pull request
Oct 20, 2026
…llm-project#57226) Signed-off-by: okorzh <okorzh@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Purpose
EngineCoreProc._make_ready_response()callsExecutor.supports_draft_weight_updates(),which is a
collective_rpcissued with no timeout (v1/executor/abstract.py). Thatresponse is built inside
process_input_sockets()beforeready_event.set(), whileEngineCoreProc.__init__waits on the same event with a fixed timeout(
v1/engine/core.py):A slow engine→worker round trip therefore stalls the handshake. With
data_parallel_size=1,addresses.coordinator_inputisNone, so the first expiryraises a bare
AssertionErrorand kills an engine whose workers are all alive and idle.The front end then reports
Failed core proc(s): {}— empty, because nothing actuallyfailed. At DP>1 the identical stall is benign: the loop logs and retries.
Observed on 8-GPU ROCm runs. The input thread's own traceback shows where it sits:
The round trip is avoidable.
Worker.supports_draft_weight_updates()short-circuits onself.weight_transfer_engine is not None, and that engine is only ever constructed underif vllm_config.weight_transfer_config is not None(v1/worker/gpu_worker.py). With noweight-transfer config every worker returns
False, so the RPC cannot change the answer.This PR consults the executor only when weight transfer is configured. Behaviour with
weight transfer configured is unchanged.
Scope note: this removes the call that was observed blocking. It does not change the
timeout=10wait or the bareassert, which remain a trap for anything else that lands onthe pre-ready path. Those are worth addressing separately; the ordering of the DP-coordinator
subscription relative to the executor call makes that a larger change than it looks.
Why this is not duplicating an existing PR
The closest open PR is #48871 ("Surface engine startup failure diagnostics where startup
actually waits"). It is complementary, not overlapping: #48871 changes
v1/engine/core_client.pyandv1/engine/utils.pyto improve reporting on the front-endside, where startup waits. This PR changes
v1/engine/core.pyto remove the engine-sidecause of one class of those stalls. Neither subsumes the other, and with both applied a
stall that still occurs would be reported better by #48871.
Note both PRs add tests to
tests/v1/engine/test_engine_core_client.py, so whichever landssecond will need a trivial rebase there.
Also reviewed and distinct: #37757 (startup error logging UX), #51018 (DP world-group port
binding), #55137 (Mooncake bootstrap registration hang).
Test Plan
Unit tests added to
tests/v1/engine/test_engine_core_client.py:pytest tests/v1/engine/test_engine_core_client.py \ -k "ready_response_skips_draft_rpc or ready_response_queries_executor" -qtest_ready_response_skips_draft_rpc_without_weight_transfer— asserts the executor isnot consulted and the reported value is
False. Fails before this change.test_ready_response_queries_executor_with_weight_transfer— asserts the executor isconsulted and its value is reported. Passes before and after, pinning the path this change
must not alter.
End-to-end: Kimi-K3, TP8, MI355X (gfx950), ROCm 10,
v0.29.1rc1.dev187, with a probewrapping
Executor.supports_draft_weight_updatesand_make_ready_response.Test Result
Unit tests, run against the installed package in both an unpatched and a patched image:
..._skips_draft_rpc_without_weight_transfer..._queries_executor_with_weight_transferEnd-to-end on 8×MI355X, after the change:
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.