{{ message }}
Repository navigation
[Fix] server: Serialize multi-tokenizer general warmups - #45584
Open
Phoenix3334 wants to merge 1 commit into
Open
Phoenix3334 wants to merge 1 commit into
Phoenix3334 wants to merge 1 commit into
Conversation
Serialize general warmup execution across tokenizer worker processes while preserving each worker's post-warmup callbacks and GC lifecycle. Fixes sgl-project#45345.
Phoenix3334
requested review from
CatherineSue,
JustinTong0323,
ispobock,
merrymercy and
slin1237
as code owners
October 21, 2026 10:24
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #45345.
Motivation
In multi-tokenizer mode, every tokenizer worker independently executes the general server warmup. Multimodal warmups can therefore overlap across all workers and exceed startup memory headroom before serving client traffic.
The investigation in #45345 reproduced this on a two-node GB300 K3 deployment (
cuda_vmm,mem_fraction_static=0.90, default 448×448 VLM warmup):Each point was run once. The scheduler-side peak remained nearly flat, suggesting that the additional pressure came from tokenizer/multimodal warmup overlap.
This is separate from #45384, which addresses eager CUDA VMM feature-transport initialization.
Modifications
flock, keyed by the existing shared main-process PID undertempfile.gettempdir().execute_warmup_funccall when multiple tokenizer workers are configured.post_warmup_callback, and subsequent GC-freeze flow. Callbacks remain outside the lock.--warmupsare outside this change's scope.Accuracy Tests
No changes to model computations or request outputs.
Local validation of this rebased change:
git diff --checkpassed.The full SGLang dependency environment is unavailable locally, so the complete registered test file and full pre-commit suite have not been run here. Full CPU-suite execution remains for CI:
Speed Tests and Profiling
The #45345 N=32 A/B retained all per-worker warmups and changed only their concurrency:
The serialized queue took about 90 seconds: the first warmup took about 9.4 seconds and later warmups about 2.3 seconds each. The follow-up hardware acceptance also reported a successful real image request.
Validation provenance: these hardware results were collected in the investigation's v0.5.22-based environment with the lazy-init branch and the same serialization mechanism. They are not a hardware rerun of this current-main commit. This PR is based on upstream
2e7c9d851c868678ddfcd93b4e8000cd23e1f779and preserves its current worker-local callback lifecycle.Checklist
CI States
Latest PR Test (Base): ❌ Run #39584344078
Latest PR Test (Extra): ❌ Run #39584344028
Latest PR Test (AMD ROCm 10): ❌ Run #39584344244