[Fix] server: Serialize multi-tokenizer general warmups by Phoenix3334 · Pull Request #45584 · sgl-project/sglang · GitHub
Skip to content

[Fix] server: Serialize multi-tokenizer general warmups - #45584

Open
Phoenix3334 wants to merge 1 commit into
sgl-project:mainfrom
Phoenix3334:fix/serialize-multi-tokenizer-general-warmups
Open

Phoenix3334 wants to merge 1 commit into
sgl-project:mainfrom
Phoenix3334:fix/serialize-multi-tokenizer-general-warmups

Conversation

@Phoenix3334

@Phoenix3334 Phoenix3334 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #45345.

Motivation

In multi-tokenizer mode, every tokenizer worker independently executes the general server warmup. Multimodal warmups can therefore overlap across all workers and exceed startup memory headroom before serving client traffic.

The investigation in #45345 reproduced this on a two-node GB300 K3 deployment (cuda_vmm, mem_fraction_static=0.90, default 448×448 VLM warmup):

Tokenizer/detokenizer workers VLM warmups observed Recorded peak GPU0 usage (MiB) Result
1 1 261,425 READY
2 2 262,357 READY
4 4 264,590 READY
8 8 266,767 READY
16 16 269,325 READY
32 32 282,299 OOM

Each point was run once. The scheduler-side peak remained nearly flat, suggesting that the additional pressure came from tokenizer/multimodal warmup overlap.

This is separate from #45384, which addresses eager CUDA VMM feature-transport initialization.

Modifications

  • Add a per-server advisory flock, keyed by the existing shared main-process PID under tempfile.gettempdir().
  • Hold the lock only around the general execute_warmup_func call when multiple tokenizer workers are configured.
  • Keep every worker's warmup, worker-local post_warmup_callback, and subsequent GC-freeze flow. Callbacks remain outside the lock.
  • Single-tokenizer mode does not open a lock file. Existing skipped-warmup paths remain outside the lock. Custom --warmups are outside this change's scope.
  • Keep the lock file in place so queued workers use the same inode. The OS releases the advisory lock if a worker exits.
  • Add two CPU unit tests: four real worker processes must complete four warmups with maximum in-flight warmups equal to one, preserving per-worker callback/GC ordering; single-tokenizer mode must not open a lock file.

Accuracy Tests

No changes to model computations or request outputs.

Local validation of this rebased change:

  • Both new tests passed using the actual production functions and test class extracted with Python AST into an isolated standard-library harness.
  • Ruff 0.15.1 formatting and repository-selected lint rules passed.
  • isort 7.0.0 passed with the repository's Black profile and first/third-party settings.
  • Python syntax compilation and git diff --check passed.

The full SGLang dependency environment is unavailable locally, so the complete registered test file and full pre-commit suite have not been run here. Full CPU-suite execution remains for CI:

python -m pytest test/registered/unit/entrypoints/test_http_server_warmup.py -q

Speed Tests and Profiling

The #45345 N=32 A/B retained all per-worker warmups and changed only their concurrency:

General warmup mode Recorded peak GPU0 usage (MiB) Result
Concurrent 282,299 OOM
Serialized 263,275 READY; all 32 warmups completed

The serialized queue took about 90 seconds: the first warmup took about 9.4 seconds and later warmups about 2.3 seconds each. The follow-up hardware acceptance also reported a successful real image request.

Validation provenance: these hardware results were collected in the investigation's v0.5.22-based environment with the lazy-init branch and the same serialization mechanism. They are not a hardware rerun of this current-main commit. This PR is based on upstream 2e7c9d851c868678ddfcd93b4e8000cd23e1f779 and preserves its current worker-local callback lifecycle.

Checklist

  • Format code with the repository-pinned Ruff/isort versions and settings.
  • Add CPU unit tests in the existing registered test file.
  • Follow the SGLang code style guidance.
  • Run the complete registered CPU test file and full pre-commit suite in the SGLang environment.
  • Documentation: N/A; internal startup synchronization.
  • Model accuracy / steady-state throughput benchmarks: N/A; startup scheduling only.

CI States

Latest PR Test (Base): ❌ Run #39584344078
Latest PR Test (Extra): ❌ Run #39584344028
Latest PR Test (AMD ROCm 10): ❌ Run #39584344244

Serialize general warmup execution across tokenizer worker processes while
preserving each worker's post-warmup callbacks and GC lifecycle.

Fixes sgl-project#45345.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Startup OOM with 32 tokenizer workers and auto-selected CUDA VMM transport on multi-node GB300

1 participant