Repository navigation
[MLX] Fix startup after logical token capacity change - #41314
Conversation
|
Thanks for fixing this. I independently verified the patch on Apple Silicon Environment:
I confirmed that the server starts successfully and an end-to-end I found one small rebase issue caused by Results on the latest
I left an inline suggestion for the test adjustment. Happy to send a follow-up |
| device="cpu", | ||
| enable_memory_saver=False, | ||
| ) | ||
| runner.token_to_kv_pool = SimpleNamespace(size=64) |
There was a problem hiding this comment.
The latest main reads token_to_kv_pool_allocator when reporting worker
capacity, so the fake runner needs a stock allocator stub.
| runner.token_to_kv_pool = SimpleNamespace(size=64) | |
| runner.token_to_kv_pool = SimpleNamespace(size=64) | |
| runner.token_to_kv_pool_allocator = SimpleNamespace() |
7e21f3b to
acd56c9
Compare

Motivation
MLX servers fail during startup after #39731 with
AttributeError: 'MlxModelRunnerStub' object has no attribute 'kv_cache_configurator'. The new capacity query reaches the Torch configurator, which the MLX stub does not create.Modifications
Report logical token capacity from the MLX pool, keeping the request pool's capacity adjustment. Add a regression covering worker startup and the request limits reported to the scheduler.
Accuracy Tests
mlx-community/Qwen3-0.6B-4bitnow starts and generates text. With default overlap scheduling, both radix-cache modes pass short and chunked-prefill requests, followed by a healthy/healthresponse.Speed Tests and Profiling
No model forward or sampling changes; no throughput claim.
Checklist
CI States
Latest PR Test (Base): ⏳ Run #36650163801
Latest PR Test (Extra): ❌ Run #36650163569
Latest PR Test (AMD ROCm 10): ⏳ Run #36650163553