Studio: overturn a chat-only MLX verdict the stack contradicts (#9120) - #9124
Conversation
…unslothai#9120) Hardware detection runs on a startup thread concurrent with the torch warm, llama.cpp probes, and GGUF precache. The first-ever import of transformers — inside mlx_lm's own import chain — can land while another thread still has it partially initialized, and 'from transformers import AutoTokenizer' then raises 'cannot import name ...' on a perfectly healthy install. Because detect_hardware() caches its verdict for the process's lifetime, one unlucky race at boot permanently greyed out Train/Export (chat-only) with a message directing the user at 'unsloth studio update', which correctly reported nothing to fix. The runtime import probe now retries only that signature, with backoff: the race clears on its own once the other thread's import finishes, while a genuinely broken install ('No module named ...', shared-library errors, ...) still fails on the first attempt and pays nothing. Retries are bounded; a persistent failure is reported as before. Three regression tests: the race shape is retried and then passes, a missing module is reported on the first attempt with no retries, and a persistent partial-import failure reports after exactly three attempts.
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e60b5023c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The diagnosis in unslothai#9120 holds, and so does the mechanism the previous commit describes: detection runs as the torch warm's first stage, early enough that another startup thread can still be part-way through the same first import of transformers, and CPython hands the second thread a partially initialised module rather than deadlock. What that commit did about it does not hold up. Retrying on `isinstance(exc, ImportError) and "cannot import name" in str(exc)` cannot separate the race from a genuinely broken install: that is also the shape a resolver backtrack produces, which is the failure this module exists to repair. `tests/test_mlx_stack_blockers.py::test_an_import_that_raises_is_reported_with_its_error` already pinned that shape under the comment "which is what a mlx-vlm built against a different transformers looks like from here", and the retry made it spend 1.5s per detection pass sleeping before reporting a verdict it was always going to reach. The retry also left the underlying problem in place: the verdict is cached for the process, so a race outlasting the backoff still greys out Train/Export for the session. Detection is not the only measurement of the stack. `start_mlx_autorepair_if_needed()` takes another one statement after `join_background_warm()`, once the warm has finished importing, and used it only to decide whether to reinstall -- discarding it in exactly the case where it contradicted the cached verdict. The self-heal saw a healthy stack, declined, and left the wrong verdict latched with nothing else that would ever revisit it, which is why `unsloth studio update` correctly reported nothing to fix. The gate now acts on both answers of that single measurement: a usable stack means the verdict was measured during the race, so it re-detects. No retries, no backoff, and no reading exception messages. The re-detect is scoped to the detection epoch read before the measurement, so a shutdown landing in it discards the pass instead of republishing for a lifespan that has ended, and the verdict read and the re-detect share one locked section. The success message reflects whether the verdict actually moved, since a pass can decline on a retired epoch or be discarded under one. UNSLOTH_DISABLE_MLX_AUTOREPAIR=1 no longer skips this -- it declines a reinstall, not a correct verdict -- but such a user reaches the measurement only when a verdict is waiting on it, so the opt-out still imports nothing on a host with nothing to overturn. The overturn reaches only a verdict published before the post-warm handoff, which is the warm's. Under the warm's own kill switch detection is deferred to the first request that needs it, and a verdict landing after this point stands unreconciled, as it does today.
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…annot say Comment-only: the parsed syntax tree is unchanged with docstrings stripped, and the tests and their mutation checks were re-run against it. Mostly deletions of prose that restated a definition. The test names here are already sentences, so the docstrings and lead comments repeating them are gone; what is left on each is the part the name cannot carry -- the issue this reproduces, why the opt-out does not apply, why two measurements would strand the verdict. The helper docstrings keep only the reason a settled verdict needs three parts and the reason the verdict read is deliberately unlocked, and drop the rest.
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Confirmed the post-warm measurement in studio/backend/utils/mlx_repair.py is still discarded when it contradicts the cached chat-only verdict, so a raced boot keeps Train and Export greyed out for the session. Will get this reviewed. |
|
@Lyxot could you take a look at this one? |
The overturn puts an unguarded call into utils.hardware on the healthy Apple Silicon path for the first time: before this branch a usable stack returned from start_mlx_autorepair_if_needed() without touching that module at all, and now every healthy Mac boot reaches overturn_the_mlx_verdict(). A half-applied update -- new mlx_repair.py over an older hardware.py, which the macOS updater permits since it rewrites site-packages under a live backend and never restarts it -- therefore lands in the post-warm handler as an AttributeError. Swallowed at debug, that leaves Train and Export greyed out for the whole session with nothing in the log to explain it, which is the same dead end unslothai#9120 was about. The block already catches everything; it just needs to be audible.
for more information, see https://pre-commit.ci
Comment and docstring wording only, no code changes. Keeps the concurrency reasoning (why the verdict read stays unlocked, why the epoch is read before the measurement, why the opt-out check moved after it) while cutting the line count.

Fixes #9120.
The bug
On Apple Silicon, Train/Export is gated on the MLX stack being usable. Hardware detection runs as the torch warm's first stage, early enough that another startup thread can still be part-way through the same first import of
transformers. CPython hands the second thread a partially initialised module rather than deadlock, sofrom transformers import AutoTokenizer— reached insidemlx_lm's own import chain — raisesImportError: cannot import name ...on a perfectly healthy install.detect_hardware()caches that verdict for the process's lifetime, so one lost race at boot greys out Train/Export behind a "rununsloth studio update" tooltip that correctly reports nothing to fix. The reporter lost the race on 3 of 3 consecutive launches.Root cause, and why the fix is this small
start_mlx_autorepair_if_needed()already re-measures the stack, one statement afterjoin_background_warm()in the post-warm worker, atif mlx_stack_available(): return False. That measurement happens once the warm has finished importing, so it is not the one that raced — and the gate was throwing it away in exactly the case where it contradicted the cached verdict. The self-heal saw a healthy stack, declined to reinstall, and left the wrong verdict latched with nothing else that would ever revisit it.So the gate now acts on both answers of that single measurement rather than only one: a usable stack means the verdict was measured during the race, and it re-detects.
Why not retry the import
The first approach on this branch retried when
isinstance(exc, ImportError) and "cannot import name" in str(exc). That signature cannot separate the race from a genuinely broken install: it is also what a resolver backtrack produces, which is the failure this module exists to repair.tests/test_mlx_stack_blockers.py::test_an_import_that_raises_is_reported_with_its_erroralready pinned exactly that shape, under the comment "which is what a mlx-vlm built against a different transformers looks like from here" — and the retry made that test spend 1.5s per detection pass sleeping before reporting a verdict it was always going to reach.Retrying also leaves the underlying problem in place. The verdict is cached for the process's lifetime, so a race that outlasts the backoff still greys out Train/Export for the session. The disagreement between two independent measurements is evidence that a substring match is not.
What changed
utils/hardware/hardware.pygainsoverturn_the_mlx_verdict(epoch), which re-detects while the published verdict still blames the MLX stack, andverdict_blames_the_mlx_stack().utils/mlx_repair.pycalls the overturn on the measurement it already took, and moves theUNSLOTH_DISABLE_MLX_AUTOREPAIRcheck after it.main.pyis untouched.Details worth a reviewer's attention:
_run_repair_and_redetectalready carries, and without it the next lifespan would findDEVICEset and skip its own detection. The repair worker now shares that epoch instead of reading its own, later one.UNSLOTH_DISABLE_MLX_AUTOREPAIR=1no longer skips the overturn. It declines a reinstall, not a correct verdict, and re-detecting mutates nothing. Such a user reaches the measurement only when a verdict is actually waiting on it, so the opt-out still imports nothing on a host with nothing to overturn — which matters because the torch warm has its own kill switch, and under it this would otherwise be the process's first MLX import./api/healthmakes.Cost
Nothing on a healthy boot, and nothing on a genuinely broken stack: the measurement is the one the gate already took. Where the verdict is overturned, the re-detect's own stack check is served from
sys.modules— measured on an Apple Silicon host at 3010 ms for the first measurement and 1.3 ms for the second.Known limitation
The overturn reaches only a verdict published before the post-warm handoff, which is the warm's. Under the warm's own kill switch (
UNSLOTH_STUDIO_DISABLE_TORCH_WARM=1) detection is deferred to the first request that needs it, and a verdict landing after this point stands unreconciled for the session. That is the behaviour onmaintoday as well; closing it means moving where the self-heal is scheduled, which is a larger change than this bug and is left as follow-up. The limitation is stated in the code.Validation
python -m pytest tests/test_mlx_repair.py tests/test_startup_defers_stack_dependent_work.py \ tests/test_warm_window_review_fixes.py tests/test_health_holds_verdict_during_mlx_repair.py \ tests/test_mlx_stack_blockers.py -q # 200 passedA broader sweep (
-k "mlx or hardware or warm or startup or health or detect") gives 1654 passed with 5 failures that reproduce identically on a cleanmaincheckout under the same command.Every added test was mutation-checked: each production line was reverted in turn and exactly the intended test failed — dropping the overturn, ignoring which gate produced the verdict, checking the opt-out before the measurement, dropping the epoch scope, reading the epoch after the measurement, reading the worker's epoch at spawn, measuring the stack twice, overturning regardless of the measurement, and announcing an overturn that never published.