Studio: overturn a chat-only MLX verdict the stack contradicts (#9120) by yzxcj797 · Pull Request #9124 · unslothai/unsloth · GitHub
Skip to content

Studio: overturn a chat-only MLX verdict the stack contradicts (#9120) - #9124

Merged
danielhanchen merged 7 commits into
unslothai:mainfrom
yzxcj797:fix/mlx-partial-import-race-9120
Aug 20, 2026
Merged

Studio: overturn a chat-only MLX verdict the stack contradicts (#9120)#9124
danielhanchen merged 7 commits into
unslothai:mainfrom
yzxcj797:fix/mlx-partial-import-race-9120

Conversation

@yzxcj797

@yzxcj797 yzxcj797 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Fixes #9120.

Note on approach. The original commit on this branch fixed the race by retrying the failing import. That approach is superseded here — the reasoning is in the "Why not retry the import" section below and in the replacing commit's message. The diagnosis in #9120, which this branch started from, is unchanged and is what made the real fix findable.

The bug

On Apple Silicon, Train/Export is gated on the MLX stack being usable. Hardware detection runs as the torch warm's first stage, early enough that another startup thread can still be part-way through the same first import of transformers. CPython hands the second thread a partially initialised module rather than deadlock, so from transformers import AutoTokenizer — reached inside mlx_lm's own import chain — raises ImportError: cannot import name ... on a perfectly healthy install.

detect_hardware() caches that verdict for the process's lifetime, so one lost race at boot greys out Train/Export behind a "run unsloth studio update" tooltip that correctly reports nothing to fix. The reporter lost the race on 3 of 3 consecutive launches.

Root cause, and why the fix is this small

start_mlx_autorepair_if_needed() already re-measures the stack, one statement after join_background_warm() in the post-warm worker, at if mlx_stack_available(): return False. That measurement happens once the warm has finished importing, so it is not the one that raced — and the gate was throwing it away in exactly the case where it contradicted the cached verdict. The self-heal saw a healthy stack, declined to reinstall, and left the wrong verdict latched with nothing else that would ever revisit it.

So the gate now acts on both answers of that single measurement rather than only one: a usable stack means the verdict was measured during the race, and it re-detects.

Why not retry the import

The first approach on this branch retried when isinstance(exc, ImportError) and "cannot import name" in str(exc). That signature cannot separate the race from a genuinely broken install: it is also what a resolver backtrack produces, which is the failure this module exists to repair. tests/test_mlx_stack_blockers.py::test_an_import_that_raises_is_reported_with_its_error already pinned exactly that shape, under the comment "which is what a mlx-vlm built against a different transformers looks like from here" — and the retry made that test spend 1.5s per detection pass sleeping before reporting a verdict it was always going to reach.

Retrying also leaves the underlying problem in place. The verdict is cached for the process's lifetime, so a race that outlasts the backoff still greys out Train/Export for the session. The disagreement between two independent measurements is evidence that a substring match is not.

What changed

  • utils/hardware/hardware.py gains overturn_the_mlx_verdict(epoch), which re-detects while the published verdict still blames the MLX stack, and verdict_blames_the_mlx_stack().
  • utils/mlx_repair.py calls the overturn on the measurement it already took, and moves the UNSLOTH_DISABLE_MLX_AUTOREPAIR check after it.
  • main.py is untouched.

Details worth a reviewer's attention:

  • The re-detect is epoch-scoped. The epoch is read before the measurement, so a shutdown landing inside it discards the pass instead of republishing for a lifespan that has ended — the same guard _run_repair_and_redetect already carries, and without it the next lifespan would find DEVICE set and skip its own detection. The repair worker now shares that epoch instead of reading its own, later one.
  • The verdict read and the re-detect are one locked section, so a forced pass landing between them does not lose its fresh answer to this one.
  • UNSLOTH_DISABLE_MLX_AUTOREPAIR=1 no longer skips the overturn. It declines a reinstall, not a correct verdict, and re-detecting mutates nothing. Such a user reaches the measurement only when a verdict is actually waiting on it, so the opt-out still imports nothing on a host with nothing to overturn — which matters because the torch warm has its own kill switch, and under it this would otherwise be the process's first MLX import.
  • The success message reflects the outcome, not the attempt. A pass can decline on a retired epoch or be discarded under one, so the answer is the same three-part settled read /api/health makes.

Cost

Nothing on a healthy boot, and nothing on a genuinely broken stack: the measurement is the one the gate already took. Where the verdict is overturned, the re-detect's own stack check is served from sys.modules — measured on an Apple Silicon host at 3010 ms for the first measurement and 1.3 ms for the second.

Known limitation

The overturn reaches only a verdict published before the post-warm handoff, which is the warm's. Under the warm's own kill switch (UNSLOTH_STUDIO_DISABLE_TORCH_WARM=1) detection is deferred to the first request that needs it, and a verdict landing after this point stands unreconciled for the session. That is the behaviour on main today as well; closing it means moving where the self-heal is scheduled, which is a larger change than this bug and is left as follow-up. The limitation is stated in the code.

Validation

python -m pytest tests/test_mlx_repair.py tests/test_startup_defers_stack_dependent_work.py \
    tests/test_warm_window_review_fixes.py tests/test_health_holds_verdict_during_mlx_repair.py \
    tests/test_mlx_stack_blockers.py -q
# 200 passed

A broader sweep (-k "mlx or hardware or warm or startup or health or detect") gives 1654 passed with 5 failures that reproduce identically on a clean main checkout under the same command.

Every added test was mutation-checked: each production line was reverted in turn and exactly the intended test failed — dropping the overturn, ignoring which gate produced the verdict, checking the opt-out before the measurement, dropping the epoch scope, reading the epoch after the measurement, reading the worker's epoch at spawn, measuring the stack twice, overturning regardless of the measurement, and announcing an overturn that never published.

…unslothai#9120)

Hardware detection runs on a startup thread concurrent with the torch warm,
llama.cpp probes, and GGUF precache. The first-ever import of transformers —
inside mlx_lm's own import chain — can land while another thread still has it
partially initialized, and 'from transformers import AutoTokenizer' then
raises 'cannot import name ...' on a perfectly healthy install. Because
detect_hardware() caches its verdict for the process's lifetime, one unlucky
race at boot permanently greyed out Train/Export (chat-only) with a message
directing the user at 'unsloth studio update', which correctly reported
nothing to fix.

The runtime import probe now retries only that signature, with backoff: the
race clears on its own once the other thread's import finishes, while a
genuinely broken install ('No module named ...', shared-library errors, ...)
still fails on the first attempt and pays nothing. Retries are bounded; a
persistent failure is reported as before.

Three regression tests: the race shape is retried and then passes, a missing
module is reported on the first attempt with no retries, and a persistent
partial-import failure reports after exactly three attempts.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e60b5023c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/utils/mlx_repair.py Outdated
@Lyxot Lyxot self-assigned this Aug 18, 2026
The diagnosis in unslothai#9120 holds, and so does the mechanism the previous commit describes: detection runs as the torch warm's first stage, early enough that another startup thread can still be part-way through the same first import of transformers, and CPython hands the second thread a partially initialised module rather than deadlock. What that commit did about it does not hold up.

Retrying on `isinstance(exc, ImportError) and "cannot import name" in str(exc)` cannot separate the race from a genuinely broken install: that is also the shape a resolver backtrack produces, which is the failure this module exists to repair. `tests/test_mlx_stack_blockers.py::test_an_import_that_raises_is_reported_with_its_error` already pinned that shape under the comment "which is what a mlx-vlm built against a different transformers looks like from here", and the retry made it spend 1.5s per detection pass sleeping before reporting a verdict it was always going to reach. The retry also left the underlying problem in place: the verdict is cached for the process, so a race outlasting the backoff still greys out Train/Export for the session.

Detection is not the only measurement of the stack. `start_mlx_autorepair_if_needed()` takes another one statement after `join_background_warm()`, once the warm has finished importing, and used it only to decide whether to reinstall -- discarding it in exactly the case where it contradicted the cached verdict. The self-heal saw a healthy stack, declined, and left the wrong verdict latched with nothing else that would ever revisit it, which is why `unsloth studio update` correctly reported nothing to fix.

The gate now acts on both answers of that single measurement: a usable stack means the verdict was measured during the race, so it re-detects. No retries, no backoff, and no reading exception messages.

The re-detect is scoped to the detection epoch read before the measurement, so a shutdown landing in it discards the pass instead of republishing for a lifespan that has ended, and the verdict read and the re-detect share one locked section. The success message reflects whether the verdict actually moved, since a pass can decline on a retired epoch or be discarded under one. UNSLOTH_DISABLE_MLX_AUTOREPAIR=1 no longer skips this -- it declines a reinstall, not a correct verdict -- but such a user reaches the measurement only when a verdict is waiting on it, so the opt-out still imports nothing on a host with nothing to overturn.

The overturn reaches only a verdict published before the post-warm handoff, which is the warm's. Under the warm's own kill switch detection is deferred to the first request that needs it, and a verdict landing after this point stands unreconciled, as it does today.
@Lyxot Lyxot changed the title fix(studio): retry the concurrent partial-import race in the MLX gate (#9120) Studio: overturn a chat-only MLX verdict the stack contradicts (#9120) Aug 18, 2026
@Lyxot

Lyxot commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: b09bdf90dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: b09bdf90dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…annot say

Comment-only: the parsed syntax tree is unchanged with docstrings stripped, and the tests
and their mutation checks were re-run against it.

Mostly deletions of prose that restated a definition. The test names here are already
sentences, so the docstrings and lead comments repeating them are gone; what is left on
each is the part the name cannot carry -- the issue this reproduces, why the opt-out does
not apply, why two measurements would strand the verdict. The helper docstrings keep only
the reason a settled verdict needs three parts and the reason the verdict read is
deliberately unlocked, and drop the rest.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: e3d33d40a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danielhanchen

Copy link
Copy Markdown
Member

Confirmed the post-warm measurement in studio/backend/utils/mlx_repair.py is still discarded when it contradicts the cached chat-only verdict, so a raced boot keeps Train and Export greyed out for the session. Will get this reviewed.

@danielhanchen

Copy link
Copy Markdown
Member

@Lyxot could you take a look at this one?

shimmyshimmer and others added 2 commits August 20, 2026 12:37
The overturn puts an unguarded call into utils.hardware on the healthy Apple Silicon path
for the first time: before this branch a usable stack returned from
start_mlx_autorepair_if_needed() without touching that module at all, and now every
healthy Mac boot reaches overturn_the_mlx_verdict().

A half-applied update -- new mlx_repair.py over an older hardware.py, which the macOS
updater permits since it rewrites site-packages under a live backend and never restarts
it -- therefore lands in the post-warm handler as an AttributeError. Swallowed at debug,
that leaves Train and Export greyed out for the whole session with nothing in the log to
explain it, which is the same dead end unslothai#9120 was about. The block already catches
everything; it just needs to be audible.
@chatgpt-codex-connector

Copy link
Copy Markdown

Comment and docstring wording only, no code changes. Keeps the concurrency reasoning (why the verdict read stays unlocked, why the epoch is read before the measurement, why the opt-out check moved after it) while cutting the line count.
@unslothai unslothai deleted a comment from Lyxot Aug 20, 2026
@unslothai unslothai deleted a comment from Lyxot Aug 20, 2026
@unslothai unslothai deleted a comment from Lyxot Aug 20, 2026
@danielhanchen
danielhanchen merged commit 7b54b42 into unslothai:main Aug 20, 2026
42 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Studio: MLX Train/Export falsely greyed out — startup thread race on first transformers import (not a broken install)

4 participants