Repository navigation
studio: serve installed mlx models through the openai-compatible api - #8768
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84835610d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4bae121fef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87cc929b42
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67ea601668
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ffc2e60fc0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ffc2e60fc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
@codex review |
ffc2e60 to
8407b3d
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4a6c1655a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2d82219f8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c5b2be285
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 473ca1b57c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e36894888
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Exact head Confirmed and fixed:
Investigated and rejected:
Verification at this head:
|
|
Why this diff is large:
There are no model files, generated bundles, new dependencies, or unrelated product features in the diff. The size is mainly the original broad feature plus defensive regression coverage. Extra agent review passes are stopped; only existing convergence threads and CI remain. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09a854d4a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Convergence fix |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a00f3417b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Convergence fix |
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
# Conflicts: # .github/workflows/studio-frontend-ci.yml # tests/studio/probe_dismiss_guard.py
Final repair and convergence summaryFinal head: Confirmed defects fixed
Why the diff is largeThe original feature already crossed local model resolution, API routing, catalog output, request preflight, MLX loading, settings copy, and twelve locale files. Repair rounds added focused regression coverage for every destructive switch edge case and the CI-only frontend failures. The final GitHub diff is 2,691 additions and 192 deletions across 19 files; most growth is tests. There are no model artifacts, generated bundles, new dependencies, or unrelated product features. The local conflict merge lists 41 files because it incorporates current Final verification
|
…ually shipped Two source pins in test_openai_auto_switch.py landed asserting a shape their own commit then refactored past, and Backend CI has carried them red since: - openai_responses and anthropic_messages hoist the image answer into a local (_responses_has_image / _anthropic_has_image) so the image preflight can reuse it; the pins still expected the guard to call the detector inline. Pin both halves: the derivation and the guard consuming it. anthropic_count_tokens still passes the call inline and keeps its original pin. - The non-streaming /v1/responses path delegates its preflight to chat, whose switch call leaves gguf_only at its False default rather than passing it, so captured["gguf_only"] KeyErrors. Read the kwarg with the same False default the callee applies; the streaming arm still requires the explicit True.

Fixes #8748.
An MLX model downloaded through Unsloth Desktop was invisible to the OpenAI-compatible API until someone loaded it by hand.
GET /v1/modelsdid not list it, and a chat-completions request naming its exact id returned 404model_not_foundwith "is not downloaded on this server", while installed GGUF models in the same setup were both listed and switchable./v1/modelsand the "Switch model by request" resolver both filtered the local catalog through a GGUF-only check (local_gguf_quants,resolve_local_gguf). The loaded-model branch is backend-agnostic, which is why the same model worked the moment it was loaded in the UI.What changed
studio/backend/core/inference/local_model_resolver.py_local_weights_entryclassifies a local non-GGUF checkpoint as servable when a rootconfig.jsonsits beside safetensors weights. Diffusers pipelines, partial downloads and bare LoRA adapters are rejected: the first belong to the image and video backends, the other two hand out an id whose load is guaranteed to fail._local_servable_entrytries GGUF first and falls back to that, and_build_indexindexes both kinds, soresolve_local_ggufnow resolves an MLX model withgguf_variant = None.local_gguf_quantsandinfo_has_local_ggufare replaced bylocal_servable_model, which returns(is_gguf, quants)from one on-disk scan.local_target_is_ggufdecides which backend serves a switch target by reading the resolved on-disk path rather than re-querying the index, so an index rebuild mid-switch cannot classify an MLX target as GGUF and strand its alias on the wrong backend.studio/backend/routes/inference.py_servable_catalog_rows(module scope, always called throughasyncio.to_thread) does the catalog pass off the event loop._openai_catalog_objectslists non-GGUF models with noquant, since their quantization is baked into the weights, and checks residency without the llama-only restriction for those rows._maybe_auto_switch_modelresolves the serving backend once intotarget_backendand routes a non-GGUF load to the inference orchestrator, recording the requested repo id as_openai_advertised_idthere._load_model_implclears that alias on every orchestrator load, mirroring the GGUF branch._orchestrator_public_model_idgivesGET /v1/models,GET /v1/models/{id}, the chat and audio response bodies and the API monitor a single id for an auto-switched model. The monitor reads it only whileactive_model_nameis set, since an unload leaves the alias behind._loaded_satisfies,_loaded_identity_satisfiesand_resident_id_is_namespacedmatch on that alias too.studio/backend/core/inference/mlx_inference.pyload_modelclampsmax_seq_lengthto 2048 when it arrives as 0, matchingcore/inference/inference.py.LoadRequestdefaults the field to 0, and an API switch with no saved per-model override sends that value straight through.Settings copy
The Settings > API description said the toggle loads a downloaded GGUF. It now says model, in all twelve locales.
autoDownloadDescriptionstill says GGUF, because automatic download remains GGUF-only.Verification
End to end on Apple Silicon (M4,
DEVICE: MLX) against the issue's own reproduction, driving the real router over HTTP with the real resolver and a real mlx-lm load ofmlx-community/Qwen3-0.6B-4bit:GET /v1/models, model never loadedloaded: false, noquantPOST /v1/chat/completionsnaming itmodelfieldmlx-community/Qwen3-0.6B-4bit/v1/modelsrow,/v1/models/{id}, monitor labelStudio UI checked by hand. 2824 backend tests pass, including
test_openai_auto_switch.py,test_openai_catalog.py,test_openai_auto_download.py,test_async_singleton_access.py,test_api_monitor.pyandtest_refactor_guard.py. New regression tests cover the resolver's rejections, the switch to an unloaded MLX model, residency of a resident non-GGUF row, id consistency across the/v1surfaces, and the monitor after an unload.ruff checkand the i18n parity check are clean.