{{ message }}
Repository navigation
Studio: apply remembered settings for MLX and safetensors models to API loads - #11904
Merged
Merged
Conversation
The OpenAI API auto-switch applies a model's server override row to MLX and safetensors loads too, but the settings panel read that row only for GGUF models, so a non-GGUF panel showed this browser's copy while API loads used another. The panel now adopts the row for every non-diffusion model, without the llama-server arguments a non-GGUF load never sends, and Load waits for the read as it does for GGUF: with Remember unchecked, Load forgets the row, so loading before it was read would delete settings the panel never showed.
"Remember for this model" only mirrored settings to the server override store for GGUF models, so an OpenAI API auto-switch loaded MLX and safetensors checkpoints with defaults (chat template, KV cache bits, context) even though the auto-switch has served them since non-GGUF weights became loadable through it. Both settings targets now share apiAutoSwitchMayLoad, which only excludes identities the resolver never serves: a LoRA adapter and a materialized Ollama link. The chat sidebar passes whether the loaded model is a LoRA adapter. A cached repo loads from its snapshot directory and /api/inference/status reports that path, while the picker keys the repo id; the backend folds those two spellings only for a quant. The sidebar and the resident config resolver now key a snapshot directory loaded without a quant by its repo id, and a record an older build saved under the path is moved to the repo id, so a bare snapshot-path row can no longer outrank the picker's row or survive its Forget.
The one-time upload of settings remembered only in this browser skipped every model that was not a GGUF quant, a .gguf file or an Ollama tag, so MLX and safetensors settings saved before the API mirror covered them kept loading with defaults through an API auto-switch. The pass now offers any remembered model except identities the resolver never keys (materialized Ollama links, bare file labels) and reruns once under a new done flag. A record an older build saved under a cached repo's snapshot path is moved to its repo id first, and one that cannot be moved is not uploaded, since a bare snapshot-path row would outrank the repo's.
Lyxot
force-pushed
the
fix/non-gguf-override-mirror
branch
from
September 25, 2026 08:29
35c2de7 to
b7f57f6
Compare
Collaborator
Author
NilayYadav
approved these changes
Sep 25, 2026
shaojun
pushed a commit
to shaojun/unsloth
that referenced
this pull request
Sep 29, 2026
…PI loads (unslothai#11904) * fix(studio): show the server settings row in the non-GGUF settings panel The OpenAI API auto-switch applies a model's server override row to MLX and safetensors loads too, but the settings panel read that row only for GGUF models, so a non-GGUF panel showed this browser's copy while API loads used another. The panel now adopts the row for every non-diffusion model, without the llama-server arguments a non-GGUF load never sends, and Load waits for the read as it does for GGUF: with Remember unchecked, Load forgets the row, so loading before it was read would delete settings the panel never showed. * fix(studio): mirror remembered settings for non-GGUF models to the API "Remember for this model" only mirrored settings to the server override store for GGUF models, so an OpenAI API auto-switch loaded MLX and safetensors checkpoints with defaults (chat template, KV cache bits, context) even though the auto-switch has served them since non-GGUF weights became loadable through it. Both settings targets now share apiAutoSwitchMayLoad, which only excludes identities the resolver never serves: a LoRA adapter and a materialized Ollama link. The chat sidebar passes whether the loaded model is a LoRA adapter. A cached repo loads from its snapshot directory and /api/inference/status reports that path, while the picker keys the repo id; the backend folds those two spellings only for a quant. The sidebar and the resident config resolver now key a snapshot directory loaded without a quant by its repo id, and a record an older build saved under the path is moved to the repo id, so a bare snapshot-path row can no longer outrank the picker's row or survive its Forget. * fix(studio): backfill browser-only settings for non-GGUF models The one-time upload of settings remembered only in this browser skipped every model that was not a GGUF quant, a .gguf file or an Ollama tag, so MLX and safetensors settings saved before the API mirror covered them kept loading with defaults through an API auto-switch. The pass now offers any remembered model except identities the resolver never keys (materialized Ollama links, bare file labels) and reruns once under a new done flag. A record an older build saved under a cached repo's snapshot path is moved to its repo id first, and one that cannot be moved is not uploaded, since a bare snapshot-path row would outrank the repo's.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11821
Problem
In Unsloth Studio, "Remember for this model" saved settings only in the browser for MLX and safetensors models. The server override store, which the OpenAI API auto-switch applies to
/v1/*loads, was updated only for GGUF targets. Since #8768 the auto-switch also loads non-GGUF weights, so an API request loaded those models with defaults and ignored the remembered context length, KV precision, or chat template.Root cause
Both settings targets (the model picker and the chat sidebar) set
apiLoadabletoisGguf && !ollamaLink. The one-time backfill filtered to GGUF ids in the same way. Neither was widened when the auto-switch gained non-GGUF support. The panel's server-row hydration also returned early for every non-GGUF model, so even a row written by another origin or a manual PUT never showed in the panel.Changes
llama_extra_args(panelOverrideRow), since a non-GGUF load never sends them. Load now waits for this read on every non-diffusion model, as it already did for GGUF. With Remember unchecked, Load forgets the server row, so a load started before the read settled would delete settings the panel had never shown. The existing release paths (settle, failure, 15 s backstop) are unchanged.apiAutoSwitchMayLoad, which excludes only what the backend resolver never serves: bare LoRA adapters and materialized Ollama links. For a cached Hugging Face repo loaded with no quant,/api/inference/statusreports the snapshot path. The sidebar now keys that model by its repo id (cachedRepoConfigId), matching the picker; otherwise a bare snapshot-path row would outrank the picker's row on the backend and survive its Forget. Browser records an older build saved under the snapshot path are adopted to the repo id lazily. A record already under the repo id wins.v3so installs that already ranv2pick them up. It still fills only absent rows and never overwrites; snapshot-path records are adopted to the repo id first, so it never uploads a snapshot-path row.The commits are ordered so the Load wait lands before non-GGUF mirroring; no commit has forget-on-load without it.
Risks and tradeoffs
resolve_override_for_loadandmodel_override_load_kwargs(is_gguf=False)already honor non-GGUF rows.Validation
Targeted frontend tests pass at each commit, and every new assertion fails when its production change is reverted:
End to end against an isolated Unsloth Studio on Apple Silicon: I saved the override row the panel now sends for
Qwen/Qwen3-0.6B(custom chat template,mlx_kv_bits=8). A/v1/chat/completionsrequest that auto-loaded the model applied both settings.Not exercised in a browser: the brief Load wait for a non-GGUF model. It's covered by source-level tests only, because the panel has no renderer in this test suite.