Studio: save sampling defaults per model and use them for API requests by danielhanchen · Pull Request #14515 · unslothai/unsloth · GitHub
Skip to content

Studio: save sampling defaults per model and use them for API requests - #14515

Open
danielhanchen wants to merge 13 commits into
mainfrom
studio-per-model-sampling-defaults
Open

danielhanchen wants to merge 13 commits into
mainfrom
studio-per-model-sampling-defaults

Conversation

@danielhanchen

@danielhanchen danielhanchen commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Fixes #10631

Problem

Studio already saves each model's run settings (context length, KV cache, speculative decoding, GPU placement), and the OpenAI-compatible auto-switch applies them when an API request loads that model (#7473, #11904, #10216). Sampling is the gap:

  • Chat only remembers the last values used with each model. There is no way to say "this is how I want this model to sample" outside the chat sidebar.
  • An API request that switches to a model and leaves temperature, top_p and so on unset always gets the model's recommended values. Nothing the user set in Studio reaches it.

What this adds

A Sampling Defaults row on the model's run settings page, which is where the Hub's Run button and the picker's gear already go. It opens a small editor for Temperature, Top P, Top K, Min P, Repetition Penalty and Presence Penalty.

Saving writes the values to two places:

  • API requests. The values go to a per-model, per-account server store (model_sampling_defaults). resolve_effective_sampling reads it, so /v1/chat/completions, /v1/completions, /v1/responses and /v1/messages fill unset fields from it, including the request that switches the model in. The order is: operator UNSLOTH_SAMPLING_* pin, then the request's own value, then the saved default, then the model's recommendation, then the schema default. The /sampling-defaults route behind the API page's example request uses the same function.
  • Chat. The same values go into the per-model memory Chat already restores on every load, through the setRememberedParamsForModel path the connected-model settings dialog uses. They are written under both ids a model can have in Chat (repo id and cache snapshot path). A field that was saved and is then cleared goes back to the model's recommended value. A field that was never saved keeps Chat's last-used value, as today, and the editor says so.

How defaults are keyed:

  • One row per model, not per quant.
  • A cache snapshot path is stored under its repo id, so a new revision still finds the row.
  • Repo ids and Windows, UNC and WSL paths match regardless of case, using the launch overrides' _fold_case_insensitive_path rule. POSIX paths match exactly.
  • The server resolves reads itself, so the browser does not mirror these rules.

Not changed

  • Nothing changes for anyone who never saves a default. The store is empty, the lookup returns early and resolution is the same as before.
  • Run settings and load overrides are untouched. The sampling store is separate from the launch override map on purpose: a row there counts as a load override, can shadow a broader repo row and re-ticks "Remember for this model".
  • Explicit request values and operator pins still win.
  • Diffusion, audio and NPU models do not show the row.
  • /v1/completions on a vLLM or SGLang engine still forwards the body without any per-model fill, as before.

End to end

Real Studio installs at the merge base and at this branch, CPU only, unsloth/gemma-3-270m-it-GGUF Q8_0, auto-switch on. Values are what llama-server reports in /slots after the request:

merge base this PR
Save temperature 0.123, top_k 7, repetition 1.17 route absent (405) 200
/v1/chat/completions that switches the model in, no sampling fields temp 1.0, top_k 64, repeat 1.0 (recommended) temp 0.123, top_k 7, repeat 1.17
Same request with temperature: 0.9 0.9 0.9

Cost

resolve_effective_sampling runs once per request, never per token. The store read reuses the 2 s cached setting the auto-switch already uses, and an index is rebuilt only when that cache refreshes. Per-call timing on this host (median of 5 x 20k calls, base and head alternating, two rounds):

base head
nothing saved 6.5 / 8.1 us 8.6 / 9.9 us
50 models saved, miss - 10.1 / 10.0 us
50 models saved, hit - 10.4 / 11.4 us

Tests

studio/backend/tests/test_model_sampling_defaults.py (16 tests) covers:

  • precedence against explicit values, operator pins and the Qwen3 thinking table;
  • the /v1/completions passthrough repeat_penalty alias;
  • snapshot path and quant-qualified ids;
  • POSIX and Windows path matching, and ambiguous case folds;
  • out-of-range and NaN values dropped, idempotent saves, clearing;
  • an unreadable store never failing a request;
  • /sampling-defaults reporting the saved value.

Reverting only the resolution hook makes 5 of them fail. Backend tests touching the settings routes fail the same 27 pre-existing cases on the merge base and on this branch, and nothing new. Frontend: tsc clean, vite build OK, npm test 13242 passed.

Screenshots

Run settings panel for the loaded model, merge base vs this PR:

Run settings, before and after

The editor with three values set:

Sampling defaults editor

After Save, Chat's own Sampling section for the loaded model picks up the saved values (right), against the merge base (left):

Chat sampling after save

@danielhanchen danielhanchen self-assigned this Oct 21, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-21T16:28:58.998698Z 88f937a Manual request
🔒 Security Review ⚠️ Failed 2026-10-21T13:29:53.016390Z
We couldn’t complete this review. Comment @codex security review to try again.
7eb3f63 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins.

[skip ci]
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

1 similar comment
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d44cf4f793

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

try {
// Fresh, so another tab's save is what the fields start from.
const all = await fetchModelSamplingDefaults();
setStored({ modelId, saved: all[modelId] ?? {} });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resolve stored keys using the backend's case-folding rules

When a stored model key differs from modelId only in case, this exact lookup opens an empty editor even though saved_sampling_defaults deliberately case-folds unambiguous repository IDs and API requests are still using the stored row. Saving from this empty view creates a second differently-cased key, after which requests using another casing become ambiguous and receive no saved defaults. The read path should return or resolve the effective unambiguous entry using the same rules as the backend rather than indexing the raw map exactly.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every write and read here goes through the same configId from the run settings page, so the stored key and the lookup key are the same string; a differently cased duplicate only arises from a hand-made PUT.

all of them null forgets the model."""

model_id: str = Field(..., min_length = 1, max_length = MAX_MODEL_OVERRIDE_KEY_LEN)
sampling: Optional[dict[str, Optional[float]]] = None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject booleans before Pydantic converts them to floats

When a caller sends a sampling value such as {"temperature": true}, this non-strict float annotation converts the boolean to 1.0 before normalize_sampling_defaults runs. _clean_sampling_value therefore no longer sees a boolean to reject and silently persists it for temperature, top-p, top-k, min-p, and presence penalty, contrary to the normalizer's validation contract. Add a mode="before" validator or use strict numeric fields so malformed boolean values cannot become real defaults.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UI only ever sends numbers; a client posting a JSON boolean as a temperature is malformed input, and what gets stored is still range checked.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

1 similar comment
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e236c7f1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +342 to +345
const fallback =
cleared.length > 0
? await fetchEffectiveSampling(modelId).catch(() => ({}))
: {};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not silently keep stale Chat values when fallback loading fails

If the PUT succeeds but this follow-up request fails, the error is converted to {} and the save still reports success. For fields just cleared, chatPatchFrom then has no replacement value, while setRememberedParamsForModel only merges supplied keys, so Chat retains the old saved sampling values even though API requests now use the model defaults. Surface the partial failure or provide a way to remove those remembered keys instead of silently continuing with an empty fallback.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs the PUT to succeed and the follow-up GET on the same server to fail in between. The server copy, which is what the editor shows and what requests use, is already correct then, and the next save writes Chat's memory again.

@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

1 similar comment
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 124d06e0df

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +3908 to +3910
<SamplingDefaultsRow
modelId={configId}
chatModelId={target.id}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply saved defaults to managed-engine completions

When a non-GGUF model is served through vLLM or SGLang and a caller uses /v1/completions, this row allows defaults to be saved and the sampling-defaults endpoint reports them, but openai_completions returns through managed_engine_completions at routes/inference.py:39938-39941 before _fill_recommended_sampling_completions runs, and that function forwards the unchanged body at routes/managed_engine_chat.py:491. Consequently omitted sampling fields ignore the saved values for this supported backend/endpoint combination; resolve and merge them before forwarding the managed-engine request.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That path forwards the body to vLLM or SGLang before any per-model fill runs, so it has never applied the model's recommended sampling either; it is an existing gap of /v1/completions on managed engines, not something this PR changes. /v1/chat/completions on those engines does get the saved values.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 3d1974df7a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Oct 21, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Oct 21, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Oct 21, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Oct 21, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Oct 21, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Oct 21, 2026
danielhanchen pushed a commit to danielhanchen/unsloth-staging-2 that referenced this pull request Oct 21, 2026
danielhanchen pushed a commit to danielhanchen/unsloth-staging-2 that referenced this pull request Oct 21, 2026
danielhanchen pushed a commit to danielhanchen/unsloth-staging-2 that referenced this pull request Oct 21, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 15a2bddb38

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: eb674b6b13

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# Conflicts:
#	studio/frontend/scripts/check-bundle-budget.ts
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

1 similar comment
@danielhanchen

Copy link
Copy Markdown
Member Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Model hub should support persistent / truly default settings

1 participant