studio: allow a per-connection max tokens limit on every provider - #8944
Conversation
Studio pinned Max Tokens at the 32,768 fallback for any external model with no documented per-model cap, and the only escape hatch was the generic Custom connection type. A MiniMax M3 connection through OpenRouter, or any of the hundreds of router models the capability table does not list, had no way to raise it. The per-connection Max Tokens limit now applies to every connection type except ChatGPT subscriptions, whose routing, model list and output cap are all fixed. A documented per-model cap still wins over the override: one connection fronts many models on a router, so a limit set for a 256k-output model must not raise the slider past what a smaller model on the same connection accepts. Fixes unslothai#8910
Review of the previous commit turned up two ways the override resolved wrongly. A documented per-model cap replaced the override outright, so a limit set below a published cap did nothing: typing 8192 on an OpenAI connection saved cleanly and left the slider and the outgoing max_tokens at 128000. getExternalMaxOutputTokens now takes Math.min of the two, so an override lowers a documented cap and still cannot raise past one. The other end had no guard at all. EXTERNAL_MIN_OUTPUT_TOKENS_BY_PROVIDER floors Kimi at 16000 so a thinking model's reasoning_content and answer both fit, and before this the override could sit under it: an 8000 limit on a Kimi connection handed the Max Tokens slider min 16000 against max 8000, and chat-adapter sent 8000 on every request. The resolved ceiling is now floored at the provider minimum, and the connections dialog rejects a sub-floor value rather than storing one that gets silently raised. Also here: sync-external-providers.ts and the localStorage load path in external-providers.ts both deliver the override to the UI and neither was pinned, so re-gating either on the Custom provider type made the feature a no-op with the suite green. The Codex rejection test asserted only status 400, which a Codex create returns from the auth contract too, and passed with the guard deleted.
Both still said the override only replaces the 32,768-token fallback. It also lowers a documented per-model cap now.
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b04f5e4628
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| exist is a no-op. | ||
| """ | ||
| if field_was_set and value is not None and provider_type != "custom": | ||
| if field_was_set and value is not None and provider_type == "openai_codex": |
There was a problem hiding this comment.
For a Kimi connection, a POST or PUT with max_output_tokens from 64 through 15,999 now passes both the Pydantic ge=64 constraint and this Codex-only check, so that value is persisted as the connection's maximum. On the frontend, however, getExternalMaxOutputTokens raises the effective cap to Kimi's 16,000-token minimum, and the adapter consequently sends up to 16,000 tokens, exceeding the configured maximum. Reject Kimi values below its provider floor in the backend as the dialog already does, rather than accepting a cap that cannot be enforced.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not reachable from the product: the connections dialog floors the value at getExternalMinOutputTokens(providerType), so a Kimi row cannot be given 64-15,999 through Studio. Only a hand-written REST call gets there, and the floor then keeps that connection safe rather than honouring a cap that truncates reasoning_content mid-stream. Enforcing it server-side would need EXTERNAL_MIN_OUTPUT_TOKENS_BY_PROVIDER copied into Python, a second source of truth for every provider floor that is free to drift from the frontend table that actually resolves the ceiling.
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Before/after evidence for this one, from two isolated Studio installs: BEFORE is the merge base
BEFORE runs Connection > API key > the Studio-tools note. AFTER inserts the Max Tokens limit row between the API key field and that note, with the 32,768 placeholder, the help text describing how the override interacts with a model's documented cap, and the amber warning about an upstream that will not accept the value. The Connection select reads OpenRouter on both halves, so the difference is the field and not a missed pick. The numeric half, read from the same two servers that were photographed.
So the 262,144 limit #8910 asks for is both settable and durable after this change, and the connection it is stored against is the one the resolver reads it from. |
…ts (unslothai#8944) parseMaxOutputTokens throws below the provider floor, and the edit form seeds its draft from the stored value, so a row holding a value under that floor failed every unrelated save. Seed the draft at the floor instead, which is what getExternalMaxOutputTokens already resolves such a value to. Also covers the ChatGPT subscription rejection on create, which was asserted on the update path only.
|
Ran the before/after simulations on this one, plus a compatibility sweep, and pushed two small changes in 737521f. What happened before, what happens nowBefore: the per-connection Max Tokens limit existed only on generic Custom connections. Every other connection resolved its ceiling from the documented per-model table, or 32,768 when the model was not in it. MiniMax M3 through OpenRouter is not in it, so the slider sat at 32,768 at its maximum and generation stopped there, which is #8910. After: any connection except a ChatGPT subscription can carry the limit. A documented cap now bounds the override rather than being replaced by it, so an OpenRouter connection set to 262,144 gives MiniMax M3 262,144 while The issue is real, not theoretical: the reporter's screenshot shows a generation cut off at the ceiling, and I reproduced the resolver behaviour on both sides. Backwards compatibilitySwept
Every single change requires the user to have explicitly set a limit. Nobody who upgrades and changes nothing sees a different ceiling. In particular the new unconditional floor is a no-op today, since the only floor is Kimi's 16,000 and Kimi has no documented cap, so the unfloored result was already 32,768. The write contract, hammered over 240 create/update/clear cases against both live backends:
Old installs: started from the original Platforms and browsersNo hardware path is involved: this is which integer goes in What does vary by platform is the browser, so I drove the dialog in all three engines behind the four browsers named (Chromium covers Chrome and Edge, Gecko covers Firefox, WebKit covers Safari):
That first row is the one worth having: On the provider floor asymmetrySeveral passes flagged that the backend accepts a Kimi value below the 16,000 floor the frontend enforces, and suggested enforcing it server side. I traced it end to end before agreeing, and I do not think the backend should change:
There was one real consequence though, which is what 737521f fixes. What changed in 737521f
The second change covers the ChatGPT subscription rejection on create, which was asserted on the update path only. Both are mutation checked: reverting the dialog line fails the new frontend assertion, and neutering the Codex condition fails the new backend one. Gates after the change: frontend typecheck clean, the three affected test files 22 passed, |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
2 participants
Fixes #8910
Studio pinned Max Tokens at 32,768 for any connected model whose output cap is not in
EXTERNAL_MAX_OUTPUT_TOKENS_BY_MODEL, and the only way to raise it was the per-connection MaxTokens limit, which
supportsCustomMaxOutputTokensoffered on generic Custom connections alone.MiniMax M3 through OpenRouter hit that, along with most of the router's catalog: the slider sat at
its maximum and the model kept stopping at "reached the Max Tokens limit".
The override now applies to every connection type except ChatGPT subscriptions, whose routing,
model list and output cap are all fixed, so a value stored against one would never be read.
How the ceiling resolves
getExternalMaxOutputTokenstakes the documented per-model cap and the connection's own limit andreturns:
Math.min(documented, override). The override lowers a published cap,which a gateway or spend policy below it needs, and never raises past one. That matters because
a single OpenRouter connection fronts models of every size, so a 262,144 limit set for MiniMax
must not raise the slider for
openai/gpt-5.3.getExternalMinOutputTokens. Kimi's floor is 16,000 so a thinking model'sreasoning_contentand answer both fit; without the floor an 8,000 limit handedParamSlidermin 16,000 against max 8,000 and
chat-adaptersent 8,000 on every request.The connections dialog rejects a value below that floor rather than storing one the resolver
raises anyway.
Code paths
studio/frontend/src/features/chat/provider-capabilities.ts:getExternalMaxOutputTokensrestructured around a new
_documentedMaxOutputTokenslookup.studio/frontend/src/features/chat/external-providers.ts:normalizeCustomMaxOutputTokens->normalizeProviderMaxOutputTokens(theproviderTypegate is gone),supportsCustomMaxOutputTokens->supportsProviderMaxOutputTokens,CUSTOM_MAX_OUTPUT_TOKENS_MIN->PROVIDER_MAX_OUTPUT_TOKENS_MIN.studio/frontend/src/features/chat/sync-external-providers.ts: carriesconfig.max_output_tokensfor every connection instead of Custom only.studio/frontend/src/features/chat/chat-providers-dialog.tsx: the Max Tokens limit row rendersfor every non-Codex connection, and
parseMaxOutputTokensfloors at the provider minimum.studio/backend/routes/providers.py:_validate_max_output_tokens_contractrefuses a non-nulloverride only for
openai_codex. An explicit null is still accepted everywhere, since a blankfield serialises as null rather than as an omission.
No schema change.
llm_providers.max_output_tokensalready exists from #8512, and no non-Customrow can hold a stale value because the old contract rejected one.
Checks
npm test2703 passed, 0 failednpm run typecheckcleanpytest tests/test_provider_max_output_tokens_contract.py tests/test_credential_routes.py59passed
ruff checkandscripts/run_ruff_format.pycleanOpenRouter connection with a 262,144 limit, read it back through
GET /api/providers, resolvedthe slider ceiling through the real
syncExternalProvidersFromBackendandgetExternalMaxOutputTokens(262,144 forminimax/minimax-m3, 16,384 foropenai/gpt-5.3),sent a chat completion and confirmed
max_tokens: 262144reached the upstream, then cleared theoverride and confirmed the connection returned to the default. The same run against
mainfailsat connection creation with
Max Tokens limit can only be overridden for generic Custom providers.