{{ message }}
fix(core): propagate resolved output limit into model request generation - #47532
Open
clopca wants to merge 1 commit into
Open
fix(core): propagate resolved output limit into model request generation#47532clopca wants to merge 1 commit into
clopca wants to merge 1 commit into
Conversation
The prepared model request only carried generation values set by hooks or context, so a model's configured limit.output never reached the provider. On Bedrock the request was sent without inferenceConfig.maxTokens and the service applied a much lower implicit budget; with adaptive thinking the whole budget could be consumed by reasoning, truncating responses at 4,096 tokens despite limit.output=128000. Project maxTokens with explicit precedence: hook/context override, model defaults, route defaults, then resolved.limit.output.
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #46595
Related: #47398 (same code site, reproduced on
@ai-sdk/openai-compatible), #29363Type of change
What does this PR do?
SessionModelRequest.preparebuilt the model request withgeneration: undefinedunless asession.contexthook set a generation value. The model's resolvedlimit.outputwas never projected into the request, so no protocol received amaxTokens:inferenceConfig.maxTokens; the service applied its implicit budget and, with adaptive thinking enabled, long-reasoning turns finished asmax_tokensat exactly 4096 output tokens, often with reasoning only and no visible text (Bedrock: configured output limit never sent, long reasoning turns truncate at 4096 tokens #46595).max_tokensfor the same reason (v2 (opencode2): max_tokens never sent for @ai-sdk/openai-compatible models, limit.output ignored, thinking turns truncate at provider default #47398).The fix always emits
generationwithmaxTokensresolved in this precedence order:session.contexthook value →model.defaults.generation.maxTokens→model.route.defaults.generation.maxTokens→resolved.limit.output. Hook and provider defaults keep winning; the catalog limit only fills the gap. Each protocol already serializedgeneration.maxTokensinto its native field (inferenceConfig.maxTokens,max_tokens,max_output_tokens,generationConfig.maxOutputTokens), so no protocol changes are needed.This is a re-submission of #46601, rebased on current
v2(the original was auto-closed by the compliance bot because the checklist section was missing). TheSessionModelRequest.context optionstest block that the original PR extended was removed upstream in #46639, so the regression test now lives in theSessionRunnerLLMscenario harness.How did you verify your code works?
packages/core/test/session-runner.test.ts("projects the resolved output limit into the request while preserving hook overrides"): assertsrequest.generationequals{ maxTokens: <limit.output> }with no hooks, and that a hook-providedmaxTokens/temperaturewin over the catalog limit. Confirmed it fails onv2without the source change and passes with it.generationwasundefined(session-compaction.test.ts,session-runner.test.ts).bun run test test/session-compaction.test.ts test/session-model-request.test.ts test/session-runner.test.tsinpackages/core: 218 pass, 0 fail.bun typecheckacross the monorepo (33 packages) clean.Screenshots / recordings
Not a UI change.
Checklist