{{ message }}
fix(core): fail content-filtered and outputless length settlements - #47533
Open
clopca wants to merge 1 commit into
Open
fix(core): fail content-filtered and outputless length settlements#47533clopca wants to merge 1 commit into
clopca wants to merge 1 commit into
Conversation
Steps ending in content_filtered, or in max_tokens without any text or tool call, settled as succeeded with empty output. Users saw blank responses recorded as success, and the terminal filtered segment stayed in history, causing subsequent generations on the session to return empty text as well. Session generate now raises ContentPolicyError on content-filter and InvalidProviderOutputError on outputless length. The runner publisher tracks useful output (text or tool call) separately from durable output: reasoning-only length keeps its reasoning and rawFinish for diagnosis but settles as provider.invalid-output, while truncated responses with partial text remain successful.
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #46596
Related: #40146
Type of change
What does this PR do?
Two settlement paths recorded provider failures as success:
SessionGenerate.generate(backingPOST /api/session/:id/generate) returnedresponse.textregardless of the finish reason. A stream finishingcontent_filtered, ormax_tokenswith no text and no tool call, produced{"data":{"text":""}}with HTTP 200.publish-llm-event.tsfailed the assistant step oncontent-filter, but a step finishinglengthwith reasoning-only output (no text, no tool call) settled assucceeded. The user saw an empty assistant message recorded as a successful turn. On Bedrock with adaptive thinking this is the common failure mode when the output budget is spent on reasoning.Changes:
generate.ts: raiseAIError(ContentPolicyError)oncontent-filterandAIError(InvalidProviderOutputError)onlengthwith empty text and no tool calls, instead of returning empty text.publish-llm-event.ts: trackusefulOutput(any non-empty text delta/end or a tool call) separately fromoutputStarted. Onfinish-stepwithlengthand no useful output, fail the assistant withprovider.invalid-output. Reasoning parts andrawFinishare still recorded, so the truncated reasoning remains available for diagnosis. Truncated responses that did produce partial text or a tool call keep settling as before, so no valid output is lost.This is a re-submission of #46602, rebased on current
v2(the original was auto-closed by the compliance bot because the checklist section was missing).generate-node.tswas folded intogenerate.tsupstream in #46639, and #46937 removed thestepStreamed/stepFailedflags; the change is adapted to both.How did you verify your code works?
packages/core/test/session-generate.test.ts: new case "fails transient generation when the provider filters or exhausts output before answering" covering bothcontent-filterand reasoning-onlylength, asserting the typedAIErrorreason and that durable session state is untouched.packages/core/test/session-runner-tool-events.test.ts: new case asserting a reasoning-onlylengthstep settles asprovider.invalid-outputwhile keeping the reasoning part, and thatlengthwith partial text still succeeds.bun run test test/session-generate.test.ts test/session-runner-tool-events.test.tsinpackages/core: 29 pass, 0 fail.bun typecheckacross the monorepo (33 packages) clean.Screenshots / recordings
Not a UI change.
Checklist