Studio: strip the trailing template placeholder once, on the finished reply - #9101
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e22e015fca
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (isExternalRequest) { | ||
| cumulativeText = stripTrailingTemplatePlaceholder(cumulativeText); |
There was a problem hiding this comment.
Preserve seeded text when a continuation emits no text
When an external Continue run immediately finishes without a text/reasoning delta (for example, it emits only a tool call), cumulativeText still contains the prior partial seeded at line 4672. The old call site was reached only after a delta, but this unconditional final call now applies the placeholder heuristic to that pre-existing partial; if it ends in legitimate text such as ${name}, the terminal yield silently deletes it. Gate the strip on this run having produced new reply text, or otherwise avoid treating the seeded continuation prefix as newly completed output.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and it is the same defect one step in. A Continue run is seeded with the previous run's partial at the top of the handler. If that run finishes without a text or reasoning delta, having emitted only a tool call, the buffer at the end of the stream holds nothing but that partial, and a partial is the middle of a reply someone is still writing rather than the end of a finished one.
Gated on this run having appended reply text of its own. producedReplyText is set where the reply grows, next to streamedChars, and the strip reads it. A continuation that does write the rest of the answer still finishes normally, artefact and all, so the gate does not become "continuations are never trimmed".
Two tests, both checked against a broken tree rather than assumed to discriminate. Dropping the gate fails the source pin, and so does setting the flag before the loop instead of at the append. The behavioural pair asserts that a seeded partial ending in a complete placeholder comes back untouched from a run that added nothing, and separately that the strip WOULD have cut it, so the gate is what saves it rather than the input happening not to match.
Before and after, on the real thingTwo isolated Studio installs, both built fresh: BEFORE is this PR's own merge base One reply, one recorded SSE body, served to both from an out-of-page HTTP server. The arrival boundaries are the point of the script. A completed Read out of the live DOM on the same servers that were photographed:
The code block, character for character: // BEFORE
function greet(name) {
return `Hi,!`;
}
const pair = ` and`;
const nest = `}`;// AFTER
function greet(name) {
return `Hi, ${name}!`;
}
const pair = `${x} and ${y}`;
const nest = `${a${b}}`;And the two paragraphs below the fence. BEFORE renders That second paragraph is worth naming, because it is worse than one deletion. On BEFORE the arrival that completed Replaying the same 43 arrivals through the shipped strip function prints every cut the old placement makes on this reply:
273 characters sent, 159 left. Only the last row is the artefact the strip exists for. The other five destroyed 102 characters of the model's own text, on a reply that finished normally. Both halves show the leaked trailing Cases covered in the one reply: the reported reproduction, several placeholders in one line, nested placeholders, all three inside a fenced code block, an unterminated Two things that differ for reasons unrelated to the PR
Both homes had their threads zeroed before shooting, so the Recents list shows one thread on each side and a harness-caused difference cannot be read as a real one. Not tested hereThe composite is Chromium at 1500x1000. Cross-platform and cross-engine coverage is on the staging runs, not on this box. Re-shot twice: once after the continuation gate went on, and again from two fresh installs after the rebase onto |
… reply Fixes #9098. The trailing `${...}` strip ran on every SSE arrival and assigned its result back, so "ends with ${...}" was tested against every prefix of the reply rather than the reply. The one arrival whose buffer ended at a complete fragment was cut, and the reassignment made the cut permanent: in return `Hi, ${name}!` 21 chars out return `Hi,!` 13 chars The strip now runs once, after the stream has finished. The fragment it was added for in #4706 is still removed, because that one really is at the end of a completed answer. Rebased onto #9049. The watch it added still gates the scan, and now saves the whole reply from being flattened rather than one arrival's worth, so nothing on the arrival path can flatten the buffer at all. Abort keeps the buffer whole: that tail is a prefix again, so stripping it would be the same bug. `producedReplyText` is the same case one step in, for a continuation that finishes without a text or reasoning delta and so holds nothing but the partial it was seeded with.
d5177d4 to
3315ea6
Compare
|
Rebased onto #9049 moved every append through What changed in the reapplication:
Re-verified after the rebase, not carried over:
The before and after pair above was shot against the previous merge base. It is being re-shot against CI is best-effort at the moment: GitHub reports Actions at degraded performance, so anything I quote from a run is provisional and the local runs are the evidence. |
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Which assertion goes red, and on whatTwo of the three files here are tests, so this is the load-bearing part. Fourteen broken trees, run against the five test files that touch this path. Per test, not in aggregate. The trees:
The result, per assertion:
Every assertion added or rewritten here has at least one tree that turns it red. Unmutated: 31 passing, 0 failing. Two of those rows exist because the first attempt at them measured nothing:
|
Cross-platformRun on the owned staging replica rather than the org queue, against this PR's actual head The Windows red is not this PR3,628 tests, 3,615 passing, 13 failing. The 13 are None of the three files in this PR appear in that list, and my two test files load their modules through a relative specifier, not The Windows gap this does not closeThe staging Windows job sets Staging PRs are closed, never merged; they exist only to host the runs. Org CI on this PR is not quoted anywhere here. GitHub Actions is degraded and the queue has not moved, so everything above is either the staging replica or a local run. |


Fixes #9098.
The bug, verified
On any external provider, a reply containing a template literal permanently loses the interpolation and some of the text around it:
The adapter ran the trailing
${...}strip on every SSE arrival and assigned the result back:The pattern is anchored at the end. Run per arrival, "ends with
${...}" was tested against every prefix of the reply rather than against the reply. The one arrival whose buffer happened to end at...${name}matched, and the reassignment made the cut permanent, so the rest of the reply streamed in on top of the hole. That is why the output reads as ordinary text rather than as truncation.Confirmed by feeding the shipped function the prefixes an append-only stream produces. The result is not merely shorter than the input, it is not a prefix of it: characters were removed from the middle.
The bug fires only when an arrival ends exactly at the closing brace. Move that boundary one character later and the same reply comes through whole, which is why it looks intermittent from the outside.
What the strip was originally for
Added in #4706 (external provider support), with this note:
The last sentence is the mistake, and it is the whole bug. An incomplete fragment mid-stream is indeed untouched. A complete one is not, and a complete one is what ordinary JavaScript is made of. #9012 later bounded the scan to a 4096 character window for cost; it did not change when the strip runs.
The fix
The strip now runs once, on the finished reply, after the SSE loop and before the reply is turned into content. Still gated on
isExternalRequest, still the same bounded function, unchanged.Rebased onto #9049, which merged while this was open. The
placeholderWatchthat #9049 added still gates the scan, and now saves the whole reply from being flattened rather than one arrival's worth: a reply that does not end in a brace is rejected without the buffer being read at all. As a side effect nothing left on the arrival path can flatten the cons string, sothe arrival loop touches the reply only in ways that cannot flatten itno longer needs the carve-out it had for the strip.Also gated on this run having appended reply text of its own. A Continue run is seeded with the previous run's partial; if it finishes without a text or reasoning delta, having emitted only a tool call, the buffer holds nothing but that partial, and a partial is a prefix too. A continuation that does write the rest of the answer is finished normally, artefact and all.
A fragment genuinely left at the end of a completed stream is still removed, because it really is at the end. A fragment that merely sits at the end for one arrival is not, because more text follows it.
Deliberately not applied on the abort or continuation paths. Those tails are prefixes again, with more text still to come, so stripping them would be the same bug one layer up; the resumed reply is stripped when it finishes.
Placed before the
</think>close, so a fragment at the end of an unterminated reasoning block is still the end of the reply when it is tested.What was rejected
${answer}case is pinned in the tests and in the scene precisely so a future change cannot do this quietly.return \Hi, ${name}renders asreturn `Hi,` for a frame and then comes back. That trades a rare one-frame artefact in one provider's output for a frequent transient deletion of real code in everyone's. Worse on the common case.Evidence
Byte-identical recorded SSE, replayed against the merge base and this head through two isolated Studio installs. Same bytes, same chunk boundaries, same scene, different builds. Details and the composite in a comment below.
Unit level, against the shipped function and the shipped call site:
return \Hi, ${name}!``return \Hi,!``The answer is 42. ${answer})`${a${b}}``}``${x} and ${y}`` and`${HOMEPlus a randomised sweep: over 4,000 generated replies the old placement disagreed with the finished-reply answer on 1,097 of them and returned a non-prefix, that is spliced the middle out, on 1,058.
Checked against deliberately broken trees
Every new assertion, stated per mutant rather than in aggregate. Re-run after the rebase, against the current diff.
the trailing strip runs on the finished reply,the corpus is run through the placement the adapter actually ships, and #9049's ownthe arrival loop touches the reply only in ways that cannot flatten itthe adapter strips the trailing fragment through the bounded scanisExternalRequestdropped from the gate${instead of the matchthe trailing strip runs on the finished replyfails. This one initially SURVIVED: every other assertion in that test passed on a strip that computes the right answer and throws it away. Caught by checking rather than believing, and the missing assertion was addedproducedReplyTextdropped from the gate}and slices to just past the previous}, so the anchor is unreachable. 0 differences over 400,000 random inputsnpm test3,628 passing, 0 failing.npm run typecheckclean.biome checkon the adapter is diagnostic-for-diagnostic identical to the merge base; on the test file it is oneuseTopLevelRegexwarning fewer; the new test file carries only the threenoNodejsModuleserrors every test file in this suite has.Also run with
chat-adapter.ts, the strip module and both test files converted to CRLF, since the Windows staging job checks out withcore.autocrlf falseand therefore does not cover that. 16 passing, 0 failing.Overlap with open work
placeholderWatch, moving the gate rather than removing it. Itstrailing-placeholder-watchandtrailing-placeholder-reseedsuites are unchanged and still pass; its two adapter source pins are updated in place, because both described a strip that lived inside the loop.