Studio: strip the trailing template placeholder once, on the finished reply by danielhanchen · Pull Request #9101 · unslothai/unsloth · GitHub
Skip to content

Studio: strip the trailing template placeholder once, on the finished reply - #9101

Merged
danielhanchen merged 1 commit into
mainfrom
fix-trailing-placeholder-prefix
Aug 17, 2026
Merged

Studio: strip the trailing template placeholder once, on the finished reply#9101
danielhanchen merged 1 commit into
mainfrom
fix-trailing-placeholder-prefix

Conversation

@danielhanchen

@danielhanchen danielhanchen commented Aug 17, 2026

Copy link
Copy Markdown
Member

Fixes #9098.

The bug, verified

On any external provider, a reply containing a template literal permanently loses the interpolation and some of the text around it:

in     return `Hi, ${name}!`      21 chars
out    return `Hi,!`             13 chars

The adapter ran the trailing ${...} strip on every SSE arrival and assigned the result back:

if (isExternalRequest) {
  cumulativeText = stripTrailingTemplatePlaceholder(cumulativeText);
}

The pattern is anchored at the end. Run per arrival, "ends with ${...}" was tested against every prefix of the reply rather than against the reply. The one arrival whose buffer happened to end at ...${name} matched, and the reassignment made the cut permanent, so the rest of the reply streamed in on top of the hole. That is why the output reads as ordinary text rather than as truncation.

Confirmed by feeding the shipped function the prefixes an append-only stream produces. The result is not merely shorter than the input, it is not a prefix of it: characters were removed from the middle.

The bug fires only when an arrival ends exactly at the closing brace. Move that boundary one character later and the same reply comes through whole, which is why it looks intermittent from the outside.

What the strip was originally for

Added in #4706 (external provider support), with this note:

Mistral's magistral occasionally emits a trailing template-literal artifact (e.g. "${response}") at the end of an otherwise complete answer. It is never part of a real reply, so strip a trailing ${...} token from external provider streams. The regex anchors to end-of-string and is idempotent, fragments mid-stream (e.g. "${re") leave the string untouched and only collapse once the closing brace arrives.

The last sentence is the mistake, and it is the whole bug. An incomplete fragment mid-stream is indeed untouched. A complete one is not, and a complete one is what ordinary JavaScript is made of. #9012 later bounded the scan to a 4096 character window for cost; it did not change when the strip runs.

The fix

The strip now runs once, on the finished reply, after the SSE loop and before the reply is turned into content. Still gated on isExternalRequest, still the same bounded function, unchanged.

Rebased onto #9049, which merged while this was open. The placeholderWatch that #9049 added still gates the scan, and now saves the whole reply from being flattened rather than one arrival's worth: a reply that does not end in a brace is rejected without the buffer being read at all. As a side effect nothing left on the arrival path can flatten the cons string, so the arrival loop touches the reply only in ways that cannot flatten it no longer needs the carve-out it had for the strip.

Also gated on this run having appended reply text of its own. A Continue run is seeded with the previous run's partial; if it finishes without a text or reasoning delta, having emitted only a tool call, the buffer holds nothing but that partial, and a partial is a prefix too. A continuation that does write the rest of the answer is finished normally, artefact and all.

A fragment genuinely left at the end of a completed stream is still removed, because it really is at the end. A fragment that merely sits at the end for one arrival is not, because more text follows it.

Deliberately not applied on the abort or continuation paths. Those tails are prefixes again, with more text still to come, so stripping them would be the same bug one layer up; the resumed reply is stripped when it finishes.

Placed before the </think> close, so a fragment at the end of an unterminated reasoning block is still the end of the reply when it is tested.

What was rejected

  • Delete the strip. Reintroduces the artefact studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) #4706 added it for. The ${answer} case is pinned in the tests and in the scene precisely so a future change cannot do this quietly.
  • Strip non-destructively into a display copy on every arrival, keeping the buffer whole. No data is lost, and it is re-entrant, but it makes every template literal flicker: return \Hi, ${name}renders asreturn `Hi,` for a frame and then comes back. That trades a rare one-frame artefact in one provider's output for a frequent transient deletion of real code in everyone's. Worse on the common case.
  • A heuristic for "the stream has probably stopped", for instance a quiet period after the last arrival. Timing-dependent, untestable, and wrong under backpressure.

Evidence

Byte-identical recorded SSE, replayed against the merge base and this head through two isolated Studio installs. Same bytes, same chunk boundaries, same scene, different builds. Details and the composite in a comment below.

Unit level, against the shipped function and the shipped call site:

case before after
return \Hi, ${name}!`` return \Hi,!`` unchanged
fragment genuinely at the end (The answer is 42. ${answer}) stripped stripped
nested, `${a${b}}` `}` unchanged
several, `${x} and ${y}` ` and` unchanged
split across arrivals, boundary at the brace text lost unchanged
unterminated ${HOME unchanged unchanged
inside a fenced code block text lost unchanged
CRLF line endings text lost unchanged
literal in the body plus a leaked fragment on the end both lost body kept, fragment removed

Plus a randomised sweep: over 4,000 generated replies the old placement disagreed with the finished-reply answer on 1,097 of them and returned a non-prefix, that is spliced the middle out, on 1,058.

Checked against deliberately broken trees

Every new assertion, stated per mutant rather than in aggregate. Re-run after the rebase, against the current diff.

broken tree result
B1 revert: strip back inside the SSE loop 4 fail, including the trailing strip runs on the finished reply, the corpus is run through the placement the adapter actually ships, and #9049's own the arrival loop touches the reply only in ways that cannot flatten it
B2 strip removed altogether 4 fail, including the existing the adapter strips the trailing fragment through the bounded scan
B3 isExternalRequest dropped from the gate 2 fail
B5 strip cuts from the last ${ instead of the match 11 fail, including 6 pre-existing ones
B6 the strip runs but its result is never assigned back the trailing strip runs on the finished reply fails. This one initially SURVIVED: every other assertion in that test passed on a strip that computes the right answer and throws it away. Caught by checking rather than believing, and the missing assertion was added
B7 strip moved after the finished reply is built 1 fails
B8 producedReplyText dropped from the gate 2 fail
B9 flag set before the loop instead of at the append 1 fails
B10 trackers repaired even when nothing was cut 1 fails
end anchor removed from the bounded pattern everything passes, and that is correct: it is an equivalent mutant, not a gap. The scan already requires the last non-whitespace character to be } and slices to just past the previous }, so the anchor is unreachable. 0 differences over 400,000 random inputs
unmutated 31 pass, 0 fail across the five files that touch this path

npm test 3,628 passing, 0 failing. npm run typecheck clean. biome check on the adapter is diagnostic-for-diagnostic identical to the merge base; on the test file it is one useTopLevelRegex warning fewer; the new test file carries only the three noNodejsModules errors every test file in this suite has.

Also run with chat-adapter.ts, the strip module and both test files converted to CRLF, since the Windows staging job checks out with core.autocrlf false and therefore does not cover that. 16 passing, 0 failing.

Overlap with open work

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e22e015fca

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +6438 to +6439
if (isExternalRequest) {
cumulativeText = stripTrailingTemplatePlaceholder(cumulativeText);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve seeded text when a continuation emits no text

When an external Continue run immediately finishes without a text/reasoning delta (for example, it emits only a tool call), cumulativeText still contains the prior partial seeded at line 4672. The old call site was reached only after a delta, but this unconditional final call now applies the placeholder heuristic to that pre-existing partial; if it ends in legitimate text such as ${name}, the terminal yield silently deletes it. Gate the strip on this run having produced new reply text, or otherwise avoid treating the seeded continuation prefix as newly completed output.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and it is the same defect one step in. A Continue run is seeded with the previous run's partial at the top of the handler. If that run finishes without a text or reasoning delta, having emitted only a tool call, the buffer at the end of the stream holds nothing but that partial, and a partial is the middle of a reply someone is still writing rather than the end of a finished one.

Gated on this run having appended reply text of its own. producedReplyText is set where the reply grows, next to streamedChars, and the strip reads it. A continuation that does write the rest of the answer still finishes normally, artefact and all, so the gate does not become "continuations are never trimmed".

Two tests, both checked against a broken tree rather than assumed to discriminate. Dropping the gate fails the source pin, and so does setting the flag before the loop instead of at the append. The behavioural pair asserts that a seeded partial ending in a complete placeholder comes back untouched from a run that added nothing, and separately that the strip WOULD have cut it, so the gate is what saves it rather than the input happening not to match.

@danielhanchen

danielhanchen commented Aug 17, 2026

Copy link
Copy Markdown
Member Author

Before and after, on the real thing

Two isolated Studio installs, both built fresh: BEFORE is this PR's own merge base 4f0d691cf, which is current main and includes #9038 and #9049, AFTER is head 3315ea60d. Each build serves its own frontend bundle, so the two halves are different code, not one Studio photographed twice.

One reply, one recorded SSE body, served to both from an out-of-page HTTP server. sent_reply_sha256 cb3c0313 and received_sse_bytes 6501 on both sides, so the two halves received the same 6,501 bytes in the same 43 arrivals. Everything downstream of the socket is the real thing: the real chat adapter, assistant-ui, React, Streamdown and Shiki.

The arrival boundaries are the point of the script. A completed ${...} is only cut if an arrival ends at its closing brace, so the splitter cuts after every }. Chunked any other way this same reply comes through whole, which is why the bug reads as intermittent from the outside.

before and after

Read out of the live DOM on the same servers that were photographed:

fact BEFORE AFTER
text_content_chars 142 240
text_content_sha256 41e61a40 e4fa5e2d
inner_html_normalised_sha256 5956a5fb 15f9c204
inner_html_normalised_chars 9,262 12,198
cases_kept 0 of 5 5 of 5
case_chars_missing 119 0
leaked_fragment_present false false
highlighted_token_count 28 47
part_count 1 1
page_errors none none

The code block, character for character:

// BEFORE
function greet(name) {
  return `Hi,!`;
}
const pair = ` and`;
const nest = `}`;
// AFTER
function greet(name) {
  return `Hi, ${name}!`;
}
const pair = `${x} and ${y}`;
const nest = `${a${b}}`;

And the two paragraphs below the fence. BEFORE renders On a shell the same text is written ; and nothing else. AFTER renders On a shell the same text is written ${HOME with no closing brace. and Windows note, sent with CRLF: const t = ${q};.

That second paragraph is worth naming, because it is worse than one deletion. On BEFORE the arrival that completed ${q} fired the strip, and [^}]* reaches back past the unterminated ${HOME to the previous }, so a single arrival took the CRLF line, the shell line and the paragraph break with it.

Replaying the same 43 arrivals through the shipped strip function prints every cut the old placement makes on this reply:

buffer length when it fired removed what went
80 8 ${name}
96 4 ${x}
101 5 ${y}
120 7 ${a${b}
235 78 ${HOME with no closing brace. + the blank line + Windows note, sent with CRLF: + const t = \${q}`
171 12 ${answer} and the CRLF in front of it

273 characters sent, 159 left. Only the last row is the artefact the strip exists for. The other five destroyed 102 characters of the model's own text, on a reply that finished normally.

Both halves show the leaked trailing ${answer} fragment absent, which is the check that a fix by deletion would fail. The scene raises rather than shooting if it is present, if the fence renders with zero Shiki tokens, or if the completions request was never intercepted.

Cases covered in the one reply: the reported reproduction, several placeholders in one line, nested placeholders, all three inside a fenced code block, an unterminated ${ with no closing brace, CRLF line endings (#9088 landed on this path), and a fragment genuinely at the end of the finished reply.

Two things that differ for reasons unrelated to the PR

  • The footer duration reads 3.92s on BEFORE and 3.93s on AFTER. That is wall clock on a scripted stream, not content.
  • highlighted_token_count differs because AFTER has more code on screen to highlight. It is asserted > 0 on both, not asserted equal.

Both homes had their threads zeroed before shooting, so the Recents list shows one thread on each side and a harness-caused difference cannot be read as a real one.

Not tested here

The composite is Chromium at 1500x1000. Cross-platform and cross-engine coverage is on the staging runs, not on this box.

Re-shot twice: once after the continuation gate went on, and again from two fresh installs after the rebase onto 4f0d691cf, since #9038 and #9049 merged and the old merge base no longer says anything about this PR. Every fact came back identical across all three runs, on both sides: same text_content_sha256, same inner_html_normalised_sha256, same cases_kept, same received_sse_bytes. Neither merge changes what the strip does or where it used to run.

danielhanchen added a commit to shimmyshimmer/unsloth-staging-4 that referenced this pull request Aug 17, 2026
… reply

Fixes #9098.

The trailing `${...}` strip ran on every SSE arrival and assigned its
result back, so "ends with ${...}" was tested against every prefix of the
reply rather than the reply. The one arrival whose buffer ended at a
complete fragment was cut, and the reassignment made the cut permanent:

    in     return `Hi, ${name}!`      21 chars
    out    return `Hi,!`             13 chars

The strip now runs once, after the stream has finished. The fragment it
was added for in #4706 is still removed, because that one really is at
the end of a completed answer.

Rebased onto #9049. The watch it added still gates the scan, and now
saves the whole reply from being flattened rather than one arrival's
worth, so nothing on the arrival path can flatten the buffer at all.

Abort keeps the buffer whole: that tail is a prefix again, so stripping
it would be the same bug. `producedReplyText` is the same case one step
in, for a continuation that finishes without a text or reasoning delta
and so holds nothing but the partial it was seeded with.
@danielhanchen
danielhanchen force-pushed the fix-trailing-placeholder-prefix branch from d5177d4 to 3315ea6 Compare August 17, 2026 13:44
@danielhanchen

Copy link
Copy Markdown
Member Author

Rebased onto 4f0d691cf, since #9049 and #9038 merged while this was open.

#9049 moved every append through appendCumulative and gated the strip on a placeholderWatch that answers from the deltas, so the branch point this was written against no longer exists. The fix is reapplied in terms of the new code rather than merged into it.

What changed in the reapplication:

  • The watch stays. It now gates the one scan on the finished reply instead of a scan per arrival, which is strictly more of what it was written for: a reply that does not end in a brace is rejected without the buffer being read at all, so the whole reply is saved from being flattened rather than one arrival's worth.
  • A cut still calls thinkTags.retract and placeholderWatch.retract, and still only when it cuts. The incremental parse repairs itself from the length mismatch, which is the path createSegmentedAssistantText documents for a removed suffix.
  • Two of Studio: stop the streamed reply being flattened on every arrival #9049's adapter source pins described a strip that lives inside the loop, so both are updated in place. the arrival loop touches the reply only in ways that cannot flatten it loses the carve-out it had for the strip: with the strip out of the loop, nothing left on the arrival path can flatten the buffer, so the assertion is now that the loop mentions cumulativeText only as .length, with no exceptions. Its watch and reseed suites are untouched and still pass.

Re-verified after the rebase, not carried over:

  • 3,628 passing, 0 failing. npm run typecheck clean.
  • Ten broken trees, each one named in the description with which assertions go red.
  • One of them, a strip whose result is never assigned back, initially survived: every other assertion in the new test passed against a strip that computes the right answer and throws it away. The missing assertion is now there and that tree fails. Worth stating rather than quietly fixing, because it is exactly the shape of a test that looks thorough and measures nothing.
  • Also run with the adapter, the strip module and both test files converted to CRLF, since the Windows staging job checks out with core.autocrlf false and so does not cover it.

The before and after pair above was shot against the previous merge base. It is being re-shot against 4f0d691cf and I will replace it; the diagnosis and the local checks are unaffected, since neither merge changes what the strip does or where it ran.

CI is best-effort at the moment: GitHub reports Actions at degraded performance, so anything I quote from a run is provisional and the local runs are the evidence.

@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 3315ea60da

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@danielhanchen

Copy link
Copy Markdown
Member Author

Which assertion goes red, and on what

Two of the three files here are tests, so this is the load-bearing part. Fourteen broken trees, run against the five test files that touch this path. Per test, not in aggregate.

The trees:

what was broken
B1 revert: the strip goes back inside the SSE loop, on every arrival
B2 the strip is removed altogether
B3 isExternalRequest dropped from the gate, so local replies are trimmed too
B4 the $ end anchor removed from the pattern
B5 the strip cuts from the last ${ instead of from the match
B6 the strip runs, but its result is never assigned back
B7 the strip moved after the finished reply has been built
B8 producedReplyText dropped from the gate
B9 the flag set before the loop instead of at the append
B10 the trackers repaired even on a call that cut nothing
B11 the strip made stateful, so the answer depends on how many times it ran
B12 the corpus coarsened so no arrival ends at a closing brace
B13 the strip splices the fragment out in place instead of cutting the tail
B14 the strip made a no-op

The result, per assertion:

test trees that turn it red
the trailing strip runs on the finished reply, not on every arrival B1 B2 B3 B6 B7 B8 B9 B10
the corpus is run through the placement the adapter actually ships B1 B2 B5 B11 B13 B14
every case survives the stream intact B5 B11 B13 B14
the cases that discriminate really did lose text before B5 B11 B12 B13 B14
the reported reproduction, character for character B5 B11 B13 B14
the finished reply does not depend on how the stream was split B11
chunk-independence is a claim the old placement fails B11 B14
the finished reply is always a prefix of what the model sent B13
the old placement spliced the middle out, which is why text vanished B5 B11 B13 B14
randomised replies keep the two placements apart B11 B13 B14
a continuation that adds nothing leaves the seeded partial alone B5 B13 B14
a continuation that does add text is finished normally B5 B13 B14
the arrival loop touches the reply only in ways that cannot flatten it (#9049's, rewritten here) B1
nothing on the arrival path is handed the accumulated reply (#9049's, rewritten here) B1 B2 B3 B8
the adapter strips the trailing fragment through the bounded scan (existing, untouched) B2

Every assertion added or rewritten here has at least one tree that turns it red. Unmutated: 31 passing, 0 failing.

Two of those rows exist because the first attempt at them measured nothing:

  • B6 initially survived. Every other assertion in the trailing strip runs on the finished reply passed against a strip that computes the right answer and throws it away, because the gate regex it matched stopped at const stripped =. The assertion that the result is assigned back was missing and is now there.
  • B4 kills nothing, and that is the right answer, not a gap. The bounded scan already requires the last non-whitespace character to be } and slices to just past the previous }, and [^}]* cannot cross a brace, so at most one } is ever in the region the pattern is run over. The end anchor is unreachable. Checked rather than argued: 0 differences between the two functions over 400,000 randomly generated strings drawn from an alphabet of a, space, backtick, $, {, }, newline, carriage return, ! and ${.

the finished reply does not depend on how the stream was split is the weakest row, red on one tree only. It is a self-consistency property of the strip rather than a claim about the adapter, and B11 is what it actually guards: a strip whose answer depends on its call history. Said plainly rather than left to look stronger than it is.

danielhanchen added a commit to shimmyshimmer/unsloth-staging-4 that referenced this pull request Aug 17, 2026
@danielhanchen

Copy link
Copy Markdown
Member Author

Cross-platform

Run on the owned staging replica rather than the org queue, against this PR's actual head 3315ea60d merged onto 4f0d691cf.

job result
ubuntu-latest success
frontend ubuntu-latest (typecheck, build, bundle budget, npm test) success
macos-14 success
frontend macos-14 success
windows-latest success
frontend windows-latest 13 failing, all pre-existing

The Windows red is not this PR

3,628 tests, 3,615 passing, 13 failing. The 13 are tests/copy-to-clipboard.test.ts and tests/microphone-permission-reset.test.ts, neither of which this PR touches, and both of which fail the same way on plain main. #8980 is open against exactly this and measures the same 13 on main with 0 on its own branch, from an ERR_UNSUPPORTED_ESM_URL_SCHEME on a fileURLToPath import specifier that Node rejects on Windows. Its own staging run of frontend windows-latest is green, which is the control.

None of the three files in this PR appear in that list, and my two test files load their modules through a relative specifier, not fileURLToPath.

The Windows gap this does not close

The staging Windows job sets core.autocrlf false, so it checks out with LF and never exercises a CRLF working tree. Both new source pins read chat-adapter.ts off disk and match against it, so a CRLF checkout is a real risk for them specifically. Covered locally instead: chat-adapter.ts, trailing-template-placeholder.ts and both test files converted to CRLF, 16 passing and 0 failing.

Staging PRs are closed, never merged; they exist only to host the runs.

Org CI on this PR is not quoted anywhere here. GitHub Actions is degraded and the queue has not moved, so everything above is either the staging replica or a local run.

@danielhanchen
danielhanchen merged commit 2d8e0a0 into main Aug 17, 2026
36 of 37 checks passed
@danielhanchen
danielhanchen deleted the fix-trailing-placeholder-prefix branch August 17, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat: an external-provider reply containing a template literal loses text mid-stream

1 participant