Studio: pin what the trailing-placeholder window does with nested placeholders by danielhanchen · Pull Request #9091 · unslothai/unsloth · GitHub
Skip to content

Studio: pin what the trailing-placeholder window does with nested placeholders - #9091

Merged
danielhanchen merged 5 commits into
mainfrom
nested-placeholder-pin
Aug 18, 2026
Merged

Studio: pin what the trailing-placeholder window does with nested placeholders#9091
danielhanchen merged 5 commits into
mainfrom
nested-placeholder-pin

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

Follow-up to #9012, which is merged. One test, no behaviour change.

Why

#9012 bounded the trailing ${...} strip to a 4096 character window. The guarantee it makes is about what is removed: the bounded scan never removes more than the unbounded pattern would, and always returns a prefix of the input. "The oversized fragment is left whole" is the usual consequence of that, but it is not the rule, and nested placeholders are where the two readings come apart.

For "answer ${" + "a".repeat(4196) + "${nested}" the outer ${ sits further back than the window, so the bounded scan does not see it as an opener, and the trailing ${nested} is an ordinary complete fragment at the end of the buffer. It gets stripped and the outer text stays:

kept removed
unbounded pattern 6 (answer) 4,208
bounded scan 4,205 9 (${nested})

That is the correct outcome, and it is the direction that matters: on this input the unbounded pattern deletes 4,208 characters of model-written text and the bounded scan deletes 9. The module comment describes it, but nothing pinned it, so a later change to match the looser "left whole" reading would have passed the suite.

The test

Asserts that exactly ${nested} is removed, that the two removal counts are 4,208 and 9, and that the result stays a prefix of the input.

Checked against deliberately broken trees rather than assumed to discriminate:

broken tree result
window widened to 100,000, so the outer opener comes back into range and the whole reply is stripped test fails
decline to strip whenever an opener predates the window, which is the change a review of #9012 asked for test fails
unmutated passes

Worth recording that the first mutant I wrote for the second row used lastIndexOf("${"), which finds the inner opener and therefore never triggered, so the mutant passed and briefly looked like a gap in the test. The row above uses indexOf.

npm test 3,588 passing, 0 failing. npm run typecheck clean. biome check on the file is unchanged at 4 diagnostics, all pre-existing.

The module comment says the window's guarantee is about what is removed
rather than about leaving an oversized fragment whole, and gives nested
placeholders as the case where those two readings come apart. Nothing pinned
it, so changing the behaviour to match the looser reading would have gone
unnoticed.

The test kills both directions: widening the window so the outer opener comes
back into range, and declining to strip whenever an opener predates the
window, which is the change a review of #9012 asked for.
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: dd400ba8ed

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 8dd8ba68ba

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: f64e498a54

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@danielhanchen

Copy link
Copy Markdown
Member Author

Independent merge-worthiness check on this branch, since a characterization test is only
worth its line count if something can actually make it red that nothing else catches.

The two mutants in the description are already covered

Both rows in the table above do turn the new test red, but neither is a gap the branch
closes. Running each mutant against the merge base, restricted to the placeholder test
files:

mutant merge base this branch
window widened to 100,000 red, the trailing placeholder strip is linear in the reply length red
decline when an opener predates the window, indexOf red, same linearity test red
same idea, O(1), keyed on previousBrace === -1 red, bounded strip matches the unbounded pattern behind a long reply red

The first two are caught for cost, not for meaning: both add an unbounded backward scan,
so the linearity test sees them before the semantics ever come up.

The mutant that this test, and only this test, catches

The "leave the oversized fragment whole" reading is implementable without giving up the
performance contract: look back a bounded distance for an opener that predates the
window, and decline if one is there. It stays O(1) in reply length, so the linearity test
cannot see it, and the long-reply cases carry no out-of-window ${, so those cannot
either.

if (from > 0) {
  const back = text.slice(Math.max(0, from - 2 * window), from);
  if (back.includes("${")) {
    return text;
  }
}
const tail = text.slice(from);

Full frontend suite, that mutant applied:

tree tests failures
suite without the new test 3,671 0
suite with the new test 3,672 1, the new test

So it is the only guard in the suite against that change. That is the row the description
should be making.

Per assertion

assertion goes red on
exact removal, inner placeholder only window widened, both decline variants, bounded look-back, no-op strip, over-trim by one, unbounded revert
removal count 9 same set
result stays a prefix of the input returning the suffix instead of the prefix
unbounded removal count 4,208 nothing

The 4,208 assertion reads only the reference regex defined in the test file, so no change
to the module can move it, and weakening the reference regex is caught by five older tests
in the same file first. It documents the comparison rather than guarding anything. Worth
keeping for the reader, not worth counting as coverage.

Why the input class is not an enumeration artefact

The shape is reachable from ordinary prose. A reply that mentions an unterminated
expansion early and ends with the leaked placeholder: an opening line
"Sure. In bash, ${VAR" with no closing brace, 120 sentences of explanation, a short fenced bash block, then the provider's trailing ${answer}`.

8,370 characters. The unbounded pattern removes 8,354 of them and leaves
Sure. In bash, + backtick. The bounded scan removes 10, the newline and ${answer}.
That is the bug #9012 fixed, and this test pins the boundary that separates the fix from
a regression back to it.

Other checks

  • studio/frontend/src tree hash is identical between the merge base and this head, and
    studio/backend differs only under tests/. No shipped byte changes, so old installs,
    existing threads, existing settings and per-model config cannot be affected.
  • The three Python files are exactly what scripts/run_ruff_format.py produces when run
    on the merge base. Byte identical. Pre-existing drift on main, not this branch fighting
    the formatter.
  • 100,000 generated inputs at the real 4,096 window, plus a 20,000 reply end-of-stream
    model and a 20,000 case interrupt/reload/second-tab model: no input where the bounded
    scan removed more than the unbounded pattern, returned a non-prefix, or where
    isCandidate() was false while the strip cut. 8,333 of the 20,000 replies were actually
    stripped, so the properties are not vacuous.
  • Edge cases all agree with the unbounded pattern: empty, one character, ${},
    unterminated ${, 3,000 nested openers, unterminated code fence, RTL, CJK, emoji inside
    the fragment, a 300,000 character reply, and a 9,000 character trailing whitespace run.
  • Chromium 151 / Firefox 153 / WebKit 26.5: identical results on the pinned case and on
    20,000 generated inputs each. The module is pure string and regex work, so the engines
    are what matter here, not the renderers.
  • npm test 3,672 passing, 0 failing. npm run typecheck clean.
  • Cross platform on a staging replica: frontend unit tests pass on ubuntu-latest and
    windows-latest, 3,672 of 3,672 on both, with the new test named in each log.

@danielhanchen

Copy link
Copy Markdown
Member Author

@danielhanchen
danielhanchen merged commit c755c9e into main Aug 18, 2026
37 of 39 checks passed
@danielhanchen
danielhanchen deleted the nested-placeholder-pin branch August 18, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant