studio: stop nudging a turn that asks the user for a missing detail by mahiatlinux · Pull Request #8946 · unslothai/unsloth · GitHub
Skip to content

studio: stop nudging a turn that asks the user for a missing detail - #8946

Merged
oobabooga merged 16 commits into
unslothai:mainfrom
mahiatlinux:fix/studio-tool-reprompt-nudge
Aug 19, 2026
Merged

studio: stop nudging a turn that asks the user for a missing detail#8946
oobabooga merged 16 commits into
unslothai:mainfrom
mahiatlinux:fix/studio-tool-reprompt-nudge

Conversation

@mahiatlinux

Copy link
Copy Markdown
Collaborator

Studio re-prompted the model for a tool call after it had only asked the user a question. On the reported turn, Qwen3.8-27B answered the prompt "Balls" with a clarification request ending "Let me know what you're after and I'll dig in.", the loop logged Re-prompt 1/3: model responded without calling tools (313 chars) and regenerated it, and the user saw two near-identical questions.

INTENT_SIGNAL in studio/backend/core/inference/tool_call_parser.py matched the trailing I'll. It already exempts "let me know", but that lookahead clears only its own alternative, so a second alternative in the same turn still fired.

What changed

is_short_intent_without_action returns False whenever the new _CLARIFICATION_REQUEST matches. A turn waiting on the user is not a plan without action, whatever else it says. That helper is the shared source of truth, so the GGUF, safetensors and MLX loops all pick it up.

_should_suppress_forced_no_tool_output in llama_cpp.py needed the same gate. It early-returns on _FORCED_PLAN_INTENT before it consults the helper, so a re-prompted retry reading "I need to use the web tool for this. Could you clarify which site you mean?" was discarded whole and the user lost the question entirely. The gate now sits ahead of that branch.

Measurements

studio/backend/tests/data/plan_vs_answer.jsonl holds 300 finished answers captured from local GGUF models, none of which produced a tool call under three forced re-prompts.

tree nudged retry discarded
before the classifier landed 36 (12.0%) 60 (20.2%)
classifier as first landed 5 (1.7%) 1 (0.3%)
this PR 4 (1.3%) 0 (0.0%)

An earlier draft suppressed only an ask that preceded the intent match. Across both the text and retry_text fields, 600 real turns, that position comparison cost two correct suppressions and gained nothing, so the final shape does not compare positions.

Tests

Three, one per path, each failing if the gate is removed:

  • test_a_turn_that_asks_the_user_for_a_detail_is_not_a_plan covers the shared classifier. It keeps its own list rather than joining the parity lists next to it, because INTENT_SIGNAL still matches every case and only the helper rejects them.
  • test_forced_turn_keeps_a_retry_that_asks_the_user_a_question covers the discard branch behind _FORCED_PLAN_INTENT.
  • test_clarification_request_is_not_nudged drives the GGUF loop with the reported turn and asserts a single generation.

tests/data/refactor_guard baselines carry the new module symbol. Only those entries changed, since a full refactor_guard.py snapshot sweeps in unrelated drift already present on main.

Prompt speed

The issue also reports the prompt speed as wrong. That readout is consistent: 925 prompt tokens over a 1.10 s prompt eval is 841 tok/s, matching the 843.1 shown. The engine_stats log line carries llama-server's server-wide prompt_tokens_seconds gauge sampled every 10 s (core/inference/llama_stats.py:92), a different measurement from the per-request timings the panel reads.

Checks

  • pytest -k "(tool_loop or nudge or parser or plan_classifier or reprompt or intent or refactor_guard or strip) and not research_internal": 1202 passed, 4 skipped
  • tests/test_refactor_guard.py: 12 passed
  • ruff check and scripts/run_ruff_format.py: clean
  • End to end against a stub llama-server over HTTP serving the reported turn: one generation, no nudge status, text delivered once. The same run on main makes two generations and logs Re-prompt 1/3: model responded without calling tools (313 chars).
  • No GPU checks. No usable GPU on the test machine, and nothing in the diff is GPU-dependent.

Fixes #8907

A clarification request that also carries an intent clause ("Let me know what you're after and I'll dig in.") matched INTENT_SIGNAL, so the tool loop re-prompted it and the user saw the same question twice. INTENT_SIGNAL already exempts "let me know", but the lookahead clears only that one alternative and the trailing "I'll" still fired.

is_short_intent_without_action now returns False whenever _CLARIFICATION_REQUEST matches, so a turn waiting on the user is never treated as a stall. An earlier draft only suppressed an ask that preceded the intent match. Over the 600 real turns in tests/data/plan_vs_answer.jsonl (both the text and retry_text fields) that comparison cost two correct suppressions and gained nothing, so it is gone. The helper is shared, so the GGUF, safetensors and MLX loops all pick it up.

_should_suppress_forced_no_tool_output needed the same gate. It early-returns on _FORCED_PLAN_INTENT before consulting the helper, so a re-prompted retry reading "I need to use the web tool for this. Could you clarify which site you mean?" was discarded whole and the user lost the question entirely.

Three tests cover the three paths, and each fails if the gate is removed: the shared classifier in test_safetensors_tool_loop.py, the discard branch in test_forced_turn_keeps_a_retry_that_asks_the_user_a_question, and the GGUF loop itself in test_clarification_request_is_not_nudged, which asserts a single generation.

On the 300-turn corpus of finished answers, wasted nudges drop from 5 to 4 and discarded retries from 1 to 0. tests/data/refactor_guard baselines carry the new module symbol; only those entries changed, since a full snapshot would sweep in unrelated drift already present on main.

Fixes unslothai#8907

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28dad9b0a7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
pre-commit-ci Bot and others added 2 commits August 16, 2026 02:00
"let me know if ..." is a sign-off, not a request for input, so a turn like "I'll search the web for the current release now. Let me know if you have any other questions." is an unfulfilled plan and still needs the nudge. _CLARIFICATION_REQUEST now skips that form and keeps matching the blocking ones ("let me know what you're after", "let me know your question").

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 83e8f03d49

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Two gaps in _CLARIFICATION_REQUEST. A question that carries the ask on its own ("Which repository should I inspect?", "Do you mean the package or the UI?") went unmatched, so the duplicate-question regression survived for those phrasings. And "let me know whether ..." reads as a closing remark exactly like "let me know if ...", so a plan behind one skipped its nudge.

"let me know when ..." keeps matching: it waits on the user rather than closing the turn.

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97fa3e954c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Patching one question word at a time kept leaving gaps, most recently "Where should I search?" and "How would you like me to proceed?". The what/which alternatives collapse into a single interrogative rule covering what, which, where, when, who and how, which is both shorter than the two it replaces and closed to the same class of gap.

Word order keeps advice out: "what you should do" needs the pronoun before the modal, while the rule wants the modal first.

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95d8b1ae30

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +211 to +212
r"|(?:could|can|would)\s+you\s+(?:please\s+)?"
r"(?:clarify|specify|confirm|provide|share|tell\s+me)\b"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Distinguish optional direct requests from blocking clarifications

When a stalled plan ends with another optional courtesy such as I'll search now. Could you confirm if you need anything else?, this unconditional arm matches and is_short_intent_without_action returns false, so the GGUF, safetensors, and hosted loops expose the unfulfilled plan without making the tool call. Fresh evidence after the earlier fix is that only the let me know arm checks for if/whether; the could|can|would you confirm|tell me arm accepts the same optional conditional without restriction.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not taking this one. Guarding the clarify/confirm arm on if/whether costs more than it saves: "Could you clarify whether you mean the package or the UI?" is a blocking question, and I measured that the proposed guard flips it to nudged. That phrasing is far more common in real output than "I'll search now. Could you confirm if you need anything else?", which reads as constructed rather than observed.

The asymmetry with the let me know arm is deliberate. There, if/whether introduce a conditional closing. After clarify/confirm the same words introduce the thing being asked for, so the two arms cannot share the guard. "Could you clarify whether you mean the package or the UI?" is pinned as a suppression case in test_a_turn_that_asks_the_user_for_a_detail_is_not_a_plan.

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
"What version are you using?" is a blocking question the interrogative arm could not reach, because the auxiliary list held only should, would, shall, do and did. Added is, are, was, were, can, could and have.

Word order still keeps statements out: "I'll tell you what the results are" has no pronoun after the auxiliary.

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9eb8b2773

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
"Are you using Windows or Linux?" and "Is this about the package or the UI?" ask for a detail without a wh-word, so the interrogative arm could not reach them.

Scoped to is/are/was/were with a sentence-start anchor. Auxiliaries like do and can are left out because "Do you want me to run it?" asks permission to act, which is the stall the nudge exists to recover, and the anchor keeps ordinary prose out ("Here is that file you wanted").

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91d10063e1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
A clarification under a heading ("One question:\nAre you using Windows or Linux?") slipped past the copular arm: the anchor accepted only . ! ? and `^` is not multiline, and model output is routinely formatted this way.

The anchor now also takes a colon, a newline and a list marker. The leading \b moved onto each word-initial arm, since a \b ahead of the group can never match at a newline preceded by punctuation, which left the newline anchor dead on arrival.

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69bac72287

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
"Why do you need this information?" is the same class as the "what ... do you" forms already covered, so why joins the interrogative list.

The copular arm was too loose. "I'll search the advisories now. Is this package vulnerable to CVE-2026-1234?" restates the lookup the model just promised rather than asking the user for anything, and suppressing it left the plan unfulfilled. The arm now takes only "is/are/was/were you" and "is/are/was/were this|that|it about", both of which can only be about the user.

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b34c68c553

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
A turn that echoes the prompt before promising work (`You asked, "Can you provide the latest CUDA release?" I'll search the web now.`) matched on the quoted words, so the gate read the user's question as the model's own and the promised call never got its nudge.

Both call sites now blank double-quoted spans before searching. Quotes are common in real output, 69 of the 597 turns in tests/data/plan_vs_answer.jsonl carry one, and stripping them changes no classification there, so the counts stay at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

1 similar comment
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 636cf32484

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
…l credits to the user

Two gaps. "Which version of the package are you using?" has four words before its auxiliary and the filler bound stopped at two. And the quote strip added last round erased the model's own emphasised question (`One question: "Which repository should I inspect?"`), classifying what was left as a stalled plan.

_ECHOED_USER_QUOTE replaces the blanket span: a quote is skipped only when the model credits it to the user ("you asked", "your question"), which is the case that fix was for. It matches no turn in tests/data/plan_vs_answer.jsonl, and the wider filler changes no count either, so both stay at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 82bc367a5a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
…reach

Three gaps. "What's your operating system?" contracts the auxiliary and the interrogative arm required a space. "I'll search the web now. How can I find the latest release notes?" is the model narrating its own task, but `can i` read as a request for user input. And `To answer your question, one clarification: "Which repository should I inspect?"` let the attribution wildcard cross a clause boundary and erase a question the model wrote itself.

Only `should` now takes `i`, since "Which repository should I inspect?" defers the choice to the user while "How can I ..." does not. The attribution gap is down to light punctuation from a 40-character wildcard.

Corpus counts in tests/data/plan_vs_answer.jsonl are unchanged at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1471ffdc31

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +217 to +218
r"|\b(?:what|which|where|when|who|why|how)\s+(?:\w+\s+){0,4}?"
r"(?:should\s+i\b|(?:should|would|shall|do|did|is|are|was|were|can|could|have)\s+you\b)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Recognize inclusive-we clarification questions

When the model asks for a user-selected choice using inclusive “we,” such as Which repository should we inspect? Once you choose, I'll check it., the trailing intent matches INTENT_SIGNAL, but this arm accepts only should I or auxiliaries followed by you. The shared tool loops therefore nudge and regenerate a valid clarification instead of waiting for the user's choice, and an obligation-led forced GGUF retry can discard it entirely.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obsolete rather than wrong. The interrogative arm this targeted was removed in 5877187 along with the rest of _CLARIFICATION_REQUEST, so there is nothing to add we to. "Which repository should we inspect?" is nudged again, as are the other interrogative shapes from rounds three onward. That is deliberate and explained in the comment on the PR: none of them moved the 300-turn corpus, and covering them needs the whole-response scan that produced this review loop.

Comment on lines +228 to +229
# only light punctuation may sit between the attribution and the quote, else it reaches past the clause it belongs to
r"[,:\s]{0,3}[\"“][^\"”]{0,300}[\"”]"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Strip attributed Markdown quotes before classifying

When the assistant echoes the prompt with normal Markdown formatting, for example You asked: **"Can you provide the latest CUDA release?"** I'll search the web now., the ** between the attribution and quote prevents this pattern from removing the echoed text. _CLARIFICATION_REQUEST then treats the user's quoted Can you provide as an assistant clarification, so the shared loops expose the unfulfilled plan without nudging its tool call. Fresh evidence after the earlier quoted-user-text fix is that the final attribution pattern permits only commas, colons, and whitespace before the quote, excluding common bold, inline-code, and blockquote markers.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obsolete rather than wrong. _ECHOED_USER_QUOTE was removed in 5877187, so there is no attribution pattern left to widen for Markdown markers. The quote handling existed only to serve the whole-response scan, and both went together.

@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

Stepping back on this one and cleaning it up.

Eleven review rounds in, the pattern of the last ten was consistent: each fix opened the next gap, and twice the new finding was a defect in the previous round's fix. That is a signal about the approach rather than about any single phrasing, so I measured it.

nudged retry discarded added pattern
before this PR 5 1 none
first fix 4 0 121 chars
after ten more rounds 4 0 496 chars

Rounds two through eleven quadrupled the pattern and moved the 300-turn corpus in tests/data/plan_vs_answer.jsonl by exactly nothing. Every one of those changes was fitted to a constructed example rather than to observed output.

The cause is that I bolted on a second regex, _CLARIFICATION_REQUEST, that vetoed the whole response. That is not how this file expresses an exclusion. INTENT_SIGNAL scopes each one as a lookahead on the alternative it qualifies, so _HELP_OFFER applies only after I'll or let me, and the let me know exemption is deliberately confined to the let me branch. A whole-response veto has unbounded surface, which is why there was always another hole.

So the fix now lives where the file already put this idea. _HELP_OFFER exists for "phrases that close a clarification request and never precede a tool call", and it gains two: dig in and help analyze. Both are guarded so they only count when no work follows, the way the neighbouring help you arm defers to _ACTION_VERB, so "I'll dig in and search the web" is still a plan. The let me arm now reaches the same sign-offs, since "let me dig in" ends a clarification exactly as "I'll dig in" does. That is the entire change, and it produces the same 4 and 0 as the 496-character version.

Worth noting for the earlier threads: every counter-example raised in rounds two through eleven passes under this shape without special handling. The optional closings, the permission asks, the model's own lookup, the echoed prompt, the self-directed question. All of those were regressions the veto introduced, so removing it removes them.

What this deliberately does not carry over are the interrogative and copular suppressions, five shapes in total. A trailing lookahead cannot express them, and none of them moved the corpus. They are the long tail of the original behaviour rather than something this PR caused, and I would rather leave that tail alone than keep growing unmeasured pattern surface inside a bug fix. If any of those shapes can be shown on real model output, that is a good case for a separate change with its own measurements.

The diff is down from 178 insertions across 7 files to 49 across 5. llama_cpp.py is untouched.

Ten review rounds each closed one phrasing and opened the next, and twice the new finding was a defect in the previous round's fix. Measuring the run explains why: rounds two onward grew the added pattern from 121 to 496 characters and moved the 300-turn corpus in tests/data/plan_vs_answer.jsonl by nothing at all. Every one of those was fitted to a constructed sentence.

The cause was a second regex, _CLARIFICATION_REQUEST, vetoing the whole response. INTENT_SIGNAL does not work that way: it scopes each exclusion as a lookahead on the alternative it qualifies, which is why _HELP_OFFER only applies after "I'll" or "let me". A whole-response veto has unbounded surface, so there was always another hole.

_CLARIFICATION_REQUEST, _ECHOED_USER_QUOTE and the llama_cpp.py changes are gone. _HELP_OFFER gains "dig in" and "help analyze", both guarded to fire only when no work follows, mirroring how the neighbouring "help you" arm defers to _ACTION_VERB. The "let me" arm reaches the same sign-offs, since "let me dig in" closes a clarification just as "I'll dig in" does.

Counts stay at 4 nudged and 0 discarded, from 5 and 1 before the PR. Every counter-example raised in rounds two through eleven passes without special handling, since all of them were regressions the veto introduced. The interrogative and copular suppressions are not carried over: a trailing lookahead cannot express them and none moved the corpus, so they stay the long tail of the original behaviour.

The fix now sits in the existing parity test's phrase lists, because INTENT_SIGNAL and the helper agree again.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5877187f14

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

r"|assist\b|help\s+you\b(?!\s+" + _ACTION_VERB + r")|give\s+you\s+accurate\b)"
r"|assist\b|help\s+you\b(?!\s+" + _ACTION_VERB + r")|give\s+you\s+accurate\b"
# name no work of their own, so they only ever sign off a question to the user (#8907)
r"|(?:dig\s+in|help\s+analy[sz]e)\b(?![\s,;:]*\w))"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep bare “dig in” promises eligible for nudging

When the model responds to an actionable request with only I'll dig in. or I'll help analyze!, this new alternative prevents INTENT_SIGNAL from matching even though the response neither asks the user for information nor performs the promised work. Consequently, the shared GGUF, safetensors, MLX, and hosted loops skip their recovery prompt and expose an unfulfilled plan; the exemption needs to depend on an actual clarification context rather than treating these terminal phrases as user-input sign-offs unconditionally.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not taking this one. The argument applies verbatim to the four _HELP_OFFER arms that predate this PR: a bare "I'll do my best to help." is equally an unfulfilled plan, and origin/main exempts it unconditionally. That was a deliberate call, recorded in the comment above the constant as measured on real turns. dig in and help analyze are the same kind of phrase and carry the same evidence, so singling them out for a context check would make the constant inconsistent with itself.

The suggested remedy, making the exemption "depend on an actual clarification context", is a whole-response scan. That is exactly what this PR removed. The previous ten rounds of this review were spent patching holes in such a scan, and the measurements are in the comment above: the pattern grew from 121 to 496 characters and moved the 300-turn corpus by nothing.

If a bare "I'll dig in." to an actionable request shows up in real model output, that is worth reopening with the turn attached.

@mahiatlinux
mahiatlinux force-pushed the fix/studio-tool-reprompt-nudge branch from 5877187 to 1471ffd Compare August 16, 2026 04:16
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5877187f14

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
Making the "to" optional on the let-me arm let the whole of _HELP_OFFER through, so "Let me assist by searching the web now." stopped being nudged even though it names work. Only the two sign-offs were meant to skip that gate.

_SIGN_OFF now carries them on its own lookahead, leaving the "to" requirement intact for the rest of _HELP_OFFER.

Corpus counts in tests/data/plan_vs_answer.jsonl stay at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e3d28d8dc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread studio/backend/core/inference/tool_call_parser.py Outdated
"I'll dig in. Starting with a web search." satisfied the old guard, which only looked past whitespace and commas, so a turn that carried on with real work lost its only intent signal. The sign-offs now have to reach the end of the response.

\Z rather than $, since INTENT_SIGNAL is compiled with re.M and a plan continuing on the next line is still a plan.

Checked by sweep rather than by example: 486 opener, sign-off and continuation combinations agree with intent, and across 360 combinations of the pre-existing _HELP_OFFER members this PR changes no behaviour at all. Corpus counts stay at 4 nudged and 0 discarded.
@mahiatlinux

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 45493b72d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Copy link
Copy Markdown
Contributor

While tracing the later #8907 reproducer, I found two orthogonal external-loop issues not covered by the classifier change here: the external local-tool path wasn't forwarding nudge_tool_calls, and a valid reprompt dropped plain assistant prose from the upstream conversation, producing user -> user on retry. I opened #9125 as a narrow complementary follow-up; it does not change this PR's classifier or Studio's current nudge default. Focused backend regressions (34 passed), syntax/diff checks, and the frontend TypeScript check are green.

@oobabooga

Copy link
Copy Markdown
Member

@oobabooga
oobabooga merged commit 657aea9 into unslothai:main Aug 19, 2026
35 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Unsloth Studio sometimes nudges models for tool calls when they did not make any tool call

3 participants