You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Praise: tasteful streaming-thinking-token handling in _process_thinking_tokens #3815
backend/utils/llm_utils.py:15-55 is the kind of code I usually expect to be a mess, and isn't. The function correctly handles three independently tricky cases:
A single streaming token containing both<think> and </think> (the partition order on lines 26 and 40 is deliberate and necessary).
A provider that emits a closing </think> without an opening one (line 28-31 — clear token_join and emit "" to the callback so the UI can drop already-streamed reasoning).
Content split across multiple tokens where the tag boundary falls mid-stream.
Two things I appreciated:
The function's return value is_thinking plus the explicit token_join: List[str] argument keeps state out of the function — the caller in call_llm_for_system_prompt (line 116) owns both. That makes it trivially unit-testable without mocking anything.
The "treat everything accumulated so far as reasoning and clear it" branch comes with a callback nudge to the UI (callback("")). That's exactly the kind of correctness-for-the-user behaviour that tends to get dropped in the rush to ship streaming.
The accompanying check on line 124 (if not result and content_tokens_seen > 0: logger.warning(...)) is a nice operational touch — it gives you a single log line that explains "the user got an empty response because everything matched <think>…</think>", which is exactly the kind of diagnostic you want when investigating "the model returned nothing" tickets.
Filing because clean streaming-tag handling is rare and this one is worth pointing at as a reference for similar code elsewhere in the codebase.
Activity