Preserve Unicode when streamed model replies split a character by Maya-Kid · Pull Request #764 · deeplethe/utopia · GitHub
Skip to content

Preserve Unicode when streamed model replies split a character - #764

Merged
WaylandYang merged 4 commits into
deeplethe:devfrom
Maya-Kid:codex/fix-audit-next
Sep 19, 2026
Merged

WaylandYang merged 4 commits into
deeplethe:devfrom
Maya-Kid:codex/fix-audit-next

Conversation

@Maya-Kid

Copy link
Copy Markdown
Contributor

Problem

A valid streamed model reply is silently corrupted when an HTTP body chunk ends inside a UTF-8 character. All three SSE readers call String::from_utf8_lossy on each transport chunk before buffering it. A split character is irreversibly replaced before the rest arrives; the resulting JSON still parses, so the corruption is returned as a successful answer or tool argument.

This affects chat_at_streaming (used by extraction and alignment), chat_stream_raw, and chat_tools_stream_with (used by the chat model adapter).

Change

Buffer bytes until the complete SSE frame arrives, then decode it. Preserve the collected reader's existing final-frame handling. No dependencies, protocol/API changes, or database changes.

Evidence and checks

Base: 816d4a39255b68849d9eecaf273ec32b3f7d793d (dev).

The new tests use the real HTTP client against a local chunked HTTP server that sends one body byte per HTTP chunk. On the original code, all three regressions fail: 你好🦀 becomes ����������. The tool-stream regression also checks that {"city":"杭州"} remains intact.

After the fix:

  • cargo +1.98.1 test -p utopia-llm --locked: 26 passed.
  • cargo +1.98.1 fmt --all --check: passed.
  • cargo +1.98.1 clippy --workspace --all-targets --locked -- -D warnings: passed.
  • git diff --check: passed.

No live model or external provider was called. Full workspace runtime tests were not repeated for this patch; database behavior is unchanged. Existing ASCII, incomplete-stream, response-error, and timeout tests pass in the LLM suite.

Signed-off-by: dada-yan <BinjunYann@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and it is my bug: #757 added chat_at_streaming with buf.push_str(&String::from_utf8_lossy(&part)), which decodes each network chunk on arrival, so a code point split across two chunks is replaced before anything can reassemble it. Chinese and emoji are where it shows first.

Verified here. With the fix, 3 new tests pass and the crate is 26 green. With hunks 1 to 3 reverted and the tests kept:

---- tests::bytewise_utf8_survives_collected_streaming stdout ----
assertion `left == right` failed
  left: "����������"
 right: "你好🦀"

Same failure on the raw and tool paths. Moving the buffer to Vec<u8> and decoding only a complete frame is the right fix, and covering all three streaming methods rather than only the one I touched is the reason this is worth merging as it is.

One correction to the framing, for the record rather than for you to change: git blame puts the same pattern in chat_tools_stream_with and chat_stream_raw at the shallow boundary of this clone (2026-08-29), so those two predate #757. #757 copied an existing pattern into a third place. Your fix is the first time any of the three has been right.

Three things I would not block on:

  • buf.windows(2).position(|w| w == b"\n\n") is a byte-by-byte scan where str::find used std's accelerated search. The rescan from zero on every chunk was already quadratic in the old code, so this is a constant factor on a pathological frame, not a new class. Mentioning it so the next person does not have to rediscover why it is a loop.
  • bytewise_sse depends on hyper surfacing each one-byte write as its own bytes_stream() item. It does today, which the revert proves, but nothing forces it: if hyper ever coalesced, the test would go green while testing nothing. Splitting deliberately on a known code-point boundary would not depend on that.
  • The trailing-frame decode still yields U+FFFD when a stream dies mid-character. Unavoidable, and #768 turns that case into an error anyway.

Comments in this repo are Chinese. The new ones are English.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants