Repository navigation
Preserve Unicode when streamed model replies split a character - #764
Conversation
Signed-off-by: dada-yan <BinjunYann@gmail.com>
WaylandYang
left a comment
There was a problem hiding this comment.
Correct, and it is my bug: #757 added chat_at_streaming with buf.push_str(&String::from_utf8_lossy(&part)), which decodes each network chunk on arrival, so a code point split across two chunks is replaced before anything can reassemble it. Chinese and emoji are where it shows first.
Verified here. With the fix, 3 new tests pass and the crate is 26 green. With hunks 1 to 3 reverted and the tests kept:
---- tests::bytewise_utf8_survives_collected_streaming stdout ----
assertion `left == right` failed
left: "����������"
right: "你好🦀"
Same failure on the raw and tool paths. Moving the buffer to Vec<u8> and decoding only a complete frame is the right fix, and covering all three streaming methods rather than only the one I touched is the reason this is worth merging as it is.
One correction to the framing, for the record rather than for you to change: git blame puts the same pattern in chat_tools_stream_with and chat_stream_raw at the shallow boundary of this clone (2026-08-29), so those two predate #757. #757 copied an existing pattern into a third place. Your fix is the first time any of the three has been right.
Three things I would not block on:
buf.windows(2).position(|w| w == b"\n\n")is a byte-by-byte scan wherestr::findused std's accelerated search. The rescan from zero on every chunk was already quadratic in the old code, so this is a constant factor on a pathological frame, not a new class. Mentioning it so the next person does not have to rediscover why it is a loop.bytewise_ssedepends on hyper surfacing each one-byte write as its ownbytes_stream()item. It does today, which the revert proves, but nothing forces it: if hyper ever coalesced, the test would go green while testing nothing. Splitting deliberately on a known code-point boundary would not depend on that.- The trailing-frame decode still yields U+FFFD when a stream dies mid-character. Unavoidable, and #768 turns that case into an error anyway.
Comments in this repo are Chinese. The new ones are English.
Signed-off-by: dada-yan <BinjunYann@gmail.com>

Problem
A valid streamed model reply is silently corrupted when an HTTP body chunk ends inside a UTF-8 character. All three SSE readers call
String::from_utf8_lossyon each transport chunk before buffering it. A split character is irreversibly replaced before the rest arrives; the resulting JSON still parses, so the corruption is returned as a successful answer or tool argument.This affects
chat_at_streaming(used by extraction and alignment),chat_stream_raw, andchat_tools_stream_with(used by the chat model adapter).Change
Buffer bytes until the complete SSE frame arrives, then decode it. Preserve the collected reader's existing final-frame handling. No dependencies, protocol/API changes, or database changes.
Evidence and checks
Base:
816d4a39255b68849d9eecaf273ec32b3f7d793d(dev).The new tests use the real HTTP client against a local chunked HTTP server that sends one body byte per HTTP chunk. On the original code, all three regressions fail:
你好🦀becomes����������. The tool-stream regression also checks that{"city":"杭州"}remains intact.After the fix:
cargo +1.98.1 test -p utopia-llm --locked: 26 passed.cargo +1.98.1 fmt --all --check: passed.cargo +1.98.1 clippy --workspace --all-targets --locked -- -D warnings: passed.git diff --check: passed.No live model or external provider was called. Full workspace runtime tests were not repeated for this patch; database behavior is unchanged. Existing ASCII, incomplete-stream, response-error, and timeout tests pass in the LLM suite.