[Bug]: RealtimeAgent split-transcript merge replaces the previous user turn's content, dropping attached DataBlocks · Issue #3237 · agentscope-ai/agentscope · GitHub
Skip to content

[Bug]: RealtimeAgent split-transcript merge replaces the previous user turn's content, dropping attached DataBlocks #3237

Open

Description

@BlueX888

Prerequisites

  • I have searched the existing issues and discussions, and this is not a duplicate.
  • This is a bug, not a usage question. (For questions, please use Discussions instead.)

Background / Description

RealtimeAgent._merge_user (src/agentscope/agent/_realtime/_agent.py:835) runs when a settled transcript continues the previous user turn — e.g. endpointing split one sentence in two, and the second half arrives inside the TurnAggregator merge window (800 ms by default). Its docstring says it appends text to the previous user turn, but the implementation replaces the whole content of that turn with a single TextBlock:

previous = context[-1].get_text_content() or ""
context[-1].content = [TextBlock(text=f"{previous}{text}")]

Every non-text block of that turn is destroyed. User messages are documented and validated to carry DataBlocks (_assert_user_content_blocks, src/agentscope/message/_base.py:32-38, and the UserMsg docstring states that only TextBlock and DataBlock are allowed), and RealtimeAgent.send appends any Msg handed to it to state.context (src/agentscope/agent/_realtime/_agent.py:528). So a photo a user attached between the two halves of a split utterance silently disappears from the conversation context once the second half settles.

Expected: merging the second half should append its text to the previous user turn while leaving the attached blocks in place, the way Msg.append_event handles TOOL_RESULT_TEXT_DELTA (src/agentscope/message/_base.py:432-435).

Error Messages

No exception — the loss is silent. agent.state.context[-1].content changes from [TextBlock, DataBlock] to [TextBlock].

Steps to Reproduce

  1. Code (repro.py; it drives the internal transcription handler directly so the split is deterministic without audio hardware):
import asyncio
import base64

from agentscope.agent import RealtimeAgent
from agentscope.message import Base64Source, DataBlock, TextBlock, UserMsg
from agentscope.realtime import _events as me


async def main() -> None:
    agent = RealtimeAgent("friday", "Be brief.", None)

    # Half 1 of a split turn: endpointing settles early.
    agent._on_transcription(
        me.InputTranscriptionEvent(item_id="u1", text="帮我订张机票"),
    )

    # Between the halves the user attaches a photo (RealtimeAgent.send
    # appends any Msg to state.context).
    agent.state.context.append(
        UserMsg(
            name="user",
            content=[
                TextBlock(text="这是行程参考图"),
                DataBlock(
                    source=Base64Source(
                        data=base64.b64encode(b"png-bytes").decode(),
                        media_type="image/png",
                    ),
                    name="trip.png",
                ),
            ],
        ),
    )

    print("context before merge:")
    for msg in agent.state.context:
        print("  ", [type(b).__name__ for b in msg.content])

    # Half 2 arrives inside the 800 ms merge window.
    agent._on_transcription(
        me.InputTranscriptionEvent(item_id="u2", text="去上海的"),
    )

    print("context after merge:")
    for msg in agent.state.context:
        print("  ", [type(b).__name__ for b in msg.content])


if __name__ == "__main__":
    asyncio.run(main())
  1. Run: python repro.py
  2. Output on current main:
context before merge:
   ['TextBlock']
   ['TextBlock', 'DataBlock']
context after merge:
   ['TextBlock']
   ['TextBlock']

The photo's DataBlock is gone from the context after the second half settles; expected the merged turn to keep [TextBlock, DataBlock] plus the appended text.

Environment

  • AgentScope Version: 2.0.10.dev0 (main, e4c1e07)
  • Python Version: 3.12.14
  • OS: macOS 15

Activity

  1. added
    triage/verifyingA bot is verifying whether this bug is real
    triage/confirmedVerified: the reported defect exists
    and removed
    triage/verifyingA bot is verifying whether this bug is real
    on Oct 11, 2026
  2. ffffff55 commented on Oct 11, 2026

    @ffffff55
    Contributor

    I would like to work on this — I will follow up with a fix and a regression test shortly.

  3. github-actions commented on Oct 11, 2026

    @github-actions
    Contributor

    It's yours, @ffffff55. If there is no pull request and no word from you by 2026-10-25 the claim is released so nobody is blocked — a comment here renews it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

triage/confirmedVerified: the reported defect exists

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions