Chat Completions replay emits an invalid sequence for two messages in one tool-call turn · Issue #5342 · openai/openai-agents-python · GitHub
Skip to content

Chat Completions replay emits an invalid sequence for two messages in one tool-call turn #5342

Open

Description

@Timotty

Please read this first

Describe the bug

When replaying a Responses-format history ordered as [function_call, message, message, function_call_output], Converter.items_to_messages merges only the first message into the assistant with tool_calls. The second message becomes a separate assistant message before the tool result. The Chat Completions request therefore contains [assistant(tool_calls), assistant, tool]. The SDK's own comment says this sequence is rejected by the API. A message after a refusal in the same tool turn has the same problem.

The malformed payload was observed in a direct converter probe and in the payload passed by Runner.run to a local mocked chat.completions.create. A live HTTP 400 was not measured; the rejection is inferred from the documented Chat Completions tool-call ordering contract and the error shown in #4727.

Debug information

  • Agents SDK version: 0.23.1 (also reproduced on main at 125efa029b4bfd84238bd2c4fd69c3406f802663).
  • Related library versions: none required for the direct probe.
  • Python version: 3.12.
  • Operating system: Windows.
  • Model and model provider: Chat Completions adapter; local mock client, no paid API call.
  • Does the issue reproduce with the latest Agents SDK release? Yes, v0.23.1.
  • Does the issue occur consistently or intermittently? Consistently for this input order.

Relevant observed roles:

user
assistant(content='Let me check the weather.', tool_calls=[call_1])
assistant(content='One moment.')
tool(tool_call_id='call_1', content='sunny')

Repro steps

from agents.models.chatcmpl_converter import Converter

items = [
    {"role": "user", "content": "What is the weather?"},
    {"id": "fc_1", "call_id": "call_1", "name": "get_weather", "arguments": "{}", "type": "function_call"},
    {"id": "msg_1", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_text", "text": "Let me check the weather.", "annotations": []}]},
    {"id": "msg_2", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_text", "text": "One moment.", "annotations": []}]},
    {"type": "function_call_output", "call_id": "call_1", "output": "sunny"},
]
for message in Converter.items_to_messages(items):
    print(message)

On v0.23.1, the second assistant message separates the tool-call assistant from its tool result. A control history with just one assistant message between function_call and function_call_output converts correctly.

Expected behavior

The two same-turn text messages should be merged into a single assistant message with the declared tool_calls, immediately followed by the tool result. A later assistant turn after the tool result should stay separate.

Activity

  1. Timotty commented on Oct 10, 2026

    @Timotty
    Author

    I've prepared a focused fix with regression tests in my fork: commit a221515 on fix/chatcmpl-multi-message-tool-turn-merge. The direct converter and mocked Runner.run probes now produce one assistant with joined text and tool_calls, followed immediately by the tool message; 128 focused tests pass. Two independent reviews found no blocking issues.

    I tried to open a PR against openai/openai-agents-python, but GitHub reports that only collaborators can create PRs for this repository; both the GraphQL and REST attempts were rejected. Could a maintainer enable outside PR creation or use the fork commit above? The four failures in the broad Windows suite are release-workflow tests that also fail on the unchanged base commit because uv tries to create a temporary venv under C:\WINDOWS.

  2. Ayushraj06-bit commented on Oct 11, 2026

    @Ayushraj06-bit
    Contributor

    I reproduced this on main at 125efa0 and, working independently, arrived at the same change as the fork commit above. The merge added in #4728 only accepts a pending tool-call message whose content is None or a list and that has no refusal. Once the first message merges, content is a str, so the next message flushes. Appending in the str case (with the same "\n" separator used for text parts within one message) and dropping the refusal guard fixes both the multi-message case and the text-after-refusal case.

    My change with two regression tests is on this branch (commit 5a63ae1). PR creation is limited to collaborators here, so I couldn't open it as a PR.

    Some extra evidence for whoever lands it:

    • Only conversions that previously emitted assistant(tool_calls) directly followed by another assistant message change. In the old flush branch, the newly opened assistant message is always the next item appended after the tool-call message. Valid histories convert identically before and after: [message, message, function_call], a user message ending the turn, a plain {"role": "assistant"} message, and the thinking-block content-part path.
    • Through Runner.run with a stubbed chat.completions.create, the request goes from user, assistant(tool_calls), assistant, tool to user, assistant(tool_calls), tool.
    • Interleaved parallel calls ([fc1, msg, fc2, msg, out1, out2]), three messages in one turn, and two consecutive tool turns also come out valid.
    • With the fix, the full suite on Windows / Python 3.11 has 10780 passing tests. The only failures are 22 tests/sandbox symlink tests that fail identically on a clean main. Format, lint, mypy and pyright are clean.

    Two cases neither change covers, in case they're worth a follow-up:

    • An easy-input {"role": "assistant", "content": "..."} item between a function_call and its output still flushes, which gives the same assistant(tool_calls), assistant, tool sequence.
    • Two refusals in one tool turn keep only the last one. That matches how several refusal parts within a single message are already handled.

    Like the original report, I didn't measure a live HTTP 400. The rejection is inferred from the documented tool-call ordering rule and the error in #4727.

  3. jbeckwith-oai commented on Oct 13, 2026

    @jbeckwith-oai
    Collaborator

    Thanks to both of you for the careful investigation and proposed fixes. The examples make the converter behavior clear: after the first text message or refusal merges, a second message can split the tool-call assistant from its tool result.

    Could you share a minimal example of the workflow that originally produces those two separate assistant messages in the same tool-call turn? The built-in Chat Completions streaming and non-streaming paths assemble one output message per turn, so understanding the source of this history will help us scope the fix beyond #4728. In particular, it would help to have:

    • The provider/model and SDK version, and whether the history comes from Responses, a custom model adapter, a saved session, or application-side history construction.
    • A sanitized originating response/history or a small example showing how the sequence is produced before it reaches the converter, including any model switch or history transformation.
    • If already observed, the actual replay error and the endpoint/provider that returned it. No need to make a paid API call just to gather this.

    Please replace prompts, tool arguments/results, and any credentials or personal data with placeholders, while preserving the item order and matching call IDs. The converter and mocked Runner examples already establish the output shape; the missing piece is the originating workflow and its practical impact. Thanks again!

  4. Ayushraj06-bit commented on Oct 15, 2026

    @Ayushraj06-bit
    Contributor

    Thanks for looking at this. To be upfront: I didn't hit this in a real workflow. I reproduced it from the history in the original report, so I don't have an originating response or an observed replay error to share. Timotty would know where their history came from.

    What I can add about the source:

    • I couldn't find a first-party SDK path that produces two messages in one tool-call turn. The Chat Completions stream handler and Converter.message_to_output_items each emit at most one message per response, LiteLLM and any-llm go through the same converter, and the experimental hosted multi-agent model drops every message except the root agent's final answer.
    • So the history would have to come from outside the Chat Completions path: a turn produced by OpenAIResponsesModel or another Responses-compatible provider and later replayed through a Chat Completions model, which the mixing models docs allow, or input built by the application. I haven't verified that the Responses API actually returns a message after a function_call within one response, so I can't confirm that this path happens in practice.

    If no supported producer turns up, I understand narrowing or closing this rather than extending #4728.

  5. zhgao commented on Oct 17, 2026

    @zhgao

    Hi, I've implemented a fix for this issue and it's ready on my fork: zhgao/openai-agents-python branch fix/issue-5342.

    Compare view: main...zhgao:fix/issue-5342

    The branch has two commits ("Fix Chat Completions replay producing invalid sequence for same-turn double message") which:

    • In Converter.items_to_messages (src/agents/models/chatcmpl_converter.py), extend the same-turn merge branch so a second same-turn text message (or a text message after a refusal) merges into the pending tool-call message instead of becoming a standalone assistant message before the tool result. The merge now accepts a str pending_content (a previously merged message appends with "\n") and drops the "refusal" not in guard, since a Responses message may carry both text and refusal parts.
    • Add a _text_message helper and two regression tests in tests/models/test_openai_chatcompletions_converter.py: two same-turn texts merge into one assistant message with tool_calls (roles become ["assistant", "tool", "assistant"]); text after a refusal merges into the pending tool-call message.

    However, this repository restricts pull request creation to collaborators, so I'm unable to open the PR myself. Could a maintainer please open a PR from the fix/issue-5342 branch on my fork (zhgao/openai-agents-python), or grant me collaborator access so I can submit it directly? I'm happy to address any review feedback. Thank you!

  6. MuneebMalik244535 commented on Oct 18, 2026

    @MuneebMalik244535

    I have investigated this issue, identified the root cause, and verified a minimal fix with complete test coverage on my fork.

    Root Cause Analysis (RCA)

    In src/agents/models/chatcmpl_converter.py, when converting items to Chat Completions messages, a tool-call turn can contain multiple assistant message items before the tool output (e.g., [function_call, message_1, message_2, tool_output]).

    On line 776, the merge condition checked:

    and (pending_content is None or isinstance(pending_content, list))
  7. 0-Actual commented on Oct 21, 2026

    @0-Actual

    Candidate replay implementation and additional boundary coverage

    Building on @Timotty's report and the prior proposals from @Ayushraj06-bit and @zhgao, I am sharing additional candidate work against 64cd5dfc5a1579870dcbed0ea42b27aa84033fee. The same-turn text merge is already proposed above; the additional contribution is regression coverage and an explicit candidate treatment of successive refusal messages.

    The originating-workflow question remains unanswered by my evidence. These checks construct Responses-format history and use a synthetic Chat Completions transport. They do not establish a live provider producing this sequence, an observed HTTP 400, or production impact. The built-in Chat Completions producers emit one message per turn. This is candidate research for assessing the converter boundary; it is not a request to mark #5342 resolved.

    For the supplied histories, the candidate keeps successive text/refusal messages in the pending assistant message that owns the tool calls, followed by matching tool results, while separating later turns. It preserves call IDs, reasoning blocks/signatures and input values in the covered cases. It also joins two successive refusals. That last behavior remains a contract question: existing handling retains only the last refusal part within one message, so concatenation across messages should not be treated as an agreed SDK guarantee.

    The final 13-case regression covers successive text, mixed text/refusal, two refusals, empty text, Unicode/newlines, parallel call IDs/results, later turns, native/legacy thinking blocks and signatures, input immutability, and the requests built by ordinary and streamed Runner calls through a fake transport.

    Recorded verification on the pinned source:

    • The initial 11 cases fail on the baseline. The corrected runtime with that initial file passes a 169-test focused population.
    • Adding two reasoning-boundary cases gives a final 13-case file, which passes. A later genuine-checkout retry also reports 13 passed, exit 0, with the candidate file hashes unchanged.
    • The earlier checkout attempt printed 13 passed but exited 1 during temporary-directory cleanup. That failed process is preserved separately; the clean retry used workspace-owned temporary directories with the same plugins enabled.

    These populations overlap and must not be added together. All checks use synthetic data without paid model calls. Strict final review and the complete final SDK verification stack remain incomplete. No upstream acceptance, live-provider validation, release readiness or deployment is claimed.

    Prepared by Sherman G. Laing with disclosed Codex assistance. Existing reporters retain credit for the issue and prior proposals. The contributed code is offered under the accompanying MIT terms.

    The complete candidate patch, final test and recorded evidence are now public, with scope and reproduction details. After extraction, see followup/patches/5342-history-replay.patch, followup/tests/models/test_history_replay_regression.py, and verification/. The archive also retains unsuccessful attempts and the MIT notices.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions