fix(studio): coerce tool-call arguments before replaying to custom endpoints (#9039) - #9300
Conversation
…dpoints (unslothai#9039) Malformed streamed tool-call JSON (concatenated delta fragments, dict-shaped arguments) was forwarded verbatim to OpenAI-compatible custom endpoints and poisoned every follow-up turn with invalid_request_error. Mirror the frontend toolCallReplayArguments helper on the backend and apply it in the external message builder, provider proxy, and studio tool loop replay path.
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 869f939392
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| """ | ||
| if isinstance(args_text, str) and args_text: | ||
| try: | ||
| json.loads(args_text) |
There was a problem hiding this comment.
When a custom endpoint streams a syntactically valid but deeply nested argument value, json.loads(args_text) can raise RecursionError. _normalized_call explicitly catches this condition so a pathological provider response does not terminate the tool loop, but this newly added second parse lets it escape, causing a server error mid-stream instead of replaying the safely wrapped structured arguments. Catch RecursionError here and proceed to the fallback.
Useful? React with 👍 / 👎.
|
I believe the issue this PR aims to fix is already solved on main, see #9039 (comment). |
3 participants
Summary
Fixes #9039: tool calling via a custom OpenAI-compatible endpoint no longer poisons the chat history with malformed
tool_calls[].function.arguments, which causedinvalid_request_erroron every follow-up send.Root cause
When Studio proxies to a custom endpoint (vLLM, llama.cpp, etc.) and runs the tool loop, assistant tool-call arguments were replayed verbatim upstream. Two common failure modes:
'{"query":"first"}{"query":"second"}'that are not valid JSON.function.argumentsas an object instead of the OpenAI wire-format JSON string.Strict endpoints reject those with
invalid tool call arguments. Once that turn is in history, every subsequent request fails and the chat freezes.The frontend already had
toolCallReplayArguments(#8754); the backend proxy/tool loop did not.Fix
coerce_tool_call_replay_arguments()andcoerce_messages_tool_calls_for_wire()mirroring the frontend helper.studio_tool_loop.py(tool loop replay to external providers)_build_external_messages()(outbound messages to custom endpoints)external_provider.py(after markup neutralization for template-applying providers)Test plan
pytest tests/test_tool_call_replay_arguments.py— new regression coverage for [Bug] Tool calling fails with "invalid tool call arguments (invalid_request_error)" on every chat via custom endpoint #9039pytest tests/test_tool_loop_controller.pypytest tests/test_external_tool_truncated_and_budget.pypytest tests/test_studio_tool_loop.py