fix(assertions): recognize Responses calls in tool-call-f1 - #10650
fix(assertions): recognize Responses calls in tool-call-f1#10650JuSe123456 wants to merge 2 commits into
Conversation
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 All Clear
I reviewed the PR’s changes to the tool-call-f1 assertion and related docs/tests. The update expands recognition of OpenAI Responses function_call outputs for evaluation only, without introducing new LLM calls, tools, or privileged actions. No LLM security-impacting behaviors (prompting, tool execution, or data movement) are added.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59d638f02a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
When a Responses provider has functionToolCallbacks configured, ResponsesProcessor.processFunctionCall replaces the function_call block with the callback's return value—for example, the existing callback test produces result.output === '11'. Consequently, handleToolCallF1 never receives an object matching this new branch and still scores a valid call as zero. Preserve the original call names in provider metadata/raw output or inspect providerResponse.raw, and cover the callback-enabled path rather than testing only an empty request config.
AGENTS.md reference: AGENTS.md:L380-L380
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
👍 All Clear
Reviewed updates to the tool-call-f1 assertion and associated tests/documentation adding support for OpenAI Responses function_call parsing. Traced data flows from simulated LLM outputs through parsing to scoring, with no execution or privileged actions. No LLM-security relevant sinks or capability changes were introduced by this PR.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
1 participant
Problem
tool-call-f1scores a correct Responsesfunction_callas 0 when it arrives as a JSON string, while the same call in an array scores 1. The shared Responses processor emits exactly this string form when no tool callback is configured, so correct tool selection can fail an agent eval.Fixes #10649.
Fix
Recognize single
function_callblocks alongside Anthropictool_useblocks, retaining the string-name check. Add regression coverage for objects, JSON, mixed text, malformed/non-tool objects, and the real Responses processor/callback-handler path. Clarify the supported format in the assertion docs.Validation
toolCallF1,toolCallF1.responses,responses/processor,assertionsResult,runAssertions,trace, andevaluator/assertions.--no-cache, exported JSON inspected: the failing Responses sample changes from score 0 to 1; all three samples pass. The two-case configuration in the issue also passes. No model API calls.git diff --checkpassed.npm run l/npm run fpassed with a non-blocking cognitive-complexity warning on the existing extraction function.--maxWorkers=2 --bail=1 --sequence.shuffle=false: 5754 passed before an unrelated WindowsEPERMatfs.symlinkSyncintest/providers/openinterpreter.test.ts:356. That provider and test are unchanged; the complete suite is not claimed green.