{{ message }}
fix(matchers): mark grader failures in factuality checks - #10641
Open
Asthenia0412 wants to merge 1 commit into
Open
fix(matchers): mark grader failures in factuality checks#10641Asthenia0412 wants to merge 1 commit into
Asthenia0412 wants to merge 1 commit into
Conversation
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
👍 All Clear
Reviewed changes to LLM grading matchers focused on error handling and malformed output classification. The PR marks provider/malformed output cases as grader failures and adds corresponding tests. No new LLM calls, capabilities, or output execution paths were introduced, and overall the change reduces risk.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10481
Summary
graderFailsemantics when factuality and closed-QA grading providers return errors or no outputmetadata.graderError=trueD/N) as ordinary assertion failuresTests
npx vitest run test/matchers/factuality.test.ts test/matchers/closed-qa.test.tsfailed on the newmetadata.graderErrorexpectationsnpx vitest run test/matchers/factuality.test.ts test/matchers/closed-qa.test.tsnpx vitest run test/matchers/factuality.test.ts test/matchers/closed-qa.test.ts test/matchers/utils.test.ts test/assertions/modelGradedClosedQa.test.ts test/assertions/contextPropagation.test.tsnpx @biomejs/biome check src/matchers/llmGrading.ts test/matchers/factuality.test.ts test/matchers/closed-qa.test.tsgit diff --checknpm run local -- eval ... --no-cache -o /tmp/promptfoo-grader-failure-metadata.jsonagainst a temporary local config with a malformedexecgrader; exported JSON showed bothfactualityandmodel-graded-closedqacomponent results includemetadata.graderError=truenpm run tscstill fails on existing optional@openai/codex-securityimport/type errors insrc/providers/openai/codex-security.ts, unrelated to this matcher diff.