fix(matchers): warn on implicit self grading fallback - #10640
fix(matchers): warn on implicit self grading fallback#10640Asthenia0412 wants to merge 1 commit into
Conversation
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 All Clear
Reviewed changes to provider fallback selection and related tests/documentation. The PR adds source-aware fallback logic and a warning when defaultTest.provider implicitly acts as the grader. No new LLM prompts, tool capabilities, or execution sinks were introduced. No LLM security vulnerabilities were identified in this diff.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6598abb51
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| logger.warn( | ||
| '[Grading] defaultTest.provider is being used as the grader because no explicit grader is configured', | ||
| logContext, | ||
| ); |
There was a problem hiding this comment.
When an embedding assertion such as similar runs with a normal chat provider in defaultTest.provider, this warning fires before getAndCheckProvider determines that the provider lacks callEmbeddingApi and returns the built-in embedding provider instead. The CLI therefore claims that defaultTest.provider is being used as the grader even though it is never called for that assertion; defer the warning until after type validation so it describes the provider actually selected.
Useful? React with 👍 / 👎.
| logger.warn( | ||
| '[Grading] defaultTest.provider is being used as the grader because no explicit grader is configured', | ||
| logContext, | ||
| ); |
There was a problem hiding this comment.
When an eval has multiple outputs or model-graded assertions and relies on defaultTest.provider, every assertion invocation reaches this branch and emits the same warning. A typical matrix can therefore produce hundreds of identical warning blocks, obscuring progress and actionable errors in normal CLI and CI output; deduplicate it per eval/provider rather than logging once per grading call.
Useful? React with 👍 / 👎.
| const logContext = { | ||
| providerId: finalProvider.id(), | ||
| }); | ||
| }; | ||
| if (fallback.source === 'defaultTest.provider') { | ||
| logger.warn( | ||
| '[Grading] defaultTest.provider is being used as the grader because no explicit grader is configured', | ||
| logContext, |
There was a problem hiding this comment.
When defaultTest.provider is a raw HTTP endpoint containing basic-auth credentials or a token-bearing query parameter, HttpProvider.id() returns that complete URL and this new warning writes it to normal CLI and CI logs under the non-sensitive providerId context key. Generic context sanitization does not apply URL sanitization to that key, so credentials that were previously confined to debug output become exposed; pass the ID through sanitizeProviderIdForLog before logging it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The reviewed commit uses the fix(matchers): scope while modifying the documentation site; split this documentation change into a commit using the required docs(site): scope.
AGENTS.md reference: site/AGENTS.md:L9-L9
Useful? React with 👍 / 👎.
1 participant
Summary
defaultTest.provideras the grader because no explicit grader was configured.defaultTest.providercan become the implicit grader and may make the target grade itself.Why
defaultTest.provideris primarily the test target override, but today it is also used as a grader fallback. That can make the target model generate and grade the same output with only a debug log, so users may miss a self-grading setup in CI or normal eval output.Fixes #10581
Validation
npx vitest run test/matchers/getGradingProvider.test.tsinitially failed on the new warning assertion before the implementation change.npx vitest run test/matchers/getGradingProvider.test.ts test/matchers/utils.test.tsnpx @biomejs/biome check src/matchers/providers.ts test/matchers/getGradingProvider.test.tsnpx prettier --check site/docs/configuration/expected-outputs/model-graded/index.mdgit diff --checkNotes
npm run tscstill fails locally on the existing optional@openai/codex-securityimport/type errors insrc/providers/openai/codex-security.ts; this branch does not touch that provider.