fix(matchers): warn on implicit self grading fallback by Asthenia0412 · Pull Request #10640 · promptfoo/promptfoo · GitHub
Skip to content

fix(matchers): warn on implicit self grading fallback #10640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Asthenia0412 wants to merge 1 commit into
base: main
Choose a base branch
from
+67 −6
Open

fix(matchers): warn on implicit self grading fallback #10640

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions site/docs/configuration/expected-outputs/model-graded/index.md
View file
Open in desktop
27 changes: 21 additions & 6 deletions src/matchers/providers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,12 +197,18 @@ export async function getGradingProvider(
const defaultTest = cliState.config?.defaultTest;
const defaultTestObj = typeof defaultTest === 'object' ? (defaultTest as TestCase) : null;
const fallbackProviders = [
defaultTestObj?.provider || undefined,
defaultTestObj?.options?.provider?.text || undefined,
defaultTestObj?.options?.provider || undefined,
{ provider: defaultTestObj?.provider || undefined, source: 'defaultTest.provider' },
{
provider: defaultTestObj?.options?.provider?.text || undefined,
source: 'defaultTest.options.provider.text',
},
{
provider: defaultTestObj?.options?.provider || undefined,
source: 'defaultTest.options.provider',
},
];

const cfg = fallbackProviders.find((candidateProvider) => {
const fallback = fallbackProviders.find(({ provider: candidateProvider }) => {
if (!candidateProvider) {
return false;
}
Expand All @@ -215,13 +221,22 @@ export async function getGradingProvider(
return true;
});

const cfg = fallback?.provider;
if (cfg) {
// Recursively call getGradingProvider to handle all provider types (string, object, etc.)
finalProvider = await getGradingProvider(type, cfg, defaultProvider);
if (finalProvider) {
logger.debug('[Grading] Using provider from defaultTest fallback', {
const logContext = {
providerId: finalProvider.id(),
});
};
if (fallback.source === 'defaultTest.provider') {
logger.warn(
'[Grading] defaultTest.provider is being used as the grader because no explicit grader is configured',
logContext,
Comment on lines +229 to +235

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Sanitize provider IDs before warning

When defaultTest.provider is a raw HTTP endpoint containing basic-auth credentials or a token-bearing query parameter, HttpProvider.id() returns that complete URL and this new warning writes it to normal CLI and CI logs under the non-sensitive providerId context key. Generic context sanitization does not apply URL sanitization to that key, so credentials that were previously confined to debug output become exposed; pass the ID through sanitizeProviderIdForLog before logging it.

Useful? React with 👍 / 👎.

);
Comment on lines +233 to +236

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Warn only after the provider passes modality validation

When an embedding assertion such as similar runs with a normal chat provider in defaultTest.provider, this warning fires before getAndCheckProvider determines that the provider lacks callEmbeddingApi and returns the built-in embedding provider instead. The CLI therefore claims that defaultTest.provider is being used as the grader even though it is never called for that assertion; defer the warning until after type validation so it describes the provider actually selected.

Useful? React with 👍 / 👎.

Comment on lines +233 to +236

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Emit the implicit-grader warning only once per eval

When an eval has multiple outputs or model-graded assertions and relies on defaultTest.provider, every assertion invocation reaches this branch and emits the same warning. A typical matrix can therefore produce hundreds of identical warning blocks, obscuring progress and actionable errors in normal CLI and CI output; deduplicate it per eval/provider rather than logging once per grading call.

Useful? React with 👍 / 👎.

} else {
logger.debug('[Grading] Using provider from defaultTest fallback', logContext);
}
}
} else {
finalProvider = defaultProvider;
Expand Down
41 changes: 41 additions & 0 deletions test/matchers/getGradingProvider.test.ts
View file
Open in desktop
Loading