openai:agents ignores invalid retry policies instead of reporting them · Issue #11678 · promptfoo/promptfoo · GitHub
Skip to content

openai:agents ignores invalid retry policies instead of reporting them #11678

Open

Description

@Shizoqua

What happens

modelSettings.retry.policy for the openai:agents provider accepts the presets never, providerSuggested, networkError, retryAfter, and httpStatus, any and all objects. Any other value is silently ignored:

  • A misspelled preset such as networkErrors resolves to no policy, so the configured retry behavior never applies.
  • Inside any or all, an invalid entry is dropped and the rest still run. all: [{ httpStatus: [429] }, networkErrors] retries on every 429 instead of only on 429 network errors.
  • Malformed objects such as { any: [] } or { httpStatus: 429 } are also ignored or crash with an unrelated error.

Reproduce

providers:
  - id: openai:agents:support-agent
    config:
      agent:
        name: Support Agent
        instructions: Answer briefly.
        model: gpt-5-mini
      modelSettings:
        retry:
          maxRetries: 2
          policy: networkErrors

The eval runs with no warning and the typo has no effect.

Expected

An invalid retry policy fails with a clear configuration error that lists the supported values, before any request is sent.

Activity

  1. rishiiicreates commented on Oct 18, 2026

    @rishiiicreates

    looks like this was resolved in commit ea8b3d0 where resolveRetryPolicy was added to agents-model-settings.ts along with unit tests validating invalid policy options. tested against main and invalid retry configurations are reported as expected. this can be closed out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions