fix(providers): remove retired OpenAI models - #10636
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10636 +/- ##
==========================================
- Coverage 82.66% 82.66% -0.01%
==========================================
Files 946 946
Lines 81074 81073 -1
Branches 27179 27179
==========================================
- Hits 67020 67016 -4
- Misses 14054 14057 +3
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 All Clear
Reviewed updates across provider model catalogs, moderation defaults, example configs, and docs. The PR primarily removes retired OpenAI models and improves defaults without introducing new LLM flows. No untrusted input paths to LLMs or execution of LLM output were added. No LLM security vulnerabilities were identified.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
There was a problem hiding this comment.
🟡 Changes recommended
The OpenAI provider docs still list a snapshot as “Supported Realtime Models” even though the code changes in this PR treat it as retired, which is user-facing documentation inconsistency.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates Promptfoo’s OpenAI provider catalogs and documentation to stop advertising OpenAI model IDs that are past their shutdown dates, while preserving historical billing/cost estimation for those retired IDs.
Changes:
- Prunes retired OpenAI model IDs from active routing/recommendation catalogs (chat, responses-only, realtime) and moves them into billing-only “retired” lists for historical cost calculation.
- Updates the OpenAI moderation provider default model to
omni-moderation-latestand removes retiredtext-moderation-*IDs from the advertised catalog. - Updates/cleans documentation and examples to remove or replace references to retired models and removes obsolete OpenAI examples.
File summaries
| File | Description |
|---|---|
| test/providers/openai/util.test.ts | Updates routing tests and adds catalog assertions ensuring retired models aren’t advertised while still costing > 0. |
| test/providers/openai/responses/models.test.ts | Adjusts registry expectations to ensure retired IDs are not present in the Responses model registry. |
| test/providers/openai/moderation.test.ts | Adds coverage for the new moderation default and ensures retired moderation IDs aren’t advertised. |
| test/providers/openai-codex-sdk.test.ts | Updates Codex SDK catalog tests and clarifies “explicitly configured outside catalog” behavior. |
| test/providers.test.ts | Updates bare-model auto-routing tests to reflect the revised Responses-only catalog. |
| src/providers/openai/util.ts | Removes retired IDs from active catalogs; introduces retired billing-only model lists; keeps billing model list comprehensive. |
| src/providers/openai/responses.ts | Removes retired IDs from the Responses provider’s advertised model registry. |
| src/providers/openai/moderation.ts | Switches default moderation model and removes retired moderation IDs from the advertised list. |
| src/providers/openai/codex-sdk.ts | Removes retired Codex model IDs from the advertised Codex SDK model list and updates inline docs. |
| site/docs/providers/truefoundry.md | Removes references to retired OpenAI models in TrueFoundry examples. |
| site/docs/providers/openai.md | Updates OpenAI provider docs to remove/replace retired model recommendations and sections. |
| site/docs/providers/openai-codex-sdk.md | Updates Codex SDK docs to remove retired models and refresh examples. |
| site/docs/providers/helicone.md | Removes retired OpenAI model examples from Helicone docs. |
| site/docs/guides/evaluate-coding-agents.md | Updates the recommended Codex model in the guide. |
| examples/redteam-dalle/save.js | Removes obsolete DALL·E redteam example hook implementation. |
| examples/redteam-dalle/README.md | Removes obsolete DALL·E redteam example documentation. |
| examples/redteam-dalle/promptfooconfig.yaml | Removes obsolete DALL·E redteam example configuration. |
| examples/provider-truefoundry/README.md | Removes a retired OpenAI model from the TrueFoundry example README. |
| examples/openai-images/README.md | Updates OpenAI images example docs to reflect DALL·E retirement and compatibility note. |
| examples/openai-deep-research/README.md | Removes obsolete deep research example documentation tied to retired models. |
| examples/openai-deep-research/promptfooconfig.yaml | Removes obsolete deep research example config tied to retired models. |
| examples/openai-deep-research/mcp-server-example.md | Removes obsolete MCP server example doc tied to retired deep research models. |
| examples/openai-deep-research/example-config.yaml | Removes obsolete deep research example matrix tied to retired models. |
| examples/openai-codex-sdk/thread-persistence/promptfooconfig.yaml | Updates the example to a supported Codex model. |
| examples/openai-codex-sdk/sandbox/promptfooconfig.yaml | Updates the example to a supported Codex model. |
| examples/openai-codex-sdk/basic/promptfooconfig.yaml | Updates the example to a supported Codex model. |
| examples/compare-agentic-sdks/promptfooconfig.yaml | Updates the example to a supported Codex model. |
| .gitignore | Removes ignore entry for an example directory that no longer exists. |
Review details
- Files reviewed: 27/28 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da2fb7fcab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
👍 All Clear
Reviewed provider/model catalog updates, moderation defaults, tests, and example/docs changes. The PR primarily removes retired models and outdated examples, with no new agent tools or execution paths added. Based on the diff, there are no new LLM security-relevant data flows or sinks introduced.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
jameshiester-oai
left a comment
jameshiester-oai
left a comment
There was a problem hiding this comment.
Reviewed base fff837f39d94394f746467150b0f07970c2b6bc4 → head 0f2f46098d848285ce29d8862f44a32f0e93f750 (full PR; first automation review of this head). CI is green across the reported matrix, docs, smoke, style and security checks. The active/billing catalog split generally preserves historical cost estimates, but I found one P2 routing regression.
P2 — undated computer-use and deep-research aliases are moved to billing-only prematurely. OpenAI's July 23 deprecation table names the dated computer-use-preview-2025-03-11, o3-deep-research-2025-06-26, and o4-mini-deep-research-2025-06-26 snapshots as shut down and shows their undated aliases as substitute IDs. The computer-use model page and deep-research model page likewise distinguish the undated ID from the deprecated dated snapshot. This PR removes the undated IDs from the Responses routing catalog and classifies them as retired; bare openai:<undated-id> now falls through to Chat Completions (inline detail). Keep the undated aliases in active Responses routing unless a separate source establishes their shutdown, and update the retirement prose/tests accordingly. No live authenticated OpenAI call was made; the API's account-specific availability was not measured.
Existing feedback: the earlier Codex/Copilot notes about audio/realtime docs, E2E override and OpenAI-page anchors appear addressed at this head. The historical DALL·E blog now prominently warns that the removed example is unavailable, but the old command and deleted-file link remain farther down; its existing thread captures that concern, so I have not duplicated it inline.
QA: inspected the 30-file diff, routing and billing callers, tests, examples/docs, existing review comments and current official OpenAI model/deprecation documentation; applied all applicable toolkit review lenses with three independent read-only specialist passes. source ~/.nvm/nvm.sh && nvm use && ./node_modules/.bin/vitest run test/providers/openai/util.test.ts test/providers/openai/moderation.test.ts test/providers/openai/responses/models.test.ts test/providers/openai-codex-sdk.test.ts test/providers.test.ts --reporter=dot: 583 passed, exit 0. source ~/.nvm/nvm.sh && nvm use && npm run build: typecheck/bundle/app/postbuild passed, exit 0. PROMPTFOO_CONFIG_DIR=/tmp/p10636-state npm run local -- eval -c /tmp/p10636-eval.yaml --no-cache -o /tmp/p10636-eval.json against a synthetic loopback server: 2/2 passed, score 1 each, no errors; server recorded the undated deep-research alias on /v1/chat/completions, confirming the wrong route even though the permissive mock returned success. Offline provider-loading probe also resolved all three undated aliases to OpenAiChatCompletionProvider, while explicit openai:responses:computer-use-preview resolved to OpenAiResponsesProvider. No live inference or credentials were used.
Simplification/test gaps and limits: the new table-driven catalog assertions are useful but should add bare-alias routing assertions for surviving Responses IDs; no other material simplification found. The removed OPENAI_DEEP_RESEARCH_MODELS source export has no in-repository caller, but downstream import compatibility was not proven.
| audioOutput: 20 / 1e6, | ||
| }, | ||
| })), | ||
| ...['computer-use-preview', 'computer-use-preview-2025-03-11'].map((model) => ({ |
There was a problem hiding this comment.
[P2] Keep the undated Responses aliases active. The July 23 OpenAI deprecation schedule shuts down computer-use-preview-2025-03-11 but lists computer-use-preview as its substitute; it likewise distinguishes the dated o3/o4-mini deep-research snapshots from their undated aliases. Moving both forms to billing-only here (and the deep-research set below) removes the undated IDs from OPENAI_RESPONSES_MODEL_NAMES. On this exact head, loadApiProvider("openai:computer-use-preview"), openai:o3-deep-research, and openai:o4-mini-deep-research all construct OpenAiChatCompletionProvider instead of the prior Responses provider; a loopback eval sent o3-deep-research to /v1/chat/completions. Retain the undated aliases in active Responses routing and only retire the dated snapshots, with a bare-route regression test and corresponding docs correction.
There was a problem hiding this comment.
👍 All Clear
Reviewed provider/catalog updates, example configs, and docs changes. The PR primarily removes retired models, updates model IDs, and switches the moderation default to a current model. No new LLM tool capabilities, prompt flows, or execution sinks were introduced, and overall risk surface is reduced.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47f90ed887
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
When users rerun the documented npx promptfoo@latest init --example openai-deep-research command, the initializer now searches the release and main, finds neither directory, and fails. The repository already handles intentionally retired examples through REMOVED_EXAMPLES in src/commands/exampleAliases.ts; register this example there with a ref containing these files (and a retirement warning) so existing scripts and copied setup instructions remain usable.
AGENTS.md reference: examples/AGENTS.md:L3-L3
Useful? React with 👍 / 👎.
Promptfoo still advertises OpenAI models whose API shutdown dates have passed. Remove 43 retired model IDs from active catalogs using the OpenAI deprecation schedule, checked on September 4, 2026.
omni-moderation-latestand remove the three retired text-moderation entries.openai-deep-researchandredteam-dalleexamples. Preserve the original DALL·E blog post with a retirement disclaimer.gpt-5.3-codex.All 144 existing billing entries are unchanged, and explicit endpoint configurations for compatible gateways remain supported. Azure and other hosting providers retain their own catalogs. Models with future shutdown dates remain available, including
gpt-4-1106-preview: OpenAI's newer April 2026 announcement gives an October 23 shutdown, conflicting with an older entry.Validation: