iframe-proxy

Copilot · 2026-09-29T05:05:01Z

Copilot auto billing can resolve to utility-suffixed model IDs that aren’t present in the model catalog, causing requests to use the wrong endpoint. For example, gpt-5.6-luna-utility must use the responses API configured for gpt-5.6-luna.

Wire API inference: When an exact catalog match is absent, look up the base model for IDs ending in -utility. Exact matches and explicit COPILOT_PROVIDER_WIRE_API overrides retain precedence.
Regression coverage: Verify the utility variant inherits its base model’s wire API.

Fixes Auto model fails for org billing #63117

Run: https://github.com/github/gh-aw/actions/runs/36528584087

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 20.5 AIC · ⌖ 8.82 AIC · ⊞ 9.4K · ◷
_{Comment /souschef to run again}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-09-29T05:46:15Z

github-actions · 2026-09-29T05:46:37Z

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #64177

github-actions · 2026-09-29T05:46:40Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-09-29T05:46:42Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

No ADR enforcement needed: PR does not have the "implementation" label and has ≤100 new lines of code in business logic directories.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-09-29T05:46:43Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-09-29T05:46:49Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions

Ponytail review: one simplification opportunity found in changed lines only.

net: -4 lines possible.

Generated by ✂️ Ponytail Reviewer for #64177 · codex · gpt53codex · 3.25 AIC · ⌖ 3.29 AIC · ⊞ 12.5K
_{Comment /ponytail to run again}

github-actions · 2026-09-29T05:49:55Z

-      return;
+  const lookupModelNames = [normalizedModelName];
+  if (normalizedModelName.endsWith("-utility")) {
+    lookupModelNames.push(normalizedModelName.slice(0, -"-utility".length));


L476: shrink: repeated linear scan via Object.entries(models).find(...) inside loop. Build one lowercase-key map once, then direct lookup for exact and -utility base.

Addressed in 2dc44d0: the shared catalog lookup now builds a lowercase-key map once and checks exact IDs before the utility base.

Copilot

Copilot review overview

🟢 Approval recommended

The focused implementation matches the reported failure and includes appropriate regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds fallback wire API inference for Copilot model IDs ending in -utility, resolving them through their cataloged base model.

Changes:

Preserves explicit overrides and exact-match precedence.
Adds regression coverage for utility-model inference.

File	Description
`actions/setup/js/copilot_harness.cjs`	Falls back from utility variants to base catalog models.
`actions/setup/js/copilot_harness.test.cjs`	Tests responses API inference for a utility variant.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-29T05:53:19Z

Comment Memory

reviewed_at: 2026-09-29T05:48:24.510+00:00
review_event: REQUEST_CHANGES
top_themes:
  - utility-model wire API fix misses reflect/SDK path
files_reviewed:
  - actions/setup/js/copilot_harness.cjs
  - actions/setup/js/copilot_harness.test.cjs
comment_count: 1

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 31.8 AIC · ⌖ 7.23 AIC · ⊞ 20.2K · ◷
_{Comment /review to run again}

github-actions

Request changes

This patch fixes the direct env auto-configure path, but the reflect-driven SDK path still does an exact models.json lookup and can keep sending *-utility Copilot models to the wrong API.

Blocking theme

The new fallback lives only in applyCopilotWireAPI(). When the harness builds SDK multi-provider config from AWF reflect data, it still uses getCatalogModelEntry() / inferWireApiForModel() with the unresolved utility model name. That means the exact same gpt-5.6-luna-utility case can still miss the catalog and fall back to completions in SDK mode. This needs to be normalized in the shared catalog lookup path and covered by a reflect-path regression.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 31.8 AIC · ⌖ 7.23 AIC · ⊞ 20.2K
_{Comment /review to run again}

github-actions · 2026-09-29T05:53:21Z

-      }
-      return;
+  const lookupModelNames = [normalizedModelName];
+  if (normalizedModelName.endsWith("-utility")) {


This only fixes the env-based lookup path; SDK/reflect mode still misses *-utility models and will keep selecting the wrong wire API.

💡 Why this still breaks and how to fix it

applyCopilotWireAPI() now falls back from gpt-5.6-luna-utility to gpt-5.6-luna, but the SDK path still resolves its provider config through resolveMultiProviderFromReflect() → getCatalogModelEntry() / inferWireApiForModel(), and those helpers still require an exact catalog key. In that flow the utility model still misses models.json, inferWireApiForModel() falls back to completions, and the original wrong-endpoint failure survives whenever reflect-driven multi-provider resolution is active.

Please move the -utility normalization into the shared catalog lookup (or a shared helper used by both call sites) and add a regression that exercises the reflect/multi-provider path, not just applyCopilotWireAPI().

Addressed in 2dc44d0: getCatalogModelEntry() now resolves utility variants through the base model, and the reflect multi-provider path has a regression test.

github-actions

Skills-Based Review 🧠

Applied /tdd — the fix logic itself is sound (correctly falls through the lookup list and returns early on the first hit, whether exact or fallback), but the new test only covers the happy path.

📋 Key Themes & Highlights

Key Themes

Precedence untested: the PR description states "Exact matches ... retain precedence" over the -utility fallback, but no test proves this — lookupModelNames = [exact, base] combined with return on first match does enforce it today, but nothing guards against a future reordering regression.
Boundary case untested: when both the exact utility key and its base are absent from the catalog, the loop should exit silently; there's no test locking that in.

Positive Highlights

✅ Minimal, well-scoped diff — reuses the existing Object.entries(...).find(...) case-insensitive lookup pattern instead of introducing a new one.
✅ Correctly preserves the "return after first match" semantics from the original code, avoiding double-processing.
✅ Query-parameter stripping (?effort=high) still applies before the -utility suffix check, so gpt-5.6-luna-utility?effort=high would also resolve correctly.

Suggested tests are in the inline comments. Neither blocks merge — the implementation is correct as written.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 36.5 AIC · ⌖ 14.5 AIC · ⊞ 10K
_{Comment /matt to run again}

github-actions · 2026-09-29T05:53:36Z

+    it("uses the catalog wire API for a Copilot utility model variant", () => {
+      process.env.COPILOT_MODEL = "gpt-5.6-luna-utility";
+      applyCopilotWireAPI({ modelsJson: makeModelsJson(), logger: () => {} });
+      expect(process.env.COPILOT_PROVIDER_WIRE_API).toBe("responses");


[/tdd] The PR description explicitly promises "Exact matches ... retain precedence" over the utility-suffix fallback, but no test asserts this. Add a regression test proving the fix's stated contract, not just the happy path.

💡 Suggested test

it("prefers an exact catalog match over the utility-suffix fallback", () => { process.env.COPILOT_MODEL = "gpt-5.6-luna-utility"; const modelsJson = makeModelsJson(); modelsJson.providers["github-copilot"].models["gpt-5.6-luna-utility"] = { wire_api: "completions" }; applyCopilotWireAPI({ modelsJson, logger: () => {} }); expect(process.env.COPILOT_PROVIDER_WIRE_API).toBe("completions"); });

Without this test, a future refactor of lookupModelNames order could silently reverse precedence and route utility-model traffic to the wrong wire API — exactly the class of bug this PR fixes.

@copilot please address this.

Added an exact-match-over-base regression for applyCopilotWireAPI; the shared lookup preserves this precedence (2dc44d0).

github-actions · 2026-09-29T05:53:36Z

+    it("uses the catalog wire API for a Copilot utility model variant", () => {
+      process.env.COPILOT_MODEL = "gpt-5.6-luna-utility";
+      applyCopilotWireAPI({ modelsJson: makeModelsJson(), logger: () => {} });
+      expect(process.env.COPILOT_PROVIDER_WIRE_API).toBe("responses");


[/tdd] No test covers the case where a -utility model's base is also absent from the catalog — the fallback loop should still leave COPILOT_PROVIDER_WIRE_API unset rather than throwing or matching something unintended.

💡 Suggested test

it("leaves COPILOT_PROVIDER_WIRE_API unset when a utility model's base is also unknown", () => { process.env.COPILOT_MODEL = "totally-unknown-model-utility"; applyCopilotWireAPI({ modelsJson: makeModelsJson(), logger: () => {} }); expect(process.env.COPILOT_PROVIDER_WIRE_API).toBeUndefined(); });

This boundary case exercises the full lookupModelNames loop (both entries miss) and guards against a future change accidentally matching on a partial/incorrect key.

@copilot please address this.

Added a regression confirming an unknown utility model with an unknown base leaves the wire API unset (2dc44d0).

github-actions

Reviewed with Impeccable audit mode (small, well-scoped bug_fix diff — no UI surface). Confirmed the utility-suffix fallback logic manually:

gpt-5.6-luna-utility → strips -utility → matches gpt-5.6-luna → correctly sets COPILOT_PROVIDER_WIRE_API=responses.
Exact catalog matches still take precedence over the fallback (the fallback list is only appended, not prioritized).
?query params are stripped before the suffix check, so gpt-5.6-luna-utility?effort=high resolves correctly too.
No match for either the exact or base name silently falls through (no throw), consistent with prior behavior.
New regression test covers the fallback path directly.

No blocking issues found.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 33.4 AIC · ⌖ 13 AIC · ⊞ 8.1K

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-09-29T05:59:11Z

@copilot copilot new default for gpt models is responses

Added a responses fallback for GPT models when the catalog does not specify a wire API, while preserving catalog values and explicit overrides. Committed as ce1e945.

gh-aw-bot · 2026-09-29T06:03:12Z

@copilot Please address the remaining requested review feedback on the utility-model wire API fallback, refresh the branch if needed, and run the pr-finisher skill.

review feedback: PR Code Quality Reviewer noted the reflect/SDK path still misses *-utility fallback handling, so the shared catalog lookup/inference path needs the same normalization plus regression coverage for the reflect-driven SDK path.

Generated by 👨🍳 PR Sous Chef

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 20.5 AIC · ⌖ 8.82 AIC · ⊞ 9.4K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-09-29T06:22:52Z

@copilot Please address the remaining requested review feedback on the utility-model wire API fallback, refresh the branch if needed, and run the pr-finisher skill.

review feedback: PR Code Qualit...

The shared catalog lookup now handles utility IDs in the reflect-driven SDK path, with regression coverage for that resolver. Committed as 2dc44d0.

github-actions · 2026-09-30T17:25:05Z

Initial plan

730e6e1

Copilot AI assigned Copilot and pelikhan Sep 29, 2026

Copilot started work on behalf of pelikhan September 29, 2026 05:06

View session

Copilot AI linked an issue Sep 29, 2026 that may be closed by this pull request

Auto model fails for org billing #63117

Closed

Infer Copilot utility model wire API

74ac984

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix auto model failure for org billing~~ Infer Copilot wire API for utility model variants Sep 29, 2026

Copilot finished work on behalf of pelikhan September 29, 2026 05:15

Copilot AI requested a review from pelikhan September 29, 2026 05:15

pelikhan marked this pull request as ready for review September 29, 2026 05:45

Copilot AI balanced review requested due to automatic review settings September 29, 2026 05:45

Copilot started reviewing on behalf of pelikhan September 29, 2026 05:46

View session

github-actions Bot reviewed Sep 29, 2026

View reviewed changes

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Copilot started work on behalf of pelikhan September 29, 2026 05:51

View session

github-actions Bot requested changes Sep 29, 2026

View reviewed changes

github-actions Bot reviewed Sep 29, 2026

View reviewed changes

github-actions Bot mentioned this pull request Sep 29, 2026

[pr-review] Daily PR Code Quality Review — 36527942649 #64197

Closed

github-actions Bot approved these changes Sep 29, 2026

View reviewed changes

Default GPT Copilot models to responses API

ce1e945

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan September 29, 2026 05:59

github-actions Bot mentioned this pull request Sep 29, 2026

[pr-sous-chef] PR Sous Chef run report #64101

Closed

Copilot started work on behalf of gh-aw-bot September 29, 2026 06:04

View session

Share utility model catalog fallback

2dc44d0

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot September 29, 2026 06:23

Copilot AI requested a review from gh-aw-bot September 29, 2026 06:23

pelikhan mentioned this pull request Sep 29, 2026

Wire mode github/gh-aw-firewall#9178

Closed

pelikhan merged commit cd5c226 into main Sep 29, 2026
11 checks passed

pelikhan deleted the copilot/fix-auto-model-billing-issue branch September 29, 2026 12:02

Copilot AI mentioned this pull request Sep 29, 2026

Reflect known model wire API modes github/gh-aw-firewall#9197

Merged

This was referenced Oct 10, 2026

Copilot harness: choose the wire API from AWF endpoint data, fail fast on a model/endpoint mismatch, and report mismatch errors clearly #67485

Closed

agentic-workflows skill: add a checklist for model and engine misconfiguration #67486

Closed

This was referenced Oct 10, 2026

Select Copilot wire APIs from AWF metadata and surface model mismatches #67487

Merged

Add model and engine misconfiguration diagnostics to agentic-workflows #67488

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

Copilot AI commented Sep 29, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

github-actions Bot commented Sep 29, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Request changes

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Sep 29, 2026

Uh oh!

gh-aw-bot commented Sep 29, 2026

Uh oh!

Copilot AI commented Sep 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

Copilot AI commented Sep 29, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading