Repository navigation
Fix stale Copilot workflow model pins exposed by fail-closed sub-agent resolution - #70652
Conversation
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
There was a problem hiding this comment.
Verdict: no blocking findings
Reviewed the changed workflow pins, sub-agent evidence assertions, and the new repository pin regression coverage. I found no actionable correctness, security, performance, or maintainability issue in the changed lines. Verification was static only; smoke runs are not available in this review environment.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · 2.11 AIC · ⌖ 0.493 AIC · ⊞ 1.6K
Comment /review to run again
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /tdd — no actionable issues found; approving.
📋 Key Themes & Highlights
Key Themes
- Root cause correctly addressed: pins are updated to live catalogue IDs/aliases rather than papering over the fail-closed resolution error, matching /diagnosing-bugs guidance to fix the cause (#70389's stricter validation), not the symptom.
- Regression test is exemplary (model_pin_check.test.cjs): the new repository-pins-reflection-regression describe block pins the fixture by SHA-256, asserts on the report's classifications (not CLI exit code), and enumerates both the 21 fixed workflows and the one known remaining failure (video-analyzer.md). This is /tdd-aligned — it prevents silent regression to dead pins and reads as a spec of the expected catalogue state.
- Sub-agent assertion upgrade in smoke-copilot-sub-agents.md's post-step now validates the actual model.resolution event (configured_model, status, resolved_model) instead of only checking the served model string — this closes a gap where an alias could silently resolve to the wrong model while still looking exact by regex alone.
- I traced the mini alias chain (mini → haiku → copilot/haiku) against pkg/workflow/data/model_aliases.json and confirmed it resolves to claude-haiku-5.5 as claimed for daily-model-resolution.md's run-analyzer, and verified the model.resolution status vocabulary (resolved/exact) is consistent with resolve_model_alias.cjs.
Positive Highlights
- ✅ Clear, well-documented mapping table in the PR description of every pin change and why
- ✅ Fixture provenance is tracked in test_data/awf_reflect/README.md with run URL, commit, and shape description
- ✅ Honest scoping: explicitly leaves video-analyzer's dead pin as an owner decision rather than silently fixing it
- ✅ New model.resolution-based sub-agent tests (smoke_copilot_sub_agents.test.cjs) cover failure modes: missing resolution record, failed status, and resolved-model drift
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 79 AIC · ⌖ 14.9 AIC · ⊞ 10.2K
Comment /matt to run again
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-engine replacements and the production model change lack branch runtime verification, and confirmed review issues remain unresolved.
2 open issues
What changed in this PR
Updates gh-aw workflows to replace retired Copilot model pins while preserving fail-closed sub-agent resolution.
Changes:
- Replace stale main, sub-agent, fallback, and routing models with aliases or compatible live IDs.
- Synchronize smoke prompts, assertions, and generated workflows.
- Add catalogue-backed regression coverage and update affected tests.
| File | Description |
|---|---|
pkg/workflow/prompts_test.go |
Expect the production sub-agent’s mini alias. |
pkg/workflow/goose_runtime_test.go |
Update Goose’s expected model. |
actions/setup/js/test_data/awf_reflect/README.md |
Record catalogue fixture provenance. |
actions/setup/js/smoke_copilot_sub_agents.test.cjs |
Test sub-agent resolution evidence. |
actions/setup/js/pi_subagent_extension.test.cjs |
Update expected GPT identity. |
actions/setup/js/model_pin_check.test.cjs |
Add repository pin regression coverage. |
.github/workflows/smoke-work-queue.md |
Replace retired fallback models. |
.github/workflows/smoke-service-ports.md |
Replace retired fallback models. |
.github/workflows/smoke-pi.md |
Replace the retired GPT fallback. |
.github/workflows/smoke-pi-sub-agents.md |
Update GPT child model and replies. |
.github/workflows/smoke-pi-sub-agents.lock.yml |
Synchronize generated sub-agent metadata. |
.github/workflows/smoke-pi-routed.md |
Update GPT child and routing assertions. |
.github/workflows/smoke-pi-routed.lock.yml |
Synchronize generated assertions and metadata. |
.github/workflows/smoke-goose.md |
Select Chat Completions-compatible Haiku. |
.github/workflows/smoke-copilot.md |
Replace the retired GPT fallback. |
.github/workflows/smoke-copilot-sub-agents.md |
Update children and verify resolved models. |
.github/workflows/smoke-copilot-sub-agents.lock.yml |
Synchronize generated resolution assertions. |
.github/workflows/smoke-copilot-sdk.md |
Replace the retired SDK fallback. |
.github/workflows/smoke-copilot-sdk-routed-claude.md |
Update GPT child and require resolution evidence. |
.github/workflows/smoke-copilot-sdk-routed-claude.lock.yml |
Synchronize generated child assertions. |
.github/workflows/smoke-copilot-routed.md |
Remove the retired routing candidate. |
.github/workflows/smoke-copilot-routed.lock.yml |
Synchronize routing configuration and assertions. |
.github/workflows/smoke-copilot-arm.md |
Replace the retired GPT fallback. |
.github/workflows/smoke-aider.md |
Select Chat Completions-compatible Haiku. |
.github/workflows/engine-conformance-pydantic-ai.md |
Switch conformance model to Haiku. |
.github/workflows/engine-conformance-pydantic-ai.lock.yml |
Synchronize generated model and costs. |
.github/workflows/engine-conformance-pi.md |
Use the GPT-5 alias. |
.github/workflows/engine-conformance-opencode.md |
Switch conformance model to Haiku. |
.github/workflows/engine-conformance-opencode.lock.yml |
Synchronize generated model and costs. |
.github/workflows/engine-conformance-goose.md |
Switch conformance model to Haiku. |
.github/workflows/engine-conformance-goose.lock.yml |
Synchronize generated model and costs. |
.github/workflows/engine-conformance-deepseek-harness.md |
Switch conformance model to Haiku. |
.github/workflows/engine-conformance-deepseek-harness.lock.yml |
Synchronize generated model and costs. |
.github/workflows/engine-conformance-crush.md |
Select a live GPT Responses model. |
.github/workflows/engine-conformance-crush.lock.yml |
Synchronize generated model and costs. |
.github/workflows/engine-conformance-aider.md |
Switch conformance model to Haiku. |
.github/workflows/engine-conformance-aider.lock.yml |
Synchronize generated model and costs. |
.github/workflows/daily-model-resolution.md |
Move run-analyzer to the mini alias. |
.github/workflows/daily-model-resolution.lock.yml |
Synchronize production sub-agent metadata. |
Review effort: Max
End-to-end results at
|
… test paths Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>


On October 19,
gpt-5-mini,gpt-5.4,gpt-5.4-mini,gpt-5.5andgrok-4.5were removed from the Copilot catalogue. Since #70389, Copilot refuses to start when a workflow-declared sub-agent model doesn't resolve. Run 39506475007's catalogue (awf-reflect.json) shows 25 dead pins across the repository's workflows. This PR fixes the 18 from the issue and 6 same-kind pins that a maintainer added to scope. It leaves one:video-analyzer'scopilot/gemini-3.7-flash, which is the owner's decision.Alias definitions, the resolver, the harnesses, AWF configuration and the pin checker are unchanged.
How replacements were chosen
model:on Pi.fallback-models, routing candidates, routed-smoke sub-agents, and engines that don't resolve aliases.copilot/claude-haiku-5.5for every engine path that uses Chat Completions, since no GPT model in this catalogue advertises it. I picked haiku over sonnet so these workflows stay within their existingmax-ai-creditslimits (5 and 20)./responsesis kept in every sub-agent smoke.Sub-agents
smoke-copilot-sub-agentsgpt5-whoami(wasmini-whoami)gpt-5-minigpt-5alias, resolves togpt-5.6-lunasmoke-copilot-sub-agentsluna-whoami(wasmini54-whoami)gpt-5.4-minigpt-5.6-luna(exact)smoke-copilot-sdk-routed-claudemini-whoamigpt-5.4-minigpt-5.6-lunasmoke-pi-routedmini-whoamicopilot/gpt-5.4-minicopilot/gpt-5.3-codexsmoke-pi-sub-agentsluna-whoami(wasmini-whoami)copilot/gpt-5-minicopilot/gpt-5.6-lunadaily-model-resolution(production)run-analyzergpt-5.4-miniminialias, now runs onclaude-haiku-5.5Notes:
run-analyzermoves from GPT to Claude. No small GPT model remains in the catalogue, and the built-ingpt-5-minialias still finds no match.smoke-pi-routedthe only candidate isgpt-5.6-luna, which is why its sub-agent usesgpt-5.3-codex.smoke-pi-sub-agentsdeliberately pins exact IDs, so it stays exact.Sub-agent assertions
smoke-copilot-sub-agents: the post-step now reads each sub-agent'smodel.resolutionrecord. It requires statusresolvedfor the alias child andexactfor the others, then checks the served model and endpoint against the resolved model rather than the literal alias:smoke-copilot-sdk-routed-claude:resolutionRoles: ["mini-whoami"], so a successful resolution of the sub-agent is now required.Top-level
copilot/gpt-5.4pinssmoke-aider,smoke-goose(Chat Completions, no alias resolution)copilot/claude-haiku-5.5/responsesfor GPT models; no alias resolution)copilot/gpt-5.6-lunacopilot/gpt-5Fallbacks and routing
Each fallback differs from the workflow's main model and from the other fallback.
smoke-copilot,smoke-copilot-arm,smoke-piautogpt-5.6-luna,claude-sonnet-5smoke-copilot-sdkgpt-5, resolves togpt-5.6-lunaclaude-sonnet-5,gpt-5.3-codexsmoke-service-ports,smoke-work-queuegpt-5.3-codexgpt-5.6-luna,claude-haiku-5.5smoke-copilot-routed: droppedgpt-5.4-mini, soallowed-modelsis[gpt-5.6-luna, claude-haiku-4.5](GPT and Claude). BothallowedModelsassertion arrays match it.Regression test (
repository-pins-reflection-regression)actions/setup/js/test_data/awf_reflect/run-39506475007-awf-reflect.json, a byte-for-byte copy of the run'sawf-reflect.json. SHA-256 is3d9fa7af7e07e93100bb614e45eb85abafbd16a3bbb2cb62a687a04cb5fff9b3; it is checked for credentials and its source is recorded in that folder's README.exactoralias;{video-analyzer.md, model, copilot/gemini-3.7-flash}.smoke_copilot_sub_agents.test.cjs,pi_subagent_extension.test.cjs,goose_runtime_test.go,prompts_test.go.Verification still needed (maintainer dispatch on this branch)
None of these smoke runs has happened on the branch yet.
smoke-copilot-sub-agents(inline_strictordelegated_sequentialvariant)exactorresolved; served-model and endpoint assertions passsmoke-copilot-sdk-routed-claude/responsessmoke-pi-routed/responsesrequestsmoke-pi-sub-agentsengine-conformance-aider,smoke-aidermodel_unavailableand no endpoint 400 on Chat Completionssmoke-copilot-routedKnown failures on
mainmodernize(embedlit) findings inclaude_engine.go,codex_engine.goandcomment_memory.go. This PR doesn't touch those files.smoke-copilot: a separate shell/URL-denial failure, out of scope here.