🔬 Test Quality Sentinel is analyzing test quality on this pull request...

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · 11.2 AIC · ◇ 14.1 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · 24.4 AIC · ⌖ 5.39 AIC · ⊞ 14.7K · ◷

Generated by Ponytail Reviewer for #70389

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (actions/setup/js/copilot_harness.cjs:676): This scans every .md file, so a repository file such as .github/agents/README.md or another non-.agent.md document with a model: key can now cause startup to fail even though the SDK loader (and the custom-agent convention) only loads .agent.md files. That makes CLI and SDK behavior inconsistent and introduces false blocking failures from files Copilot will ignore. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
3. Review (actions/setup/js/copilot_harness.cjs:1689): This resolves and potentially rejects every CLI sub-agent before the existing noop short-circuit below runs. A workflow that has already emitted a noop can now fail with model_unavailable (or even a reflect refresh error) instead of exiting successfully without starting Copilot, contradicting the preflight contract documented immediately below and changing the no-op path into a model-dependent path. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
4. Review (actions/setup/js/copilot_harness.cjs:677): [/codebase-design] This CLI-mode agent-file lister accepts any *.md file, but the SDK-mode loader (loadCopilotSDKCustomAgents in extract_inline_sub_agents.cjs) only accepts *.agent.md, and the docs (engines.md line 609) say Copilot "treats .github/agents/*.agent.md files as custom agents." This repo already has non-agent .md files sitting in .github/agents/ (developer.instructions.md, agentic-workflows.md). - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
5. Review (actions/setup/js/copilot_harness.cjs:871): [/tdd] hideTrackedAgentFileRewrite is the safety net that keeps a rewritten tracked agent file out of git status/commits, but no test exercises the branch where git ls-files/git update-index --skip-worktree fails (non-zero exit or spawn error), which is what triggers the restore-original-content-and-exit(1) path in applyCopilotCLISubagentModels. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
6. Review (actions/setup/js/copilot_harness.cjs:711): [P2] Replace the complete YAML model scalar, not only its first line. The loader accepts model: >- followed by an indented small, but this rewrite leaves that continuation beneath model: "claude-haiku-5.5", producing invalid frontmatter. Copilot then starts with a corrupted agent file despite a successful resolution record. Replace the scalar's full source range while preserving other fields and the body, or reject unsupported multiline forms before modifying the file. Add a multiline-model CLI regression case. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
7. Review (actions/setup/js/copilot_harness.cjs:742): Preserve the existing inherited sentinel. The Copilot SDK workflow .github/workflows/dependabot-burner.md:405 declares it for dependabot-remediator, and model_pin_check.cjs:681 explicitly treats it as using the parent's model. This filter now sends it through literal-id validation, where it fails with model_unavailable and aborts the workflow at startup. Either remove that declaration from the workflow and recompile, or treat inherited as no model in both the harness and SDK session so it inherits the main model without a resolution record. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
8. Review (actions/setup/js/copilot_harness.cjs:801): [P1] Preserve provider scope through the catalogue check and SDK qualification. normalizeForCopilotCLI removes copilot/, so copilot/gpt-5.4-mini passes this check when only OpenAI advertises that id, and the SDK then selects openai/gpt-5.4-mini. The resolution record still reports Copilot. If both providers advertise the id, qualification instead throws an ambiguity error despite the explicit prefix. Use the resolver's provider identity for validation and SDK qualification; keep the bare id for CLI dispatch. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
9. Review (actions/setup/js/copilot_harness.cjs:801): [P2] Handle qualification exceptions through the failure-recording path. qualifyModelForMultiProvider throws rather than returning null when two configured Copilot endpoints advertise the same sub-agent id. That exception bypasses fail(): this agent keeps the successful record already written by resolveEngineModel, and later agents get no record. Catch the qualification error, record a failed resolution, and continue collecting results before throwing the aggregated failure. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
10. Review (actions/setup/js/copilot_harness.cjs:801): [P2] Refresh the SDK provider configuration along with /reflect. If the main model is present initially but a sub-agent model appears only after refresh, the liveness check succeeds while this lookup rejects the model against the old providerConfig.models. The driver JSON was also serialized before sub-agent resolution. Rebuild and propagate the provider configuration from the refreshed catalogue before qualification and driver startup. Extend case 3c to SDK mode; it currently omits providerConfig. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
11. Review (actions/setup/js/copilot_harness.cjs:836): [P2] Match CLI agent discovery to the effective engine working directory. process_runner.cjs:162 launches Copilot in GH_AW_ENGINE_CWD, but this code always scans GITHUB_WORKSPACE. When engine.cwd points at another checkout, that checkout's .github/agents models bypass resolution and recording, while unrelated workspace agents can abort startup. Use the CLI's actual discovery paths and precedence, including the restored inline-agent location. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
12. Review (actions/setup/js/copilot_harness.cjs:851): [P2] Skip-worktree hides the agent's edits as well as the harness rewrite. In a non-sparse checkout, once a tracked agent file's model is rewritten, later prompt or tool changes to that file are omitted by git add -A, git commit -a, and working-tree patch collection. Nothing clears the flag or reconciles the file after Copilot exits. A workflow tasked with updating .github/agents can therefore silently publish an incomplete change. Keep runtime agent definitions separate from tracked sources, or preserve agent edits while removing only the harness substitution and its index flag before changes are captured. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
13. Review (actions/setup/js/copilot_harness.cjs:876): [P2] Do not treat a failed tracking check as proof that the file is untracked. A spawn error or git ls-files failure, such as exit 128 for unsafe ownership or an unreadable index, returns success here. The caller then leaves a rewritten tracked file without skip-worktree protection. A later process with working repository access can include it in commits or patches. Distinguish confirmed untracked/non-checkout cases from unexpected Git errors; return false for indeterminate checks so the caller restores the file and aborts. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
14. Review (actions/setup/js/copilot_harness.cjs:882): [P2] Verify that skip-worktree actually took effect. With cone-mode sparse checkout that materializes .github, Git can clear this bit for a file present in the working tree even though update-index --skip-worktree exits 0. Returning true then allows the rewritten model into subsequent commits and safe-output patches. Sparse checkout is supported by checkout.sparse-checkout. Check the resulting index flag and return false if it is absent, so the caller restores the original file and stops. - Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (comment)
15. Fix failing check agent (FAILURE): https://github.com/github/gh-aw/actions/runs/39347479634/job/118384702986.
16. Fix failing check conclusion (FAILURE): https://github.com/github/gh-aw/actions/runs/39347479634/job/118385883190.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef state: a78ac41f377c4917ec6ac2c621f496cdb22cd3fb707d550007467f7a31bd0d12

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 12.4 AIC · ⌖ 7.06 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

Review of b81266d7fc: three design changes before end-to-end runs

I reviewed the head commit, ran the five affected suites in a clean worktree (628 tests pass), and ran the new functions against the recorded /reflect capture and alias map from run 39304573724. The resolution logic itself works: small → claude-haiku-5.5, large → claude-sonnet-5.5, copilot/opus → claude-opus-5.5, and the dead gpt-5.4-mini fails closed. Three design problems remain. The first comes from the issue we wrote (#70388, step 1), so it's on us, not the implementation.

1. Only workflow-declared sub-agents should fail closed (blocking)

The PR resolves every .github/agents/*.agent.md in the checkout and stops the run when any one of them doesn't resolve, even if the workflow never uses that agent. Those files are shared with VS Code and the Copilot coding agent, and VS Code pins models by display name:

• GitHub code search finds about 7,900 .github/agents Markdown files containing (copilot). Examples: model: GPT-5.4 (copilot) in adamgell/cmtraceopen planner.agent.md, and model: GPT-5.2-Codex (copilot) in PEZ/epupp commit.agent.md.
• Against run 39304573724's catalogue, Claude Sonnet 4.5 (copilot), GPT-5 (copilot) and Auto all fail with model_unavailable.

So after this PR, every gh-aw Copilot workflow (CLI or SDK) in such a repository would fail at startup, even though nothing in the workflow changed. gh-aw's own .github/agents files declare no model:, which is why our CI doesn't show this.

Requested behaviour:

• Declared sub-agents (inline ## agent: blocks, staged under <tmp>/gh-aw/.github/agents): keep this PR's behaviour. Resolve, record, and fail closed.
• Other repository agent files: resolve when possible and use the resolved id. When resolution fails, log a warning, record model.resolution with status: failed and the cause, and leave that agent as it is today: CLI gets the file unchanged, and SDK inherits the session model as before. Do not stop the run.

@pelikhan, does that split match how you want repository agent files treated? The alternative is to fail closed on every agent file in the checkout, as the PR does now.

2. Skip sub-agent resolution in threat detection (blocking)

The detection job runs "Checkout repository for patch context" and then the same harness in CLI mode (smoke-copilot.lock.yml around L3179 and L3378). Detection never uses sub-agents, but with this PR it resolves (and in CLI mode rewrites) every repository agent file, so problem 1 also breaks threat detection. The harness already has isDetectionPhase(); sub-agent resolution should not run in that phase.

3. CLI mode: write resolved copies to Copilot's personal agents directory instead of rewriting tracked files (blocking)

The in-place rewrite plus git update-index --skip-worktree plus restore is about 420 lines and produced 13 review threads of git edge cases. Two problems are still there at this head:

• The agent's own edits to a rewritten agent file are lost. While Copilot runs, skip-worktree hides the file from git add -A / git commit -a. create_pull_request builds its patch with git format-patch from the agent's commits during the run, so an edit to that agent file never reaches the PR. Clearing the flag after the run is too late. The test "clears skip-worktree after CLI and preserves edits to a tracked agent file" asserts this: it expects git status --porcelain to be empty after the agent's edit.
• Restore is not byte-identical for block scalars. A tracked file with model: >- followed by an indented small comes back with an extra blank line after the scalar, so the file stays modified after the run. CRLF, BOM, quoted values and trailing comments do round-trip.

Copilot CLI documents a simpler mechanism (Creating custom agents for Copilot CLI): "If personal and repository custom agents have the same ID, the personal agent is used." For an agent file, the ID is the file name without .agent.md. gh-aw already writes $HOME/.copilot/settings.json and mcp-config.json before Copilot starts. So in CLI mode the harness can write each resolved agent as a copy with the same file name in ${COPILOT_HOME:-$HOME/.copilot}/agents/, with only model: changed, and never touch the checkout. That removes hideTrackedAgentFileRewrite, restoreCopilotCLIAgentFiles, the skip-worktree handling and their tests. The block-scalar bug and the lost-edit bug both go away.

The end-to-end CLI run has to confirm that Copilot CLI 1.0.98 in offline BYOK mode under AWF loads the personal copy over the repository file. Its subagent.started event should show the resolved model, and git status should show no change under .github/agents.

Merge order with #70387

With this PR, smoke-copilot's file-summarizer (whose unexpanded ${{ experiments.subagent_model }} is fixed by #70387) fails at startup instead of mid-run. Today AWF resolves the expanded small/large at request time, so #70387 is safe on its own. Merge #70387 first, then this PR.

Non-blocking

• When the round is done, update the "Behaviour changes" section and engines.md to describe the declared/repository split and the personal-agents copy.
• The two github-actions CHANGES_REQUESTED reviews are from 7f35145678; I'll dismiss them once their points are verified at the new head.

Review of b81266d7fc: three design changes before end-to-end runs

I reviewed the head commit, ran the five affected suites in a clean worktree (628 tests pass), and ran the new functions again...

Addressed in b9f7f48 (after make merge-main):

1. Declared vs repository agents. listCopilotSubagents marks inline ## agent: blocks under <os.tmpdir()>/gh-aw/.github/agents as declared. An inline agent with the same name as a repository file overrides it. Declared sub-agents still resolve, record and fail closed in CLI and SDK mode, routed runs included. Every other .github/agents/*.agent.md in the engine working directory goes through the same resolver and live-catalogue check, and uses the resolved id when that succeeds. When it fails, the harness logs a warning naming the agent, file and model, and records failed with the cause and resolved_model: null. The run continues: CLI leaves that agent unchanged, and in SDK mode it gets the session model with the existing warning and subagent.model_unavailable event, as before this PR.
2. Detection. Sub-agent resolution is skipped when isDetectionPhase(GH_AW_PHASE) is true. Nothing is read, resolved, recorded or written in that phase.
3. CLI personal-agent copies. When the resolved id differs from the declared value, the harness writes a copy with the same file name to ${COPILOT_HOME:-$HOME/.copilot}/agents/. Only the full model: scalar changes. The checkout is never modified. hideTrackedAgentFileRewrite, restoreCopilotCLIAgentFiles, the skip-worktree logic, cliAgentFileRestorations and the finally restore have been removed, along with their tests. If a copy can't be written, a declared sub-agent exits 1 before Copilot starts; a repository agent logs a warning and the run continues.

New tests use the run 39304573724 fixtures:

• A repository agent with GPT-5.4 (copilot) records failed/model_unavailable, logs the warning and doesn't stop the run, in both CLI and SDK. In the SDK session it gets the session model.
• An inline gpt-5.4-mini sub-agent in the same setup still fails closed.
• A repository agent with small resolves to claude-haiku-5.5.
• CLI personal copies keep the file name and the resolved model, including a block scalar. The workspace files stay byte-identical and git status is clean.
• In the detection phase nothing is resolved, recorded or written, even with dead models in both repository and inline agents.

Checks run: make fmt, make lint, tsc, the harness, SDK session, SDK driver, inline sub-agent, model-pin and resolver suites (670 tests passing), and make agent-report-progress. make recompile produced no lock-file diffs. I updated engines.md and the PR body ("Behaviour changes", "CLI mode", and the note that #70387 should merge first).

The end-to-end check that Copilot CLI 1.0.98 under AWF loads the personal copy instead of the repository file still needs a maintainer to dispatch a CLI smoke run on this branch.

Verified at b9f7f48a3b

All three requested changes are in: only declared sub-agents fail closed, detection is skipped, and CLI mode writes personal-agent copies. The skip-worktree and restore code is gone. All 648 tests in the six affected JS suites pass locally, CI is green, and the PR touches only its 9 files.

End to end

1. SDK mode, dead pins: smoke-copilot-sub-agents 39499460077

The harness stopped before the SDK driver started (0 subagent.started). usage/aw_session.jsonl has:

Each dead pin gets its own failed record and the shared "The requested model is not supported" message. gpt-5-mini has also left the catalogue since 19 Oct, so it now fails as well (tracked with the other pins in #70367).

2. SDK mode, routed: smoke-copilot-sdk-routed-claude 39499464895

It stopped before the driver started. mini-whoami (gpt-5.4-mini) is recorded as failed / model_unavailable. There is no main record, because the main model comes from the router, as intended.

3. CLI mode: private sandbox run, compiled with --gh-aw-ref b9f7f48a3b

The workflow had:

• an inline quick-checker with model: small;
• an inline file-summarizer with model: claude-haiku-4.5;
• a repository .github/agents/conventions-helper.agent.md with model: GPT-5.4 (copilot), a VS Code display name.

The harness log shows:

warning: repository agent 'conventions-helper' (.../.github/agents/conventions-helper.agent.md) declares model 'GPT-5.4 (copilot)', which could not be resolved (cause: model_unavailable); Copilot CLI uses the agent file unchanged.
copilot sub-agent 'file-summarizer': model 'claude-haiku-4.5' -> 'claude-haiku-4.5'
copilot sub-agent 'quick-checker': model 'small' -> 'claude-haiku-5.5'
copilot sub-agent 'quick-checker': wrote model 'claude-haiku-5.5' to personal agent /home/runner/.copilot/agents/quick-checker.agent.md

The records are quick-checker small → claude-haiku-5.5 resolved, file-summarizer exact, conventions-helper failed / model_unavailable and main exact. The run succeeded.

• The personal-agent override works in Copilot CLI 1.0.98. The session's subagent.started for quick-checker shows model: claude-haiku-5.5 with taskModelSource: custom_agent_definition. Copilot CLI loaded the resolved copy, not the workspace file with small. The checkout was not modified.
• The repository agent no longer blocks the run. Before this round it would have stopped startup.

Unrelated failure in run 3. Both Claude sub-agents then failed with AWF's 400 Cannot translate Copilot request feature 'include' between Responses and Chat Completions. The main model gpt-5.6-luna is on /responses, and Copilot CLI sends one wire API for the whole session. This is the existing cross-family sub-agent limitation of Copilot CLI's BYOK mode, not caused by this PR: file-summarizer, whose model this PR doesn't change, fails the same way.

Review state

• The 13 earlier review threads are resolved. Each point is either still fixed at this head or no longer applies after the redesign.
• The two github-actions CHANGES_REQUESTED reviews from 7f35145678 are dismissed.

Before merge

• Merge Resolve experiment references in inline sub-agent frontmatter before writing the agent file; fail smoke-copilot when file-summarizer doesn't run #70387 first. On this branch, smoke-copilot's file-summarizer still declares the unexpanded ${{ experiments.subagent_model }}, so it now stops at startup.
• Six production SDK workflows change behaviour. The PR body lists their sub-agents moving from the main model to claude-haiku-5.5 (small) or claude-sonnet-5.5 (large).
• Scope question. @pelikhan, please confirm the declared/repository split from the earlier review comment: only workflow-declared sub-agents fail closed, and other repository agent files warn and record failed.