Claude on Copilot: pass Claude Code a modelOverrides map for Copilot's dotted Claude model ids by Copilot · Pull Request #68839 · github/gh-aw · GitHub
Skip to content

Claude on Copilot: pass Claude Code a modelOverrides map for Copilot's dotted Claude model ids - #68839

Merged
SivaKesava1 merged 8 commits into
mainfrom
copilot/smoke-claude-on-copilot-fix
Oct 15, 2026
Merged

SivaKesava1 merged 8 commits into
mainfrom
copilot/smoke-claude-on-copilot-fix

Conversation

Copilot AI commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Copilot writes Claude model versions with a dot (claude-haiku-4.5), but Anthropic uses a hyphen (claude-haiku-4-5). Claude Code doesn't recognise the Copilot id, so it falls back to guessed defaults, including reasoning effort high. Copilot rejects that effort for Haiku 4.5 with 400 invalid_reasoning_effort, which fails smoke-claude-on-copilot on every run. The other dotted models run on a guessed 200k-token context window.

Compiler (pkg/workflow)

  • claude_copilot_model_overrides.go (new): a static list of the Copilot Claude ids in the catalog. From it, the compiler builds a fixed-order --settings JSON value that maps each Anthropic id to the Copilot id for the 11 dotted ids. Ids that already match (claude-sonnet-5, claude-opus-5, claude-fable-5, claude-sonnet-4) get no entry.
  • claude_engine.go: when the resolved provider is GitHub (model-provider: github or a copilot/ prefix), --settings <map> is added to the Claude Code command. The map doesn't depend on the model, so fixed, run-time (ANTHROPIC_MODEL) and routed models all get it. The Anthropic provider's output is unchanged.
  • Workflow-provided --settings (--settings X or --settings=X, in engine.args or safe-outputs.threat-detection.engine.args): it is kept as is, nothing is added, and compilation emits a warning (configured_model_validation.go).
claude ... --settings '{"modelOverrides":{"claude-fable-5-1":"claude-fable-5.1","claude-haiku-4-5":"claude-haiku-4.5",...}}' ...

Threat detection

  • Default (Claude-on-Copilot workflow): still runs threat-detect --engine copilot, with no map.
  • threat-detection.engine: {id: claude, model-provider: github}, inline path (features.gh-aw-detection: false): uses gh-aw's Claude command builder, so it gets --settings directly.
  • Same engine, external threat-detect --engine claude path: threat-detect builds its own Claude Code arguments, so the detection step instead sets GH_AW_CLAUDE_SETTINGS. claude_harness.cjs appends it as --settings unless the arguments already have one. Covered from gh-aw only; github/gh-aw-threat-detection is unchanged. threat-detect launches Claude Code through the harness only when the harness file is present; without it, the map doesn't reach Claude Code.

Catalog drift guard

  • models.json is embedded in pkg/cli and isn't reachable from pkg/workflow. pkg/cli/model_costs_claude_copilot_ids_test.go fails if the compiler's list drifts from the github-copilot Claude ids in models.json. models.json itself is unchanged.

Lock files and docs

  • Recompiled the six Claude-on-Copilot smoke workflows (one-line agent-command change each).
  • docs/.../engines/claude.md documents the map and how a workflow-provided --settings is handled.

Notes for reviewers

  • Acceptance cases 1–2 need live runs: dispatch smoke-claude-on-copilot and smoke-claude-routed on this branch. smoke-claude-routed's allowed-models is intentionally left unchanged.
  • These failures are unrelated to this change:
    • pkg/cli and actions/setup session-parser tests fail with Cannot find module './otlp_header_helpers.cjs'. Two of them were re-run without this change and fail the same way.
    • TestAgentExecutionNumericReaderParity also fails. It was not re-run without this change.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 14, 2026 22:28
Copilot AI max review requested due to automatic review settings October 14, 2026 22:29
Copilot AI changed the title [WIP] Fix modelOverrides map for Copilot's Claude model ids Claude on Copilot: pass Claude Code a modelOverrides map for Copilot's dotted Claude model ids Oct 14, 2026
Copilot AI requested a review from SivaKesava1 October 14, 2026 22:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

External Claude detection can still receive neither workflow-provided settings nor the required model overrides.

2 open findings
What changed in this PR

Adds Claude model-ID overrides to gh-aw’s Copilot inference path so Claude Code uses the correct reasoning and context defaults.

Changes:

  • Add a static override map and warnings for workflow-provided settings.
  • Add detector-harness transport and compiler, harness, and catalog tests.
  • Document the behavior and regenerate six smoke workflows.
File Description
pkg/​workflow/​configured_model_validation.go Warn about custom Claude settings.
pkg/​workflow/​compiler_orchestrator_workflow.go Invoke settings validation.
pkg/​workflow/​claude_engine.go Pass overrides through commands and detector environment.
pkg/​workflow/​claude_copilot_model_overrides.go Define deterministic model-ID mappings.
pkg/​workflow/​claude_copilot_model_overrides_test.go Cover provider, routing, settings, and detection cases.
pkg/​cli/​model_costs_claude_copilot_ids_test.go Guard against catalog drift.
docs/​src/​content/​docs/​engines/​claude.md Explain overrides and custom settings.
actions/​setup/​js/​claude_harness.test.cjs Test environment-based settings injection.
actions/​setup/​js/​claude_harness.cjs Append settings supplied through the environment.
.github/​workflows/​smoke-github-claude.lock.yml Refresh GitHub-provider smoke command.
.github/​workflows/​smoke-claude-routed.lock.yml Refresh routed-model smoke command.
.github/​workflows/​smoke-claude-on-copilot.lock.yml Refresh Haiku-on-Copilot smoke command.
.github/​workflows/​smoke-claude-copilot.lock.yml Refresh dynamic-model canary command.
.github/​workflows/​smoke-claude-copilot-auto.lock.yml Refresh Copilot auto-selection command.
.github/​workflows/​smoke-claude-auto.lock.yml Refresh auto-selection smoke command.

🧠 Review effort: Max


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/workflow/claude_engine.go Outdated
Comment thread pkg/workflow/configured_model_validation.go Outdated
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (pkg/workflow/claude_engine.go:498): When safe-outputs.threat-detection.engine.args supplies --settings, this guard omits GH_AW_CLAUDE_SETTINGS. The external detector copies only the generated environment (threat_detection_external.go:605–610), and buildThreatDetectCommand does not forward those arguments. Claude therefore receives neither the supplied settings nor the automatic map, even if the supplied value contains the correct Haiku override. Following the new warning cannot fix this path. Forward the supplied settings through the existing environment channel, using the automatic map only when no override exists, an... - Claude on Copilot: pass Claude Code a modelOverrides map for Copilot's dotted Claude model ids #68839 (comment)
  3. Review (pkg/workflow/configured_model_validation.go:76): With main model copilot/claude-haiku-4.5 and detection engine {id: claude, args: ["--settings", "{}"]} but no detection model/provider, both detector paths inherit the main model and use GitHub inference. This temporary workflow instead has an empty model, resolves to Anthropic, and emits no warning. In particular, inline detection runs without the automatic map and without the warning explaining how to supply it. Use the same inheritedDetectionModel fallback as execution before resolving the provider, and cover this inherited-model case. - Claude on Copilot: pass Claude Code a modelOverrides map for Copilot's dotted Claude model ids #68839 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 1281085
Sous-chef work: 38ba8efae4d0df62418da03733a29e90312b12a5270ddae3b6f361f8b5c0d62d 6a7c631158823c8ae5add21bfb9bc0a4ba1b80dc4995d0c2631750cb7e6d877c
Sous-chef state: 88be2386acc2f811b7e869f04c1dd9a8199dcb842578cfdc9f567cce6ac8854d

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.99 AIC · ⌖ 12 AIC · ⊞ 1.1K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits October 15, 2026 01:05
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
…ed detection model for warning

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@SivaKesava1
SivaKesava1 merged commit d91f5c9 into main Oct 15, 2026
56 of 60 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/smoke-claude-on-copilot-fix branch October 15, 2026 04:14
@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claude on Copilot: pass Claude Code a modelOverrides map for Copilot's dotted Claude model ids (smoke-claude-on-copilot 400 invalid_reasoning_effort)

4 participants