fix(agentHost): ignore the failed execution's idle in the next Copilot turn by Yoyokrazy · Pull Request #340421 · microsoft/vscode · GitHub
Skip to content

fix(agentHost): ignore the failed execution's idle in the next Copilot turn - #340421

Merged
roblourens merged 3 commits into
mainfrom
agents/investigate-flaky-copilot-agent-test
Oct 9, 2026
Merged

roblourens merged 3 commits into
mainfrom
agents/investigate-flaky-copilot-agent-test

Conversation

@Yoyokrazy

@Yoyokrazy Michael Lively (Yoyokrazy) commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes the flaky Agent Host E2E test Copilot › provider errors: bad request remains classified and allows a subsequent turn ('' !== 'RECOVERED'), seen on #340401, #340407 and #339896.

  1. A root session.error now records that the failed execution's terminal session.idle is still outstanding.
  2. That idle is ignored while the failed-execution marker is outstanding, instead of completing a replacement empty. A child's turn-start cannot bypass the guard.
  3. The marker clears at the next root boundary (user.message echo, assistant.turn_start, turn-starting notification), or is consumed by the failed execution's root idle.
Root cause
  1. wrapper.onSessionError clears the active turn as soon as a root error arrives, so the client sees chat/error and can send the next turn right away.
  2. The SDK still emits the failed execution's session.idle afterwards. wrapper.onIdle completes even a pending turn (defensive handling for degenerate no-op sends), so a late idle completed the new turn with no response. send() then logged Turn changed during preparation; dropping send and the prompt never reached the model.
  3. Failing CI trace (job 113088628120): Session error … 400 at 44.377 → chat/error 44.381 → after-error turn started 44.388 → sendMessage called 44.421 → Session idle 44.422 → after-error turnComplete 44.426 → Turn changed during preparation; dropping send 44.483.
Session Context
  • This is a real product race, not just a test problem. A user who retries right after a classified error can hit it too: the retry finishes as an empty turn and the prompt is silently dropped.
  • Root execution ownership. The existing resumed-turn guard protects zero-message continuations. Normal replacement sends additionally track the failed execution's pending root idle. That marker, rather than providerTurnStarted, is authoritative: a background child's turn-start also changes the latter flag, without establishing a new root boundary.
  • Only the turn completion is skipped. The stale idle still runs its existing side effects (Fusion phase chat completion, activity clear), so the current behavior of a phase chat reopened while its close still drains an edit stays open is unchanged.
  • Compatibility. Root boundary events clear the marker when the next execution starts, including when the old idle is missing. Once the failed execution's idle is consumed, valid zero-output completions and pre-consumption errors do not require a user/assistant event. Regression coverage retains these cases and the existing cancellation tests.
  • Bounded host mitigation, not a general SDK completion contract. This protects the confirmed failed-execution tail. The separate upstream investigation, github/copilot-agent-runtime#25706, found that generic SDK completion needs authoritative request correlation to handle every pre-consumption path; this PR does not implement a blanket consumed-user-message gate or claim to solve that broader API ambiguity.
  • Not caused by agentHost: advertise only Copilot models over Mission Control #340401. That PR only gates Mission Control model advertisement; the test uses the local AHP connection.
Changes
File Δ
src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts Add _failedExecutionIdlePending. Set it on root session.error; consume it on the next root idle; clear it only on new root execution boundaries. Protect the replacement independently of a child changing the root turn's started flag.
src/vs/platform/agentHost/test/node/copilotAgentSession.test.ts Stale-idle cases during preparation and after dispatch, each with/without an intervening child turn-start; zero-output completion for three old-idle timings; pre-consumption error handling; missing old idle followed by a new root boundary.
src/vs/platform/agentHost/test/node/e2e/README.md Document the failure sequence and root-versus-child ownership invariant, preserving strict response assertions.
Verification
  • The original new unit tests fail without the original fix. On original PR head 7e88f0be3155, the added child-start cases reproduce the review finding: 7 focused tests passed, 2 failed. With fcf1bd35b53c, all 9 focused ownership/compatibility tests passed.
  • env -u COPILOT_HOME VSCODE_SKIP_PRELAUNCH=1 ./scripts/test.sh --run src/vs/platform/agentHost/test/node/copilotAgentSession.test.ts --run src/vs/platform/agentHost/test/node/copilotAgent.test.ts --reporter dot — final merged revision 7b9aadf2922: 1579 passed, 1 existing pending. The ambient developer COPILOT_HOME must be unset so unit tests use their injected homes. Earlier takeover revision had 1490 passes; an unrelated startup-migration notification test failed once under combined load, passed on isolated unchanged-baseline comparison, and subsequent full runs passed. No migration code/test was edited.
  • env -u COPILOT_HOME VSCODE_SKIP_PRELAUNCH=1 ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotAgentHostE2E.integrationTest.ts --grep 'provider errors:|resumes a failed turn in place|resumes the same turn after repeated failures' --reporter dot — five final-revision executions, all 6 tests passed each time: 30 strict-replay passes.
  • npm run eslint -- src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts src/vs/platform/agentHost/test/node/copilotAgentSession.test.ts — passed.
  • node build/lib/formatter.ts --verify src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts src/vs/platform/agentHost/test/node/copilotAgentSession.test.ts and git diff --check — passed.
  • npm run typecheck-client — passed on the final merged revision. The older PR base had one unrelated Auto-tier test incompatibility with the shared installed SDK; incorporating current main resolved that baseline mismatch without changing shared dependencies or adding an unrelated fix.
  • Couldn't reproduce the E2E race locally on unfixed code (0/24 under CPU load; this machine delivers the idle before the next dispatch), so the deterministic unit test is the regression guard.
  • Main was merged without rewriting the existing PR history. The root user-message conflict preserves both the failed-idle marker reset and main's managed-plugin failure display. Net changes against main remain the mitigation, regression tests, and directly related documentation only.
  • Final-revision GitHub 37865257477 completed green overall at 7b9aadf2922, including Compile & Hygiene and all three Electron integration jobs. A Copilot extension test setup dependency download timed out (ETIMEDOUT), then passed on its isolated retry without code changes.
  • Both independent definition-111 packaged Windows/Linux/macOS Electron integration matrices passed at the same revision: 482533 and 482534. Task logs explicitly confirm all four Copilot error/recovery cases passing on every execution (24 packaged recovery passes). Both builds completed green overall; 482534 required the isolated Windows smoke retry described below. Publishing and release are disabled. Old-base runs 482527/482528 were canceled as superseded.
  • The GitHub Linux Electron-Smoke model-picker failure (272K, detached DOM) also occurred identically on unchanged main 482526. Its isolated retry passed without code changes. No UI or smoke code changes and no required-check bypass.
  • Azure 482534's Windows Electron-Smoke test Starts and reopens a Dev Container session over WSL initially timed out waiting for DEV_CONTAINER_WSL_RESPONSE. The same test passed in 482533 at the identical head. Failure artifacts are preserved; the mock model request arrived after the response assertion had already timed out. The isolated Windows retry (forceRetryAllJobs=false) explicitly executed and passed this test in 57.970 seconds, with 92 smoke tests passing and no code changes. Both the build and all reported PR checks are now green. This is not the empty RECOVERED signature; the original smoke timeout's root cause remains unclassified despite the passing retry.

(Written by Copilot)

…t turn

A root session.error ends the protocol turn immediately, but the SDK emits
that failed execution's terminal session.idle afterwards. When the client
starts the next turn first, the stale idle completed the new pending turn
empty and its send was dropped during preparation. Track the outstanding
idle and ignore it until the next turn's provider boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 03:10
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Base: 7537e22c Current: 34273be5

No screenshot changes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A subagent turn-start can bypass the new guard, allowing the stale idle to complete and drop a retry.

1 open finding
What changed in this PR

Prevents a failed Copilot execution’s delayed idle event from prematurely completing a retry in Agent Host.

Changes:

  • Tracks the outstanding idle and clears it at root turn boundaries.
  • Adds regression tests for retry timing and missing terminal idle events.
File Description
src/​vs/​platform/​agentHost/​test/​node/​copilotAgentSession.test.ts Tests delayed-idle recovery and completion safety.
src/​vs/​platform/​agentHost/​node/​copilot/​copilotAgentSession.ts Adds failed-execution idle tracking and filtering.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts Outdated
roblourens and others added 2 commits October 8, 2026 17:12
Ignore the preceding failed execution's idle independently of child turn-start state. Cover child events during preparation and after admission, zero-output recovery and pre-consumption errors without weakening lifecycle assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve both the root failed-idle boundary reset and managed-plugin failure handling. Revalidated 1579 Copilot unit tests and 30 strict error/recovery and resume executions on the merged revision.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@roblourens
roblourens merged commit 1bcc86e into main Oct 9, 2026
98 of 103 checks passed
@roblourens
roblourens deleted the agents/investigate-flaky-copilot-agent-test branch October 9, 2026 02:28
@vs-code-engineering vs-code-engineering Bot added this to the 1.142.0 milestone Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants