You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2bbc29d
Browse filesBrowse the repository at this point in the historyBrowse files
agentHost: wait for MCP readiness in hook E2E tests (microsoft#338460)
* agentHost: wait for MCP readiness in hook E2E tests
Wait for the default chat's plugin MCP server to report Ready before replaying hook tool calls, and verify real tool completion results so recorded assistant text cannot mask a failed local invocation.
Document the readiness and primary-outcome rules in the Agent Host E2E skill.
(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* agentHost: clarify E2E authoring workflow
Separate the basic test authoring and evaluation steps from a curated set of broadly reusable quality principles, so future root-cause learnings can improve the skill without turning it into an incident log.
(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* agentHost: assert MCP hook tool success
Require successful tool completion as well as the expected result text in the plugin hook E2E scenarios, so failed local calls cannot satisfy the regression guard.
(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy file name to clipboardExpand all lines: .github/skills/agent-host-e2e-tests/SKILL.md
+30-21Lines changed: 30 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,28 +11,39 @@ These tests run the whole agent host end-to-end (real server, real bundled provi
11
11
12
12
It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the *workflows*; the README is the source of truth for *how it works*.
13
13
14
-
## Non-negotiable rules
14
+
## Core invariants
15
15
16
-
1.**Replay is default and strict.** No env var → serves committed fixtures, no token, no network. An unrecorded request is a hard cache miss that fails the run.
17
-
2.**A model-backed fixture's filename is derived from the test title** (`${provider}-${slug}.yaml`). Renaming such a test orphans its fixture — re-record after any rename. Tests explicitly registered with `hostOnlyTest(...)` share `captures/empty.yaml`.
18
-
3.**Recording needs a real token** (`GITHUB_TOKEN` or `gh auth token`) and talks to real CAPI. Only run it intentionally, with trivial/read-only prompts in temp dirs.
19
-
4.**Never hand-write or hand-edit fixture contents** (especially not secrets/paths). Fixtures are always produced by recording; normalization/redaction is the proxy's job.
20
-
5.**Gate, don't fight.** If a behavior can't replay deterministically, gate the test (see Workflow C) instead of loosening timeouts or the strict check.
21
-
6.**Track every disabled variant.** Keep `e2e/KNOWN_ISSUES.md` current with the test title, scope, expected and observed behavior, and a focused reproduction command. For suspected product bugs, begin with a self-contained explanation in complete sentences of what the user is trying to do, what fails, and the likely user impact; define feature-specific terms instead of relying on test names or implementation details. Record symptoms, not speculative root causes.
22
-
7.**Always verify new end-to-end tests in Azure DevOps.** Every new Agent Host E2E or real-provider integration test must pass a focused build from VS Code pipeline definition `111` before merge. Local runs and GitHub pull-request CI are useful signals, but neither substitutes for the Azure build because its packaged Electron path and runner environments differ.
16
+
1.**Replay is default and strict.** No env var serves committed fixtures without a token or network. An unrecorded request is a hard cache miss.
17
+
2.**Fixture names derive from test titles.** Renaming a model-backed test orphans `${provider}-${slug}.yaml`; re-record after a rename. Tests explicitly registered with `hostOnlyTest(...)` share `captures/empty.yaml`.
18
+
3.**Recording is intentional and credentialed.**`AGENT_HOST_REPLAY_RECORD=1` talks to real CAPI and needs `GITHUB_TOKEN` or `gh auth token`. Keep prompts trivial and read-only in temporary directories.
19
+
4.**Fixtures are generated, never hand-edited.** Fix normalization or redaction and re-record if a capture contains unstable or sensitive data.
20
+
5.**Disabled variants stay accountable.** Record every gate in `e2e/KNOWN_ISSUES.md` with its scope, expected and observed behavior, and focused reproduction command.
21
+
6.**Azure validation is required.** Every new Agent Host E2E or real-provider integration test must pass a focused build from VS Code pipeline definition `111` before merge.
23
22
24
-
## Workflow A — Add a cross-provider test
23
+
## Basic workflow: write and evaluate an E2E test
25
24
26
-
1. Add a test to the closest module under `e2e/suites/`, or create and register a focused suite module when the behavior is distinct. Use `hostOnlyTest(context, ...)` when crossing the model boundary would be a bug; otherwise use `test(...)`. Drive turns with `dispatchTurn(...)` + `context.client.waitForNotification(...)`; assert on AHP notifications, never on wall-clock timing.
27
-
2. Keep the prompt minimal and deterministic (fewer model turns → smaller, more robust fixtures).
28
-
3. Record fixtures for every enabled provider (Workflow B). Host-only tests need no per-test recording: the shared empty fixture remains strict and fails on any model request.
29
-
4.**Review the diff** (Workflow B step 3), then run the test in plain replay mode to confirm it's green, then commit the test + fixtures together.
30
-
5. Run the full deterministic suite and coverage workflow described in the E2E README.
31
-
6. Open or update a draft PR, then complete the cross-platform Azure validation in Workflow D before considering the tests ready to merge.
25
+
1.**Choose the boundary and tier.** Put provider-invariant AHP behavior in conformance, provider behavior in parity, and use `providerHostOnlyTest(...)` only when provider-specific behavior must not cross the model boundary.
26
+
2.**Add the test to the closest suite.** Create and register a focused suite module only when the behavior is distinct. Keep provider-specific assertions in the provider's `*.integrationTest.ts`.
27
+
3.**Implement the smallest deterministic scenario.** Keep prompts minimal, drive behavior over AHP, wait for the exact prerequisite and completion states, and drain every model-backed turn.
28
+
4.**Assert the contract's primary observable result.** Check the real protocol result or external side effect; snapshot additional traffic only when ordering, routing, or lifecycle is part of the contract.
29
+
5.**Record every enabled provider fixture.** Host-only tests use the shared strict empty fixture. Follow the recording workflow below for model-backed tests.
30
+
6.**Review generated artifacts.** Check fixture and snapshot diffs for the intended behavior, normalized paths, no credentials or usernames, and no unintended model or request changes.
31
+
7.**Evaluate locally.** Run the focused test in strict replay, run adjacent tests when helpers or shared state changed, repeat timing- or lifecycle-sensitive scenarios, then run the full deterministic suite and required coverage, type-check, hygiene, and layer checks from the E2E README.
32
+
8.**Evaluate in CI.** Open or update a draft PR and complete the cross-platform Azure validation below. Timing, process-lifecycle, filesystem-watching, reconnect, restart, and worktree tests require two clean executions on every supported platform.
32
33
33
-
Provider-specific assertions go in that provider's `*.integrationTest.ts` after the `defineAgentHostE2ETests(config)` call.
34
+
## General best practices and mistakes to avoid
34
35
35
-
## Workflow B — Record / re-record fixtures
36
+
Keep this section curated. Add or refine a principle only when root-cause analysis reveals a pattern likely to improve other tests; keep failure-specific commands, logs, and mechanics in the E2E README or investigation report.
37
+
38
+
-**Wait for the contract state, not a proxy.** Existence, discovery, or an emitted intermediate action may not mean a dependency is operational. For example, wait for the target chat's MCP server to report `Ready`, not merely for its plugin child to appear.
39
+
-**Assert primary outcomes before secondary effects.** A hook, notification, assistant response, or persisted record can be downstream of the behavior under test. First prove the actual tool result or external side effect succeeded.
40
+
-**Do not confuse replay success with execution success.** Replay controls model traffic; live tools, MCP servers, hooks, commands, and filesystem operations can still fail. Recorded assistant text is not an oracle for those operations.
41
+
-**Synchronize on observable state, never elapsed time.** Use protocol notifications or exact state polling instead of sleeps, timeout increases, or existence-only checks.
42
+
-**Respect scope and lifecycle.** Readiness can belong to a specific chat, session, provider process, or workspace. Materialize and observe the same scope the operation will use, drain work before teardown, and clean up owned resources.
43
+
-**Centralize recurring lifecycle barriers.** If several tests need the same multi-step prerequisite, encode it in a shared helper so tests cannot choose a weaker intermediate condition.
44
+
-**Gate genuine nondeterminism narrowly.** Use record-only or provider/platform gates only for behavior that cannot replay deterministically or is unsupported; never use a gate to hide an unexplained failure.
45
+
46
+
## Record or re-record fixtures
36
47
37
48
Re-record when you add a test, or when a bundled SDK/CLI bump changes its wire behavior (new endpoint, different turn count, changed tool schema).
38
49
@@ -48,7 +59,7 @@ Re-record when you add a test, or when a bundled SDK/CLI bump changes its wire b
48
59
49
60
If an SDK now hits a new **ancillary/bootstrap** endpoint (a probe, not a real model turn), add it to `capiStubs.ts` (served, not recorded) instead of recording it — see how `/models/session` is handled.
50
61
51
-
## Workflow C — When a test can't replay deterministically
62
+
## Gate a variant that cannot replay deterministically
52
63
53
64
Real-time streaming, mid-turn aborts, and POSIX-specific local execution (shell tools, `pwd`, git worktrees) don't replay reliably. Gate them precisely so you keep coverage where it works:
54
65
@@ -59,7 +70,7 @@ Real-time streaming, mid-turn aborts, and POSIX-specific local execution (shell
59
70
60
71
Always add a comment explaining *why* the gate exists. Also add or update the corresponding entry in `e2e/KNOWN_ISSUES.md`. When the variant is enabled again, remove or update the entry in the same change.
61
72
62
-
## Workflow D — Cross-platform Azure validation
73
+
## Cross-platform Azure validation
63
74
64
75
New Agent Host E2E and real-provider integration tests are not ready to merge after local or GitHub pull-request CI alone. Push the branch, open or update a draft PR, then use the `azure-pipelines` skill to validate the real packaged Electron integration-test path.
65
76
@@ -70,8 +81,6 @@ New Agent Host E2E and real-provider integration tests are not ready to merge af
70
81
5. After a platform-specific fix, rerun at least that platform. Rerun all three platforms when the fix can affect shared behavior, provider fixtures, process lifecycle, or cross-platform paths.
71
82
6. Cancel obsolete builds after pushing a replacement commit.
72
83
73
-
For additions involving timing, filesystem watching, process lifecycle, worktrees, reconnect/restart, or other known flake surfaces, require **two clean executions of every new test on each supported platform** before merge. A full three-platform build plus a targeted second build is sufficient when the second build runs the relevant tests on all affected platforms.
74
-
75
84
## Verifying & troubleshooting
76
85
77
86
- Run a single provider in replay: `./scripts/test-integration.sh --run <path>` (no env var).
Copy file name to clipboardExpand all lines: src/vs/platform/agentHost/test/node/e2e/captures/copilotcli-non-json-plugin-hook-output-is-ignored-without-failing-the-provider-turn.yaml
+23Lines changed: 23 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,22 @@ exchanges:
5
5
model: claude-sonnet-5
6
6
system: ${system}
7
7
messages:
8
+
- role: user
9
+
content: Reply exactly "MCP_READY". Do not call tools.
10
+
response:
11
+
content: MCP_READY
12
+
stopReason: end_turn
13
+
- request:
14
+
model: claude-sonnet-5
15
+
system: ${system}
16
+
messages:
17
+
- role: user
18
+
content: Reply exactly "MCP_READY". Do not call tools.
19
+
- role: assistant
20
+
content:
21
+
- type: thinking
22
+
- type: text
23
+
text: MCP_READY
8
24
- role: user
9
25
content: Call customization_probe exactly once, then reply with only its exact result.
10
26
response:
@@ -18,6 +34,13 @@ exchanges:
18
34
model: claude-sonnet-5
19
35
system: ${system}
20
36
messages:
37
+
- role: user
38
+
content: Reply exactly "MCP_READY". Do not call tools.
39
+
- role: assistant
40
+
content:
41
+
- type: thinking
42
+
- type: text
43
+
text: MCP_READY
21
44
- role: user
22
45
content: Call customization_probe exactly once, then reply with only its exact result.
Copy file name to clipboardExpand all lines: src/vs/platform/agentHost/test/node/e2e/captures/copilotcli-plugin-posttooluse-hook-runs-after-an-mcp-tool-result.yaml
+23Lines changed: 23 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,22 @@ exchanges:
5
5
model: claude-sonnet-5
6
6
system: ${system}
7
7
messages:
8
+
- role: user
9
+
content: Reply exactly "MCP_READY". Do not call tools.
10
+
response:
11
+
content: MCP_READY
12
+
stopReason: end_turn
13
+
- request:
14
+
model: claude-sonnet-5
15
+
system: ${system}
16
+
messages:
17
+
- role: user
18
+
content: Reply exactly "MCP_READY". Do not call tools.
19
+
- role: assistant
20
+
content:
21
+
- type: thinking
22
+
- type: text
23
+
text: MCP_READY
8
24
- role: user
9
25
content: Call customization_probe exactly once, then reply with only its exact result.
10
26
response:
@@ -18,6 +34,13 @@ exchanges:
18
34
model: claude-sonnet-5
19
35
system: ${system}
20
36
messages:
37
+
- role: user
38
+
content: Reply exactly "MCP_READY". Do not call tools.
39
+
- role: assistant
40
+
content:
41
+
- type: thinking
42
+
- type: text
43
+
text: MCP_READY
21
44
- role: user
22
45
content: Call customization_probe exactly once, then reply with only its exact result.
Copy file name to clipboardExpand all lines: src/vs/platform/agentHost/test/node/e2e/captures/copilotcli-plugin-pretooluse-hook-runs-before-an-mcp-tool.yaml
+23Lines changed: 23 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,22 @@ exchanges:
5
5
model: claude-sonnet-5
6
6
system: ${system}
7
7
messages:
8
+
- role: user
9
+
content: Reply exactly "MCP_READY". Do not call tools.
10
+
response:
11
+
content: MCP_READY
12
+
stopReason: end_turn
13
+
- request:
14
+
model: claude-sonnet-5
15
+
system: ${system}
16
+
messages:
17
+
- role: user
18
+
content: Reply exactly "MCP_READY". Do not call tools.
19
+
- role: assistant
20
+
content:
21
+
- type: thinking
22
+
- type: text
23
+
text: MCP_READY
8
24
- role: user
9
25
content: Call customization_probe exactly once, then reply with only its exact result.
10
26
response:
@@ -18,6 +34,13 @@ exchanges:
18
34
model: claude-sonnet-5
19
35
system: ${system}
20
36
messages:
37
+
- role: user
38
+
content: Reply exactly "MCP_READY". Do not call tools.
39
+
- role: assistant
40
+
content:
41
+
- type: thinking
42
+
- type: text
43
+
text: MCP_READY
21
44
- role: user
22
45
content: Call customization_probe exactly once, then reply with only its exact result.
awaitdriveTurnToCompletion(context.client,sessionUri,'turn-hook-pre-tool','Call customization_probe exactly once, then reply with only its exact result.',2);
awaitdriveTurnToCompletion(context.client,sessionUri,'turn-hook-post-tool','Call customization_probe exactly once, then reply with only its exact result.',2);
constresult=awaitdriveTurnToCompletion(context.client,sessionUri,turnId,'Call customization_probe exactly once, then reply with only its exact result.',2);
0 commit comments