Attribute chat edits to resolved Auto tiers by spurwaha · Pull Request #336757 · microsoft/vscode · GitHub
Skip to content

Attribute chat edits to resolved Auto tiers - #336757

Merged
Logan Ramos (lramos15) merged 10 commits into
microsoft:mainfrom
spurwaha:spurwaha-auto-tier-code-survival
Sep 25, 2026
Merged

Logan Ramos (lramos15) merged 10 commits into
microsoft:mainfrom
spurwaha:spurwaha-auto-tier-code-survival

Conversation

@spurwaha

@spurwaha Shivi Purwaha (spurwaha) commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Attributes chat edits to the client-resolved Auto tier, including defaults, overrides, cache hits, and reroutes. The tier is carried on the edits themselves: ChatResponseTextEditPart and ChatResponseNotebookEditPart gain an optional autoTier (typed as ChatAutoModeTier), which flows per edit batch through the DTO into the chat model's editMetadata snapshots and on into the edit source. Attribution stays tied to the originating request through async queues and restored history. Unknown, failed, older-producer, and non-Auto cases remain unset. No visible UI changes.

Adds optional autoTier detail metadata to prevent mixed-tier aggregation while preserving existing model IDs, source predicates, and pre-tier top-10/30 source coverage. Each selected source expands to at most five known or unknown tier rows.

Design

  • The tier is attribution for the edits Auto produced, so it lives on the edit parts next to the existing $modelId/$harness/$origin edit-source metadata rather than in a response-level side channel. ChatResponseAutoModeResolutionPart keeps its original shape, ChatModel does not track a live tier, and the subagent tool needs no tier plumbing.
  • In the Copilot extension, reportAutoModeRouting reports endpoint and tier through one routing lifecycle event (started / resolved / failed), returns a stream wrapper that stamps the current tier onto edits, and exposes clearTier() so the quota fallback stops attributing once a turn switches off Auto. Every caller receives a result, including cache hits and coalesced calls; only actual routing rounds produce visible routing rows.
  • Auto-specific normalization stays in chat-common rather than the editor layer.

Validation

  • npm run gulp compile-client and the affected core suites (chat model, operation log, language models, subagent tool, chat editing, notebook entries, edit-source telemetry, ext-host chat agents and type converters, agent-host chat contribution): 893 tests passed.
  • Copilot extension type-check and the routing suite (51 tests) passed.
  • Changed-file hygiene, ESLint (including the vscode-dts literal-type rule), and git diff --check passed.

The all-extension npm run compile is blocked by a missing local esbuild dependency in an unrelated built-in extension; the targeted workbench and Copilot checks above passed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings September 18, 2026 11:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Quota fallback can leave stale Auto-tier metadata and misattribute utility-model edits.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds invisible Auto-tier metadata so chat edits are attributed to the resolved routing tier across queues, reroutes, caching, and persistence.

Changes:

  • Adds and transports the proposed Auto-tier progress part.
  • Snapshots tier metadata per edit batch.
  • Subdivides edit telemetry by validated tier while preserving source caps.
File Description
src/​vscode-dts/​vscode.proposed.chatParticipantAdditions.d.ts Declares the proposed tier progress part.
src/​vs/​workbench/​contrib/​editTelemetry/​test/​browser/​editSourceTrackingImpl.test.ts Tests tier aggregation and caps.
src/​vs/​workbench/​contrib/​editTelemetry/​browser/​telemetry/​editSourceTrackingImpl.ts Groups and reports tier telemetry.
src/​vs/​workbench/​contrib/​editTelemetry/​browser/​telemetry/​arcTelemetrySender.ts Excludes tiers from cleaned ARC keys.
src/​vs/​workbench/​contrib/​chat/​test/​common/​model/​chatModel.test.ts Tests tier snapshots and persistence.
src/​vs/​workbench/​contrib/​chat/​test/​browser/​chatEditing/​chatEditingService.test.ts Tests queued edit attribution.
src/​vs/​workbench/​contrib/​chat/​common/​model/​chatSessionOperationLog.ts Persists tier and notebook metadata changes.
src/​vs/​workbench/​contrib/​chat/​common/​model/​chatModel.ts Stores tiers and per-batch snapshots.
src/​vs/​workbench/​contrib/​chat/​common/​editing/​chatEditingService.ts Extends streaming edit contracts.
src/​vs/​workbench/​contrib/​chat/​common/​chatService/​chatService.ts Defines the internal tier progress type.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​notebook/​chatEditingNotebookCellEntry.ts Forwards cell edit metadata.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​chatEditingTextModelChangeService.ts Creates tier-aware edit sources.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​chatEditingSession.ts Preserves metadata through edit queues.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​chatEditingServiceImpl.ts Matches snapshots to edit batches.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​chatEditingModifiedNotebookEntry.ts Propagates notebook metadata.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​chatEditingModifiedFileEntry.ts Extends the edit-entry API.
src/​vs/​workbench/​contrib/​chat/​browser/​chatEditing/​chatEditingModifiedDocumentEntry.ts Propagates document metadata.
src/​vs/​workbench/​api/​test/​common/​extHostTypeConverters.test.ts Tests tier conversion.
src/​vs/​workbench/​api/​test/​common/​extHostChatAgents2.test.ts Tests progress stream ordering.
src/​vs/​workbench/​api/​common/​extHostTypes.ts Implements the extension-host part.
src/​vs/​workbench/​api/​common/​extHostTypeConverters.ts Converts tier progress DTOs.
src/​vs/​workbench/​api/​common/​extHostChatAgents2.ts Accepts and reports tier parts.
src/​vs/​workbench/​api/​common/​extHost.api.impl.ts Exposes the proposed API class.
src/​vs/​platform/​telemetry/​common/​editTelemetry.ts Adds classified tier telemetry.
src/​vs/​editor/​test/​common/​model/​textModelEditSource.test.ts Tests tier validation and keys.
src/​vs/​editor/​common/​textModelEditSource.ts Adds validated tier source metadata.
extensions/​copilot/​src/​vscodeTypes.ts Re-exports the proposed part.
extensions/​copilot/​src/​util/​common/​test/​shims/​vscodeTypesShim.ts Registers the test shim.
extensions/​copilot/​src/​util/​common/​test/​shims/​chatTypes.ts Implements the shimmed part.
extensions/​copilot/​src/​platform/​endpoint/​node/​test/​automodeService.spec.ts Tests tier resolution scenarios.
extensions/​copilot/​src/​platform/​endpoint/​node/​automodeService.ts Emits request-scoped resolved tiers.
extensions/​copilot/​src/​extension/​test/​node/​services.ts Updates the test service stub.
extensions/​copilot/​src/​extension/​conversation/​vscode-node/​chatParticipants.ts Connects routing events to response streams.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Keep attribution in explicit per-edit snapshots, clear it on quota fallback, and isolate subagent tiers. Normalize in chat-common rather than the editor layer and reuse existing regression coverage.

Refs microsoft#336757

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bring microsoft#336757 up to date after the attribution and test refinements.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lramos15
Logan Ramos (lramos15) marked this pull request as ready for review September 25, 2026 16:08
Emit started, resolved and failed states from one event. Deliver endpoint and tier together to every caller while preserving routing-row behavior for cached and coalesced resolutions.

Refs microsoft#336757

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@spurwaha

Copy link
Copy Markdown
Contributor Author

Resolve the grouping conflict without exposing internal chat keys or merging chat scopes when applying source caps. Extend the existing focus-window regression to cover capped chat groups.

Refs microsoft#336757

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the separate tier part. The resolution part now reports model and tier together, with a hidden variant for reused sessions, surfaces without a routing row, and attribution clears.

Refs microsoft#336757

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Auto's tier is attribution for the edits it produced, so it now rides on
ChatResponseTextEditPart/ChatResponseNotebookEditPart.autoTier and flows
per batch through the DTO into the chat model's editMetadata. This drops
the stateful side channel: ChatResponseAutoModeResolutionPart is back to
its original shape, ChatModel no longer intercepts resolution parts or
tracks a request tier, and the subagent tool needs no tier plumbing.

The extension stamps the tier in reportAutoModeRouting by wrapping the
turn's stream; the handler clears it when the quota fallback switches the
turn off Auto. Tests follow the data: the quota fixture is replaced by
reporter-level assertions, and the model/edit tests set autoTier directly
on the edit parts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Restore existing AutomodeService tests to their original assertions,
  keeping routing lifecycle and tier stamping concentrated in the
  dedicated edit attribution suite.
- Focus edit source telemetry test on Chat.applyEdits and deduplicate
  focus window capping checks.
- Simplify queued edits test in ChatEditingService.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lramos15
Logan Ramos (lramos15) enabled auto-merge (squash) September 25, 2026 22:27
@lramos15
Logan Ramos (lramos15) merged commit d7132ac into microsoft:main Sep 25, 2026
35 checks passed
@vs-code-engineering vs-code-engineering Bot added this to the 1.140.0 milestone Sep 25, 2026
Bhavya U (bhavyaus) added a commit that referenced this pull request Oct 8, 2026
* agentHost: attribute edits to the committed Auto tier

Propagate the committed Copilot Auto tier into Agent Host edit telemetry,
matching extension-host attribution from #336757.

- Seed the root tier from `session.model.getCurrent()` (session.start and
  session.resume fire before the session subscribes) and keep it current from
  `session.model_change`; Auto-to-Auto changes that omit the tier keep it.
- Track subagent tiers separately; subagents never inherit the parent tier.
- Snapshot the tier per edit and add it to the `$autoTier:` source-key segment
  and `autoTier` property of `editTelemetry.editSources.details` (Microsoft
  and GitHub), the persisted edit marker, and `agentHost.trackEditSurvival`.
- Keep mixed-tier marker chains from merging in the workbench.

Fixes microsoft/vscode-internalbacklog#9981

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: snapshot Auto tier at tool start and cap sources by group

- Capture the Auto tier when a tool starts, including a pending runtime read,
  so a later tier change or an in-flight read cannot relabel or drop it. A read
  superseded by a later model change resolves as unknown.
- Keep a retained subagent's tier across idle and resume.
- Rank and cap reported sources by pre-tier group before subdividing by tier.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: claim completed edits before resolving a pending Auto tier

Pass the captured tier snapshot into FileEditTracker, which claims the
completed edit before awaiting it. Awaiting first let a later same-file edit
overwrite the pending snapshot, and delayed output-only tool results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: keep edit claim order while an Auto tier read is pending

- Claim every file of a multi-file tool before awaiting any, so a later tool
  cannot take a pending patch's snapshot.
- Process each file's completed edits in claim order, so a later edit with a
  known tier cannot reach attribution before an earlier edit waiting on a
  pending tier read.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants