iframe-proxy

Bryan Chen (bryanchen-d) · 2026-10-21T13:23:40Z

Fixes the Checkout microsoft/vscode-distro failures that cost the 05:00 UTC insider 487409 about 5h and failed main CI 487475.

What happened

download-distro.yml fetches the full vscode-distro history with fetchDepth: 0: 1,415,967 objects, about 455 MiB. On a healthy agent that takes about 12 s at roughly 38 MiB/s. Both failures ran on macOS AcesShared agents downloading at about 100 KiB/s:

Every retry then failed within a second with fatal: Cannot prompt because user interactivity has been disabled. That applies to the agent's inner retries and to all 3 retryCountOnTaskFailure attempts. They reuse the same extraheader, and it's a GitHub App installation token that expires after one hour. Once a fetch outlives the token, no retry can succeed.

Scan of the last 30 days

I scanned all 1,663 builds of definition 111, covering 38,568 distro checkouts and including previous job attempts:

On AcesShared macOS agents, only 2 of 9,308 checkouts took more than 5 min before 2026-10-20 20:00 UTC. Since then it's 24 of 432, across 13 different agents. Three of those ran into the token limit: 487409, 487475, and 487285, which was canceled at 63 min.
Linux and Windows p99 is under 30 s.

The slow network is on the macOS pool side and needs routing to its owners. This PR stops one slow pool from costing a release.

Fix

Check out distro with fetchDepth: 1 and fetchTags: false. Then Checkout distro commit fetches only the full 40-character commit pinned in package.json (currently f74ed5fd22f722226243d6e993b17c7e98587baa), using the persisted credentials, and checks it out. That step now also has retryCountOnTaskFailure: 3. A depth-1 fetch of the pinned commit is about 12.7 MiB, roughly 36 times smaller. At the observed 100 KiB/s that's about 2 min instead of more than an hour.
checkDistroCommit.ts used to read refs/remotes/origin/<branch>, which a shallow checkout no longer has. It now runs git ls-remote --heads origin refs/heads/<branch>. A branch that's missing from distro still takes the existing "expected for feature branches" path. In Quality Checks this step runs about 9 min after the checkout, well within the token's lifetime.
No other consumer needs distro history or tags. The mixins read files from the working tree, and lfs pull works on HEAD.

Validation

I replayed the new flow locally against vscode-distro with the same commands:

Depth-1 checkout of main, then fetching and checking out the pinned f74ed5fd, took 1 s for 12.74 MiB in total. npm/ and mixin/{exploration,insider,stable} are present.
checkDistroCommit.ts reports the same main mismatch as Quality Checks in 487522 (f74ed5fd vs 2733515d). It resolves a release/* branch head and takes the feature-branch path for a missing branch.
The YAML parses, and checkDistroCommit.ts type-checks against build/tsconfig.json.
The lookup-failure warning escapes Azure DevOps logging-command data (percent, CR and LF). A targeted regression check failed before the fix and passes afterward; missing-branch, matching-pin and mismatched-pin checks also pass. The normal pre-commit hygiene hook passes.

Authored with Copilot

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

This builds a shell command using the dynamic branch string, which risks shell injection and… · New
The script checks $LASTEXITCODE after git fetch but not after git checkout. In PowerShell,… · New

What changed in this PR

Reduces Azure Pipelines failures/timeouts when checking out microsoft/vscode-distro by avoiding a full-history fetch and updating the branch-head resolution logic to work with shallow checkouts.

Changes:

Switch download-distro.yml to a shallow checkout and explicitly fetch the pinned distro commit with retries.
Update checkDistroCommit.ts to resolve branch heads via git ls-remote instead of relying on local remote-tracking refs.

File	Description
build/azure-pipelines/distro/download-distro.yml	Shallow checkout + explicit fetch of pinned commit to avoid long full-history downloads and token expiry.
build/azure-pipelines/common/checkDistroCommit.ts	Use remote query (`ls-remote`) to resolve branch heads in shallow checkouts; adjust logging message.

_{Review effort: Lite}

Bryan Chen (bryanchen-d) · 2026-10-21T13:51:52Z

Validation run 487569 on this branch, with publish off, covering Quality, Linux x64 and macOS ARM64:

Pool	Jobs	`Checkout microsoft/vscode-distro`	`Checkout distro commit`
macOS AcesShared (agents 1, 3, 6, 7, 8, 9, 10, 17, 18)	Compile, CLI, Sign, 6 test jobs	12-15 s, 421 objects, 12.73 MiB	6 s
Linux 1es-ubuntu	Compile, CLI, Quality Checks, Unit Tests	2 s	1 s
Windows 1es-windows (API scan)	1	3 s	2 s

The agent ran git fetch --no-tags --depth=1 +<main tip>. Checkout distro commit then fetched the pinned f74ed5fd over the persisted credentials, and HEAD is now at f74ed5f.
Pull Git LFS objects, Mixin distro quality and Quality Checks all succeeded.
For comparison, the full-history fetch had a macOS p50 of 50 s and peaked at 63 min in 487409 and 487475.
Check distro commit only runs when VSCODE_PUBLISH is true, so it was skipped here. Its ls-remote uses the same persisted credential that the fetch above proved. I validated it locally: same main mismatch as 487522, resolves release/*, and handles a missing branch the same as before.

Co-authored with Copilot

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

This changes the check from a local ref lookup to a networked git ls-remote, which can hang in CI… · New
Because $DistroVersion is read from package.json, it’s safer to prevent option-style injection… · New

2 resolved since last review

_{Review effort: Lite}

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

This catch treats any failure resolving the branch head as an expected “feature branch” case… · New
The validation regex only accepts lowercase hex, but commit SHAs are case-insensitive hex. To avoid… · New

2 resolved since last review

_{Review effort: Lite}

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

This changes behavior from “treat as missing branch” to “skip the check entirely” on any transient… · New
Fetching by raw SHA relies on the remote allowing SHA-based wants and the commit being… · New
Interpolating ${error} into an Azure DevOps logging command can introduce newlines/formatting… · New

2 resolved since last review

_{Review effort: Lite}

Bryan Chen (bryanchen-d) · 2026-10-21T15:39:56Z

Raymond Zhao (@rzhao271) Alex Ross (@alexr00) could you take an early look? Raymond made the last change to download-distro.yml (#331867), and Alex added the early distro-commit warning (#293104) that checkDistroCommit.ts builds on.

This PR fetches only the pinned vscode-distro commit, after two macOS checkouts failed mid-transfer (fatal: early EOF) and the retry couldn't re-authenticate. One of them cost the 05:00 UTC insider release about 5 hours.

The question I'd most like your eyes on: does fetching only the pinned commit break any consumer that expects the full distro history or other branches?

Results from the full definition 111 validation run (487592) will be posted here shortly, so you can review against real evidence.

Co-authored with Copilot

Every job fetched the full vscode-distro history (1.4M objects, ~455 MiB). Since Oct 20 ~20:00 UTC, macOS AcesShared agents intermittently download from GitHub at ~100 KiB/s, so that fetch ran for an hour. The first attempt then died with "early EOF" after ~62 min, and every retry failed with "Cannot prompt because user interactivity has been disabled" because the one-hour GitHub App token had expired. That failed the 05:00 insider 487409 (macOS ARM64 Sign) and main CI 487475 (Smoke Tests). Check out distro with depth 1 and no tags, then fetch just the commit pinned in package.json (~13 MiB). The distro commit check no longer has local branch refs, so it asks the remote for the branch head with ls-remote. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…the checkout exit code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…distro commit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ating it as a feature branch Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

The new validation requires a 40-character SHA, but the PR description and common practice often… · New
Azure DevOps logging commands are sensitive to newlines/special characters. Interpolating the raw… · New

3 resolved since last review

_{Review effort: Lite}

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

Interpolating ${error} can produce unhelpful output like [object Object] for non-Error throws.… · New
The validation rejects uppercase hex SHA values (A–F). Git commit IDs are hex and are commonly… · New

2 resolved since last review

_{Review effort: Lite}

Bryan Chen (bryanchen-d) · 2026-10-21T17:17:21Z

Full pipeline 111 validation, as required for pipeline changes: run 487592, covering all platforms with publish, release and sanity tests off. It ran a throwaway branch: main plus this PR at 4b1aaebd7bc plus #343150. Since then, the PR head 940f0362b2f only adds escaping to the warning text in the lookup-failure path, and no step that ran here changed. The branch also carried a throwaway-only commit, a4d8637e1c2, that ungated Check distro commit. That commit is not part of this PR.

Distro checkout per platform. Every job fetched 421 objects, 12.73 MiB, and none failed. Before this change it was 1.4M objects and 455 MiB.

Platform	Jobs	`Checkout microsoft/vscode-distro`	`Checkout distro commit`
macOS ARM64 (incl. Sign)	9	12-19 s	0-1 s
macOS X64 (incl. Sign)	3	12-13 s	1 s
macOS Universal	1	11 s	1 s
Windows x64 (incl. Sign)	9	2-3 s	1-2 s
Windows ARM64 (incl. Sign)	3	3 s	2 s
Linux x64 / ARM64 / ARMHF (incl. Sign)	9 / 3 / 3	1-2 s	0-2 s
Alpine x64 / ARM64, Alpine CLIs, Web, APIScan	6	1-2 s	0-1 s
Quality	1	1 s	1 s

Pull Git LFS objects, the distro mixins and every Compile/Sign job succeeded on attempt 1. All stages are green except Windows X64 attempt 2, explained below.

Check distro commit. I tried to override BUILD_SOURCEBRANCH to main for this step, but it's a read-only predefined variable, so the step saw the real branch name:

Distro commit in package.json: 2733515d0a50cde08bdfbbe4368f0f75b7828882
Current branch: copilot/vscode/validate-343150-343219
Branch 'copilot/vscode/validate-343150-343219' not found in vscode-distro

That proves ls-remote authenticates on the agent with the persisted credential. A credential failure would have taken the new warning path instead. The compare path was validated locally. The step can't fail a build: it's continueOnError, lookup errors become a warning, and any other error makes it succeededWithIssues, the same as today's mismatch warning. The first official build after merge will log it in Quality Checks.

Windows X64 stage failure (test artifact, not this PR). To exercise #342967's skip branch, I retried the green Windows X64 stage with forceRetryAllJobs. That also re-ran Windows (X64) - Compile, whose Publish compilation artifact failed with Artifact compilation_win32_x64 already exists for build 487592. So the forced retry never reached Sign. A forced whole-stage retry can't test that branch, which stays covered by #343150's unit test.

Co-authored with Copilot

vs-code-engineering · 2026-10-21T17:23:47Z

Copilot AI balanced review requested due to automatic review settings October 21, 2026 13:23

Bryan Chen (bryanchen-d) self-assigned this Oct 21, 2026

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread build/azure-pipelines/common/checkDistroCommit.ts Outdated

Comment thread build/azure-pipelines/distro/download-distro.yml

Bryan Chen (bryanchen-d) requested a balanced review from Copilot October 21, 2026 13:52

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread build/azure-pipelines/common/checkDistroCommit.ts Outdated

Comment thread build/azure-pipelines/distro/download-distro.yml

Bryan Chen (bryanchen-d) requested a balanced review from Copilot October 21, 2026 14:14

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread build/azure-pipelines/common/checkDistroCommit.ts

Comment thread build/azure-pipelines/distro/download-distro.yml

Bryan Chen (bryanchen-d) requested a balanced review from Copilot October 21, 2026 14:35

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread build/azure-pipelines/common/checkDistroCommit.ts Outdated

Comment thread build/azure-pipelines/distro/download-distro.yml

Comment thread build/azure-pipelines/common/checkDistroCommit.ts Outdated

Bryan Chen (bryanchen-d) and others added 4 commits October 21, 2026 09:09

Engineering - pass the distro branch to git as an argument and check …

de2407b

…the checkout exit code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Engineering - bound the distro branch lookup and validate the pinned …

fd37b6c

…distro commit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Engineering - warn when the distro branch lookup fails instead of tre…

95ab254

…ating it as a feature branch Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bryan Chen (bryanchen-d) force-pushed the copilot/vscode/shallow-distro-checkout branch from 4b1aaeb to 95ab254 Compare October 21, 2026 16:09

Bryan Chen (bryanchen-d) requested a balanced review from Copilot October 21, 2026 16:09

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread build/azure-pipelines/distro/download-distro.yml

Comment thread build/azure-pipelines/common/checkDistroCommit.ts Outdated

Engineering - escape distro lookup warning diagnostics

940f036

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bryan Chen (bryanchen-d) requested a balanced review from Copilot October 21, 2026 16:33

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread build/azure-pipelines/common/checkDistroCommit.ts

Comment thread build/azure-pipelines/distro/download-distro.yml

Bryan Chen (bryanchen-d) mentioned this pull request Oct 21, 2026

Engineering - let the artifact check exit on its own so it doesn't crash Node on Windows #343150

Merged

Bryan Chen (bryanchen-d) marked this pull request as ready for review October 21, 2026 17:22

Bryan Chen (bryanchen-d) enabled auto-merge (squash) October 21, 2026 17:23

Benjamin Christopher Simmonds (benibenj) approved these changes Oct 21, 2026

View reviewed changes

Build	Job	Fetch started	Died	Progress
487409	macOS (ARM64) - Sign, attempt 1	05:33:26	06:36:14 (`early EOF`)	71%, 343 MiB
487475	Smoke Tests (Electron)	08:45:48	09:48:15 (`early EOF`)	66%, 322 MiB

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

Bryan Chen (bryanchen-d) commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What happened

Scan of the last 30 days

Fix

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Bryan Chen (bryanchen-d) commented Oct 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bryan Chen (bryanchen-d) commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Bryan Chen (bryanchen-d) commented Oct 21, 2026

Uh oh!

vs-code-engineering Bot commented Oct 21, 2026

📬 CODENOTIFY

Ladislau Szomoru (@lszomoru)

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Bryan Chen (bryanchen-d) commented Oct 21, 2026 •

edited

Loading

Bryan Chen (bryanchen-d) commented Oct 21, 2026 •

edited

Loading