{{ message }}
Bound every apt step in CI, and fix the retry that never ran - #9256
Merged
Conversation
…ver ran
Two problems, one of them mine.
The retry I added to the Playwright install last round was dead on arrival.
GitHub runs `run:` blocks as `bash -e`, and I wrote the loop as
timeout --signal=TERM --kill-after=30 480 python -m playwright "$@"
rc=$?
A bare failing command under -e aborts the step then and there, so the second
attempt never ran. The step exited 124 having printed none of its own warnings --
indistinguishable from the unbounded stall it was meant to replace, which is
exactly how it read on #9251's indicator and banner shards, both dying at 8m05s.
The previous version used `if timeout ...; then` and was exempt from -e; changing
it to read the exit code is what broke it. Reproduced both forms under `bash -e`.
The second is broader. `Linux deps` is a bare `apt-get update && apt-get install`
with no bound at all, and on #9251 it sat on the Azure mirror for 28 minutes. Of
24 apt-invoking steps across 13 workflows, 2 were bounded. An unbounded apt step
is not slow, it is silent: it spends the job's whole budget, GitHub scores the
result as "cancelled" rather than a failure, prints no reason, and skips every
step after it.
.github/scripts/retry-with-apt-lock.sh is now the single definition: per-attempt
timeout, retry, dpkg-lock wait between attempts, and exit codes reported honestly
(124/137 as a stall, anything else as the status it actually returned). It carries
no `set -e` and uses `|| rc=$?`, so it works whatever the caller set. Six paths
exercised locally, including the regression above: called from a `bash -e` step,
both attempts still run.
Sizing respects what this file already learned. The note on ui-smoke's
timeout-minutes records that cutting the JOB budget to 20 turned this same stall
from a slow pass into a red build, because one attempt had nowhere to go. The
retry is what changes that, not a shorter wait: 180s is ~14x the 13 seconds
`Linux deps` takes healthy, so only a mirror that is genuinely gone burns all
three attempts. Every step's worst case fits inside its own bound and every step
bound sits under the job's 30, which stays the final backstop and is deliberately
not tightened.
Listed in the workflow's paths filter, since both apt steps now execute it.
The other 20 unbounded apt steps are left alone: every stall observed so far has
been in this workflow, and converting release workflows unreviewed is a worse
trade than leaving them until one of them actually stalls.
Three jobs were lost to an unbounded apt step in one day, each in a different workflow and each reported as "cancelled" with no reason: Chat UI Tests (chat) at 30m18s, Frontend build at 16m38s inside playwright install --with-deps, and Source lint at 5m02s on a job whose whole budget was 5 minutes. None of the three said anything about its actual subject, and the last was noticed only because someone happened to open a job that said "cancelled". Routes the remaining twelve on-runner apt steps through the shared retry helper, gives each an explicit step timeout, and adds a guard so the next one cannot be added without one. update and install now go as a single unit: retrying the install after a stalled update just re-reads the same broken package list. playwright install --with-deps counts as an apt step, because that is what it is. Leaving it out by name is what cost the Frontend build job. Job budgets were raised where the step's bounded worst case no longer fit: source-lint 5 to 20, api-smoke 12 to 20, frontend build 20 to 40. A job timeout is a backstop for the unforeseen; using it as the bound on a known-flaky step is what turns a diagnosable failure into a silent one. Not converted, with reasons recorded in the guard: clean-machine-install-ci runs apt inside bare containers and WSL with no checkout and deliberately no sudo, and the two steps where apt is the assertion rather than the setup.
for more information, see https://pre-commit.ci
danielhanchen
added a commit
that referenced
this pull request
Aug 19, 2026
* Make apt fail fast, so the retry has something to retry #9256 bounded the stalls. This says why they happened and stops them. Reading the four logs, every one ends the same way: 04:47:02 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB] 05:16:29 ##[error]The operation was canceled. Twenty-nine minutes of silence mid-fetch of a 126 kB index file, and two of the four hung on that same file. What came before it is the other half: azure.archive.ubuntu.com was Ign'd four times over thirty seconds, so apt had already failed over through /etc/apt/apt-mirrors.txt to the public archive, which is not provisioned for this fleet. apt did not treat any of that as an error. Acquire::http::Timeout defaults to 120s and is an idle timeout, so a socket that is open and trickling never trips it. Cutting it to 20s with internal retries turns a 29 minute hang into a ~1 minute failure, and that matters beyond speed: the wall-clock kill is what orphans the dpkg lock, so an apt that gives up on its own is one we never have to kill. Not pinning a mirror. The evidence does not support it: in one stall Azure was dead and the public archive hung, in another Azure was serving fine and the transfer stalled at a 13.6 MB package. Neither is reliably better, and the mirrorlist failover is already the right mechanism. It just needed to be allowed to give up. The written config is validated against apt itself (apt-config dump round-trips all four options), and the helper's retry, timeout, giving up, recovery and no-command paths are exercised end to end. * Wait on all four apt locks, and stop racing a slow mirror Two bugs, both mine, both caught by CI on this branch. 1. apt-get update takes /var/lib/apt/lists/lock and nothing else. The helper waited on /var/lib/dpkg/lock-frontend only, so after an attempt was killed mid-update the wait saw a free lock, retried immediately, and produced E: Could not get lock /var/lib/apt/lists/lock. It is held by process 2420 (apt-get) twice in under two seconds. Three attempts, one real one -- exactly the 'a retry that cannot succeed is worse than none' failure the header warns about, in the one case it was written for. It now waits on all four: dpkg's two, lists, and archives. 2. 150s per attempt was racing the mirror, not bounding it. The mirror was degraded rather than dead, so every attempt got killed at the same point and none finished. Two attempts of 360s instead of three of 150: the bound exists to stop an infinite hang, not to beat a slow server. studio-update-smoke's job budget goes 15 to 25 to fit the new worst case; the guard checks that arithmetic. * Ask the image's package lists first, and refresh only if they miss The mirror outage is still live. From this branch's own GGUF smoke: 06:55:33 Ign:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease 06:55:35 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB] 07:01:11 (nothing, for 5m36s, then the attempt's cap) Azure is unreachable and the public archive it fails over to is not provisioned for this fleet. Acquire::http::Timeout does not save us there: it is an idle timeout, and a server dribbling a byte every few seconds never trips it, so the wall-clock bound is what ends the attempt. The bound is working as intended -- the step now fails in 14 minutes with the reason printed, where before it spent 30 and was reported as "cancelled" with nothing said. But a bounded failure is still a failure, and the operation that fails is one we mostly do not need. `apt-get update` refreshes every index for every suite. The runner image already ships populated lists, and these steps install a handful of ordinary packages. So try the install first, and refresh only when it misses: apt-get install -y X || { apt-get update && apt-get install -y X; } In the common case the slowest and most failure-prone apt operation is never performed at all. When the image's lists really are too stale -- the "Unable to locate package" case the runner-images maintainers warn about -- the update still runs and the install is retried, so nothing is traded away. Not applied to clean-machine-install-ci: a fresh WSL image genuinely has no lists, so install-first would always miss, and a bare machine is that leg's whole premise. The three outcomes were checked as shell rather than reasoned about: resolvable (update never reached, rc=0), not resolvable (update runs, rc=0), and both failing (rc=1, so the failure still propagates under the `bash -e` GitHub runs steps with).
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
Three CI jobs were lost to an unbounded apt step in a single day, each in a different workflow, and each reported by GitHub as cancelled rather than failed:
Chat UI Tests (chat)Frontend build + bundle sanityplaywright install --with-depsSource lintThis is the worst failure shape CI has. It does not go red on its subject: it spends the job's entire
timeout-minutes, prints no reason, names no step, and skips everything downstream. The last one was noticed only because someone happened to open a job that said "cancelled".What this changes
A shared entry point.
.github/scripts/retry-with-apt-lock.shbounds each attempt withtimeoutand retries. Two things make the retry actually work, and both were learned the hard way:apt-getchild alive holding/var/lib/dpkg/lock-frontend, and the next attempt dies two seconds later with "Could not get lock". A retry that cannot succeed is worse than none, because it buries the real reason under a second, different failure.set -e. GitHub runsrun:blocks asbash -e, so a bare failing command aborts the step where it stands. A loop written astimeout ... ; rc=$?never reaches its second attempt: the step exits 124 with no output, indistinguishable from the original stall. That is exactly how the first version of this shipped, and why it is a script now rather than eight inline copies.Every on-runner apt step now goes through it, with an explicit step timeout:
lint-ci,studio-api-smoke,studio-inference-smoke,studio-update-smoke,studio-ui-smoke,studio-tauri-smoke,studio-frontend-ci,interrupted-install-ci,consolidated-tests-ci,desktop-app-clean-machine-ci,release-desktop, and the four copies inlocal-agent-guides-ci.playwright install --with-depscounts as an apt step, because that is what it is. Excluding it by name is what cost the Frontend build job.updateandinstallgo as one unit. Retrying the install after a stalled update just re-reads the same broken package list, and two calls double the worst case for no extra coverage.Job budgets raised where the step's bounded worst case no longer fits:
source-lint5 to 20,api-smoke12 to 20, frontendbuild20 to 40. A job timeout is a backstop for the unforeseen; using it as the bound on a known-flaky step is what converts a diagnosable failure into a silent one.Guard
tests/studio/test_apt_steps_are_bounded.pyreads the workflows and asserts:timeout-minutespathslists the helper, so a change to it reruns its consumersAll six corresponding mutants were reintroduced and confirmed to go red. Wired into
workflow-trigger-lint.yml, the only job with no paths filter, since the edit that breaks it is by definition a workflow-only edit.Not converted, deliberately
clean-machine-install-ci.ymlruns apt inside bare distro containers and inside WSL, as root, with no checkout to read the helper from, and one leg asserts thatsudodoes not exist on the image at all..debto find out whether it declares its own runtime dependencies. Retrying those would retry the assertion. The.debstep's apt preamble does go through the helper, and the step gained a budget anyway.Verification
tests/studio: 4456 passed, 4 skipped.actionlintclean on every changed file (the two findings it reports are pre-existing and unrelated: amacos-26label andconcurrency.queue, both newer than actionlint 1.7.7 knows).