Bound every apt step in CI, and fix the retry that never ran by danielhanchen · Pull Request #9256 · unslothai/unsloth · GitHub
Skip to content

Bound every apt step in CI, and fix the retry that never ran - #9256

Merged
danielhanchen merged 3 commits into
mainfrom
fix-apt-stalls
Aug 19, 2026
Merged

Bound every apt step in CI, and fix the retry that never ran#9256
danielhanchen merged 3 commits into
mainfrom
fix-apt-stalls

Conversation

@danielhanchen

@danielhanchen danielhanchen commented Aug 19, 2026

Copy link
Copy Markdown
Member

Why

Three CI jobs were lost to an unbounded apt step in a single day, each in a different workflow, and each reported by GitHub as cancelled rather than failed:

job time in apt outcome
Chat UI Tests (chat) 30m18s job's whole budget, everything after skipped
Frontend build + bundle sanity 16m38s in playwright install --with-deps browser smokes and lifecycle tests never ran
Source lint 5m02s on a 5-minute job all fifteen lint checks skipped

This is the worst failure shape CI has. It does not go red on its subject: it spends the job's entire timeout-minutes, prints no reason, names no step, and skips everything downstream. The last one was noticed only because someone happened to open a job that said "cancelled".

What this changes

A shared entry point. .github/scripts/retry-with-apt-lock.sh bounds each attempt with timeout and retries. Two things make the retry actually work, and both were learned the hard way:

  • The dpkg lock. apt runs as root, so killing an attempt leaves the apt-get child alive holding /var/lib/dpkg/lock-frontend, and the next attempt dies two seconds later with "Could not get lock". A retry that cannot succeed is worse than none, because it buries the real reason under a second, different failure.
  • set -e. GitHub runs run: blocks as bash -e, so a bare failing command aborts the step where it stands. A loop written as timeout ... ; rc=$? never reaches its second attempt: the step exits 124 with no output, indistinguishable from the original stall. That is exactly how the first version of this shipped, and why it is a script now rather than eight inline copies.

Every on-runner apt step now goes through it, with an explicit step timeout: lint-ci, studio-api-smoke, studio-inference-smoke, studio-update-smoke, studio-ui-smoke, studio-tauri-smoke, studio-frontend-ci, interrupted-install-ci, consolidated-tests-ci, desktop-app-clean-machine-ci, release-desktop, and the four copies in local-agent-guides-ci.

playwright install --with-deps counts as an apt step, because that is what it is. Excluding it by name is what cost the Frontend build job.

update and install go as one unit. Retrying the install after a stalled update just re-reads the same broken package list, and two calls double the worst case for no extra coverage.

Job budgets raised where the step's bounded worst case no longer fits: source-lint 5 to 20, api-smoke 12 to 20, frontend build 20 to 40. A job timeout is a backstop for the unforeseen; using it as the bound on a known-flaky step is what converts a diagnosable failure into a silent one.

Guard

tests/studio/test_apt_steps_are_bounded.py reads the workflows and asserts:

  • every apt step routes through the helper
  • every apt step has its own timeout-minutes
  • the retry budget it authorises actually fits inside that step timeout, so the last attempt is not silently deleted
  • the step timeout fits inside the job timeout, so the job stays alive to report which step stalled
  • a workflow that calls the helper and filters on paths lists the helper, so a change to it reruns its consumers
  • the detector is not vacuous (it must still match at least 15 steps across at least 10 workflows)

All six corresponding mutants were reintroduced and confirmed to go red. Wired into workflow-trigger-lint.yml, the only job with no paths filter, since the edit that breaks it is by definition a workflow-only edit.

Not converted, deliberately

  • clean-machine-install-ci.yml runs apt inside bare distro containers and inside WSL, as root, with no checkout to read the helper from, and one leg asserts that sudo does not exist on the image at all.
  • Two steps where apt is the assertion rather than the setup: removing curl to construct the no-transport case, and installing the built .deb to find out whether it declares its own runtime dependencies. Retrying those would retry the assertion. The .deb step's apt preamble does go through the helper, and the step gained a budget anyway.

Verification

tests/studio: 4456 passed, 4 skipped. actionlint clean on every changed file (the two findings it reports are pre-existing and unrelated: a macos-26 label and concurrency.queue, both newer than actionlint 1.7.7 knows).

…ver ran

Two problems, one of them mine.

The retry I added to the Playwright install last round was dead on arrival.
GitHub runs `run:` blocks as `bash -e`, and I wrote the loop as

    timeout --signal=TERM --kill-after=30 480 python -m playwright "$@"
    rc=$?

A bare failing command under -e aborts the step then and there, so the second
attempt never ran. The step exited 124 having printed none of its own warnings --
indistinguishable from the unbounded stall it was meant to replace, which is
exactly how it read on #9251's indicator and banner shards, both dying at 8m05s.
The previous version used `if timeout ...; then` and was exempt from -e; changing
it to read the exit code is what broke it. Reproduced both forms under `bash -e`.

The second is broader. `Linux deps` is a bare `apt-get update && apt-get install`
with no bound at all, and on #9251 it sat on the Azure mirror for 28 minutes. Of
24 apt-invoking steps across 13 workflows, 2 were bounded. An unbounded apt step
is not slow, it is silent: it spends the job's whole budget, GitHub scores the
result as "cancelled" rather than a failure, prints no reason, and skips every
step after it.

.github/scripts/retry-with-apt-lock.sh is now the single definition: per-attempt
timeout, retry, dpkg-lock wait between attempts, and exit codes reported honestly
(124/137 as a stall, anything else as the status it actually returned). It carries
no `set -e` and uses `|| rc=$?`, so it works whatever the caller set. Six paths
exercised locally, including the regression above: called from a `bash -e` step,
both attempts still run.

Sizing respects what this file already learned. The note on ui-smoke's
timeout-minutes records that cutting the JOB budget to 20 turned this same stall
from a slow pass into a red build, because one attempt had nowhere to go. The
retry is what changes that, not a shorter wait: 180s is ~14x the 13 seconds
`Linux deps` takes healthy, so only a mirror that is genuinely gone burns all
three attempts. Every step's worst case fits inside its own bound and every step
bound sits under the job's 30, which stays the final backstop and is deliberately
not tightened.

Listed in the workflow's paths filter, since both apt steps now execute it.

The other 20 unbounded apt steps are left alone: every stall observed so far has
been in this workflow, and converting release workflows unreviewed is a worse
trade than leaving them until one of them actually stalls.
@chatgpt-codex-connector

Copy link
Copy Markdown

Three jobs were lost to an unbounded apt step in one day, each in a
different workflow and each reported as "cancelled" with no reason:
Chat UI Tests (chat) at 30m18s, Frontend build at 16m38s inside
playwright install --with-deps, and Source lint at 5m02s on a job whose
whole budget was 5 minutes. None of the three said anything about its
actual subject, and the last was noticed only because someone happened
to open a job that said "cancelled".

Routes the remaining twelve on-runner apt steps through the shared
retry helper, gives each an explicit step timeout, and adds a guard so
the next one cannot be added without one. update and install now go as
a single unit: retrying the install after a stalled update just re-reads
the same broken package list.

playwright install --with-deps counts as an apt step, because that is
what it is. Leaving it out by name is what cost the Frontend build job.

Job budgets were raised where the step's bounded worst case no longer
fit: source-lint 5 to 20, api-smoke 12 to 20, frontend build 20 to 40.
A job timeout is a backstop for the unforeseen; using it as the bound
on a known-flaky step is what turns a diagnosable failure into a silent
one.

Not converted, with reasons recorded in the guard: clean-machine-install-ci
runs apt inside bare containers and WSL with no checkout and deliberately
no sudo, and the two steps where apt is the assertion rather than the
setup.
@danielhanchen danielhanchen changed the title Give apt one bounded, retrying entry point, and fix the retry that never ran Bound every apt step in CI, and fix the retry that never ran Aug 19, 2026
@danielhanchen
danielhanchen merged commit cabed07 into main Aug 19, 2026
45 of 50 checks passed
@danielhanchen
danielhanchen deleted the fix-apt-stalls branch August 19, 2026 06:18
danielhanchen added a commit that referenced this pull request Aug 19, 2026
* Make apt fail fast, so the retry has something to retry

#9256 bounded the stalls. This says why they happened and stops them.

Reading the four logs, every one ends the same way:

  04:47:02  Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
  05:16:29  ##[error]The operation was canceled.

Twenty-nine minutes of silence mid-fetch of a 126 kB index file, and two
of the four hung on that same file. What came before it is the other
half: azure.archive.ubuntu.com was Ign'd four times over thirty seconds,
so apt had already failed over through /etc/apt/apt-mirrors.txt to the
public archive, which is not provisioned for this fleet.

apt did not treat any of that as an error. Acquire::http::Timeout
defaults to 120s and is an idle timeout, so a socket that is open and
trickling never trips it. Cutting it to 20s with internal retries turns
a 29 minute hang into a ~1 minute failure, and that matters beyond
speed: the wall-clock kill is what orphans the dpkg lock, so an apt that
gives up on its own is one we never have to kill.

Not pinning a mirror. The evidence does not support it: in one stall
Azure was dead and the public archive hung, in another Azure was serving
fine and the transfer stalled at a 13.6 MB package. Neither is reliably
better, and the mirrorlist failover is already the right mechanism. It
just needed to be allowed to give up.

The written config is validated against apt itself (apt-config dump
round-trips all four options), and the helper's retry, timeout, giving
up, recovery and no-command paths are exercised end to end.

* Wait on all four apt locks, and stop racing a slow mirror

Two bugs, both mine, both caught by CI on this branch.

1. apt-get update takes /var/lib/apt/lists/lock and nothing else. The
   helper waited on /var/lib/dpkg/lock-frontend only, so after an attempt
   was killed mid-update the wait saw a free lock, retried immediately,
   and produced

     E: Could not get lock /var/lib/apt/lists/lock. It is held by process 2420 (apt-get)

   twice in under two seconds. Three attempts, one real one -- exactly the
   'a retry that cannot succeed is worse than none' failure the header
   warns about, in the one case it was written for. It now waits on all
   four: dpkg's two, lists, and archives.

2. 150s per attempt was racing the mirror, not bounding it. The mirror
   was degraded rather than dead, so every attempt got killed at the same
   point and none finished. Two attempts of 360s instead of three of 150:
   the bound exists to stop an infinite hang, not to beat a slow server.
   studio-update-smoke's job budget goes 15 to 25 to fit the new worst
   case; the guard checks that arithmetic.

* Ask the image's package lists first, and refresh only if they miss

The mirror outage is still live. From this branch's own GGUF smoke:

  06:55:33 Ign:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease
  06:55:35 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
  07:01:11 (nothing, for 5m36s, then the attempt's cap)

Azure is unreachable and the public archive it fails over to is not
provisioned for this fleet. Acquire::http::Timeout does not save us there:
it is an idle timeout, and a server dribbling a byte every few seconds
never trips it, so the wall-clock bound is what ends the attempt.

The bound is working as intended -- the step now fails in 14 minutes with
the reason printed, where before it spent 30 and was reported as
"cancelled" with nothing said. But a bounded failure is still a failure,
and the operation that fails is one we mostly do not need.

`apt-get update` refreshes every index for every suite. The runner image
already ships populated lists, and these steps install a handful of
ordinary packages. So try the install first, and refresh only when it
misses:

  apt-get install -y X || { apt-get update && apt-get install -y X; }

In the common case the slowest and most failure-prone apt operation is
never performed at all. When the image's lists really are too stale --
the "Unable to locate package" case the runner-images maintainers warn
about -- the update still runs and the install is retried, so nothing is
traded away.

Not applied to clean-machine-install-ci: a fresh WSL image genuinely has
no lists, so install-first would always miss, and a bare machine is that
leg's whole premise.

The three outcomes were checked as shell rather than reasoned about:
resolvable (update never reached, rc=0), not resolvable (update runs,
rc=0), and both failing (rc=1, so the failure still propagates under the
`bash -e` GitHub runs steps with).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant