Stop the benchmark CI job hanging on apt by bdraco · Pull Request #13489 · aio-libs/aiohttp · GitHub
Skip to content

Stop the benchmark CI job hanging on apt - #13489

Merged
bdraco merged 3 commits into
masterfrom
benchmark-apt-hang
Aug 18, 2026
Merged

bdraco merged 3 commits into
masterfrom
benchmark-apt-hang

Conversation

@bdraco

@bdraco bdraco commented Aug 18, 2026

Copy link
Copy Markdown
Member

What do these changes do?

The Benchmark job has been getting cancelled at its 15 minute limit, so CodSpeed stopped reporting on some PRs (for example https://github.com/aio-libs/aiohttp/actions/runs/32157680880/job/95779217306). The job logs show it stuck right after the CodSpeed runner prints Installing packages: /tmp/valgrind-codspeed.deb, libc6-dbg; that is a plain apt-get update plus apt-get install libc6-dbg inside the runner with no timeout, and it has taken anywhere from 37 seconds to over 14 minutes on the same runner image today. It runs even after the runner restores valgrind from its cache, because the runner also checks dpkg -s libc6-dbg and the cache only copies files.

This adds a step that installs libc6-dbg before the CodSpeed action with a timeout and a retry loop, so a stalled mirror fails fast and retries instead of hanging; once it is registered in dpkg the runner finds it present after its cache restore and skips apt entirely. It also raises the job timeout to 30 minutes, since a slow install plus the ~9 minute benchmark suite already blew past 15 in one run.

Are there changes in behavior for the user?

No, CI only.

Is it a substantial burden for the maintainers to support this?

No; one extra workflow step and a timeout bump.

Related issue number

None.

Checklist

  • I think the code is well written
  • Unit tests for the changes exist N/A
  • Documentation reflects the changes N/A
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt N/A
    • The format is <Name> <Surname>.
    • Please keep alphabetical order, the file is sorted by names.
  • Add a new news fragment into the CHANGES/ folder
    • name it <issue_or_pr_num>.<type>.rst (e.g. 588.bugfix.rst)

    • if you don't have an issue number, change it to the pull request
      number after creating the PR

      • .bugfix: A bug fix for something the maintainers deemed an
        improper undesired behavior that got corrected to match
        pre-agreed expectations.
      • .feature: A new behavior, public APIs. That sort of stuff.
      • .deprecation: A declaration of future API removals and breaking
        changes in behavior.
      • .breaking: When something public is removed in a breaking way.
        Could be deprecated in an earlier release.
      • .doc: Notable updates to the documentation structure or build
        process.
      • .packaging: Notes for downstreams about unobvious side effects
        and tooling. Changes in the test invocation considerations and
        runtime assumptions.
      • .contrib: Stuff that affects the contributor experience. e.g.
        Running tests, building the docs, setting up the development
        environment.
      • .misc: Changes that are hard to assign to any of the above
        categories.
    • Make sure to use full sentences with correct case and punctuation,
      for example:

      Fixed issue with non-ascii contents in doctest text files
      -- by :user:`contributor-gh-handle`.

      Use the past tense or the present tense a non-imperative mood,
      referring to what's changed compared to the last released version
      of this project.

@bdraco bdraco added backport-3.14 Trigger automatic backporting to the 3.14 release branch by Patchback robot backport-3.15 Trigger automatic backporting to the 3.15 release branch by Patchback robot labels Aug 18, 2026
@pyba-chronographer pyba-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Aug 18, 2026
@codecov

codecov Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

@codspeed

codspeed Bot commented Aug 18, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 84 untouched benchmarks
⏩ 83 skipped benchmarks1


Comparing benchmark-apt-hang (5e8a260) with master (5e95a13)2

Open in CodSpeed

Footnotes

  1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on master (8b9cfb5) during the generation of this report, so 5e95a13 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@bdraco
bdraco marked this pull request as ready for review August 18, 2026 20:19
@greptile-apps

greptile-apps Bot commented Aug 18, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge with no actionable blocking or non-blocking issues identified.

The workflow now bounds each apt operation, retries failed installations, and retains a job-level timeout while preserving the benchmark execution path after successful installation.

Reviews (1): Last reviewed commit: "Reword change note for the spell checker" | Re-trigger Greptile

@bdraco
bdraco merged commit 069e9cf into master Aug 18, 2026
89 of 94 checks passed
@bdraco
bdraco deleted the benchmark-apt-hang branch August 18, 2026 20:28
@patchback

patchback Bot commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Backport to 3.15: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.15/069e9cfc913a937da353baaf8244ec8023d91eb3/pr-13489

Backported as #13490

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@patchback

patchback Bot commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

bdraco added a commit that referenced this pull request Aug 18, 2026
… on apt (#13490)

Co-authored-by: J. Nick Koston <nick@koston.org>
bdraco added a commit that referenced this pull request Aug 19, 2026
… on apt (#13491)

Co-authored-by: J. Nick Koston <nick@koston.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-3.14 Trigger automatic backporting to the 3.14 release branch by Patchback robot backport-3.15 Trigger automatic backporting to the 3.15 release branch by Patchback robot bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant