ci(build): isolate each PR build on its own ephemeral runner VM by myasnikovdaniil · Pull Request #2939 · cozystack/cozystack · GitHub
Skip to content

ci(build): isolate each PR build on its own ephemeral runner VM#2939

Closed
myasnikovdaniil wants to merge 3 commits into
ci/build-cache-mode-maxfrom
ci/build-ephemeral-runner
Closed

ci(build): isolate each PR build on its own ephemeral runner VM#2939
myasnikovdaniil wants to merge 3 commits into
ci/build-cache-mode-maxfrom
ci/build-ephemeral-runner

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

What this PR does

Runs the PR Build job on an ephemeral one-job-per-VM runner (oracle-vm-4cpu-16gb-x86-64) instead of the shared self-hosted runner. The debug label still routes to self-hosted for the interactive breakpoint path (mirrors the existing e2e job idiom).

Why. The shared self-hosted runner runs several runner agents against a single dockerd-embedded BuildKit. Concurrent build jobs serialize on BuildKit's single-writer bbolt cache-metadata lock + exporter mutex and stall to the 30-minute job timeout (root-caused via a live goroutine dump — full analysis in #2937). A standalone build of an image is ~2 min; under concurrent multi-PR load the same image hangs ~30 min while CPU/RAM/disk stay idle. It's the sharing, not resource exhaustion.

One VM per job ⇒ one BuildKit per job ⇒ the cross-job lock contention is severed by construction.

Sizing. make build is serial (one image at a time, no -j), so it does not need the e2e job's 24-CPU shape — a 4cpu/16gb VM is enough. It relies on the warm mode=max registry cache (#2938, the base of this PR) to stay well under the 30-min wall on cold VMs.

This is Phase 0.5 of #2937: the interim contention fix is sequenced with the cache layer (#2938), since the cache is what isolated/ephemeral builders need to start hot — it is not itself the hang fix.

Stacked on #2938

Base branch is ci/build-cache-mode-max (#2938). Merge #2938 first (or merge this into it); ephemeral builders depend on #2938's cache to start hot.

Verification gates (CI-observable, settle before merge)

  • oracle-vm-4cpu-16gb-x86-64 is genuinely one-job-per-VM (if VMs are reused/co-hosted, contention returns)
  • the runner image carries the full build toolchain (buildx, skopeo, jq, gh, helm, mikefarah yq, GNU tar/sed/awk) and can run the privileged talos-nocloud imager
  • cold-VM full build stays under timeout-minutes: 30 (bump the shape or the timeout if it does not)

Release note

NONE

Part of #2937.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added area/build Issues or PRs related to image build infrastructure, multi-arch support size/S This PR changes 10-29 lines, ignoring generated files labels Jun 17, 2026
@dosubot dosubot Bot added the area/ci Issues or PRs related to CI workflows, GitHub Actions, automation label Jun 17, 2026
@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4c4614a0-6712-4734-b165-118bde43eaf0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/build-ephemeral-runner

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

The shared self-hosted runner runs 4 agents against one dockerd-embedded buildkit; concurrent build jobs serialize on buildkit's single-writer bbolt cache lock + exporter mutex and stall to the 30-min timeout (proven via a live SIGUSR1 goroutine dump of a hung build). Run the Build job on a small ephemeral oracle-vm shape (4cpu/16gb x86-64) instead, so each build gets its own VM and its own buildkit -- the cross-job contention is severed by construction.

make build is serial (one image at a time), so the small shape suffices; it relies on the warm mode=max cache (#2938) to stay under the 30-min wall. The `debug` label still routes to self-hosted for the breakpoint path.

Phase 0.5 of #2937.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The ephemeral oracle-vm image doesn't carry the helm unittest plugin that the persistent self-hosted runner had installed, so 'make unit-tests' fails immediately with 'unknown command "unittest" for "helm"'. Install it if absent before the test step (idempotent; the debug/self-hosted path already has it).

Stopgap only -- the durable fix is a container-based builder (separate PR) that bakes the full toolchain into an image rather than depending on host-accreted tools.

Part of #2937.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The installer's image-packages target runs 'flux push artifact' to publish the cozystack-packages OCI artifact, whose digest is written into the installer values.yaml (platformSourceUrl/Ref) and pulled by cozystack-operator as its PackageSource. flux is absent on the ephemeral oracle-vm image, so 'make build' fails at image-packages with 'flux: not found'. Install it if absent (idempotent), alongside the helm-unittest plugin.

Stopgap -- the durable fix is to bake both tools into the oracle-vm runner image and drop this step.

Part of #2937.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@myasnikovdaniil myasnikovdaniil force-pushed the ci/build-cache-mode-max branch from 478509d to e1781ac Compare June 17, 2026 16:03
@myasnikovdaniil myasnikovdaniil force-pushed the ci/build-ephemeral-runner branch from 36dfcaa to b8c886c Compare June 17, 2026 16:03
@myasnikovdaniil myasnikovdaniil marked this pull request as draft June 19, 2026 08:47
myasnikovdaniil added a commit that referenced this pull request Jun 22, 2026
## What

Consolidated landing branch for a set of interdependent platform/CI
stabilization fixes. It began as the source PRs in the table below (in
dependency order) and has since grown with review-driven fixes and a few
production-behavior changes — the full current inventory is under
**Scope** below. Landing them as one unit lets CI run against the real
combined state instead of a fragile bottom-up merge train.

## Source PRs (dependency order)

| # | Commit | Source PR | Addresses |
|---|--------|-----------|-----------|
| 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi
crashloop during cert provisioning |
| 2 | test(e2e): bucket.bats port-forward + S3 client reliability |
#2944 | flaky bucket test |
| 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop
volumeSizeLimitMB | #2943 | missing bucket/access classes |
| 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps)
| #2874 | cilium "IP already in use" leak |
| 5 | refactor(build): mode=max registry cache + main-only warmer |
#2938 | build cache |
| 6 | ci(build): isolate each PR build on its own ephemeral runner VM |
#2939 | shared-buildkit contention |
| 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline |
#2928 | LINSTOR provisioning timeouts |
| 8 | test(metallb): assert digest-pinned image form, not version
literal | #2873 | brittle metallb assertion |

## Why consolidated

These form a dependency DAG (verified from CI logs): e.g. 2943 needs
2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested
bottom-up, each lower PR runs with none of the fixes above it and can't
go green alone. This branch carries all of them, so CI runs against the
real combined state.

## Verification status

CI is green on the latest head (`5653c30`): full-suite E2E passes
end-to-end and is reproducible (2 of 3 runs on this SHA green). The one
red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the
sandbox, unrelated to the diff — a different environmental subsystem
failed each run (details in the comments below). The earlier 3-hour
crust-gather snapshot hang is fixed in `5653c30`.

## Scope beyond the original 8

The branch has grown past the 8 source PRs above with review-driven
fixes and a few production-behavior changes. For reviewer transparency,
the full set:

**Review fixes (@lexfrei review):**
- **B1 (blocker)** — `8f41910` converts the SeaweedFS
`-lock`/`-readonly` BucketClasses and the s3 service-name override into
`patches/`, wired into `make update`, so a re-vendor no longer drops
them.
- **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods
(both delete remedies were RBAC-forbidden).
- **FU3** — `014de3f` adds a unit test for the HelmRelease update
conflict-retry path.
- **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any
non-terminal owner phase, not just `Running`.
- **FU7** — `f3914d2` sets `persist-credentials: false` on the
build-cache checkout (SHA-pinning is handled repo-wide in #2849).

**Production-behavior changes (not in the table above):**
- VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring
(`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops
install-time eviction churn.
- cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease
update path (`728f676`).

**E2E robustness follow-ups:** per-test crust-gather snapshots
(`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node
single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget
(`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin
(`95f42aa`).

**Not addressed (by decision):** FU2 — the s3 Service keeps
`seaweedfs.name`-based naming via patch (rendered output unchanged;
cozystack renders `fullname == seaweedfs`, so it matches the
`componentName`/`fullname` siblings in practice). FU6 — fork PR build
push is handled by the existing mirror-to-same-repo-branch workflow.

## Relates to

#2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those
can be closed; otherwise they remain the granular per-PR review path.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added CI build cache warming for main branch builds.
* Added a best-effort in-cluster Cilium endpoint self-heal watchdog for
e2e installs.
* Extended SeaweedFS COSI with object-lock support plus separate
readonly access.

* **Improvements**
* Reduced e2e install flakiness with a shared readiness deadline, safer
waits, and improved cleanup/diagnostics capture.
* Prevented install-time churn by switching VPA update mode to
**Initial**.
  * Added container startup probes and updated SeaweedFS volume sizing.

* **Tests**
* Improved e2e robustness (etcd backup schedule waiting, digest-pinned
image checks, and corrected S3 port-forwarding).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
pull Bot pushed a commit to medampudi/cozystack that referenced this pull request Jun 22, 2026
…stack#2939)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

myasnikovdaniil added a commit that referenced this pull request Jun 25, 2026
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:

- .github/workflows/pull-requests.yaml: keep #2983's
  plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
  resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
  interface (pr-patch + talos-image) is identical on both sides, so e2e depends
  on finalize/build-talos with no input rewiring. main's top-level permissions
  and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Issues or PRs related to image build infrastructure, multi-arch support area/ci Issues or PRs related to CI workflows, GitHub Actions, automation size/S This PR changes 10-29 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant