ci(build): isolate each PR build on its own ephemeral runner VM#2939
ci(build): isolate each PR build on its own ephemeral runner VM#2939myasnikovdaniil wants to merge 3 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The shared self-hosted runner runs 4 agents against one dockerd-embedded buildkit; concurrent build jobs serialize on buildkit's single-writer bbolt cache lock + exporter mutex and stall to the 30-min timeout (proven via a live SIGUSR1 goroutine dump of a hung build). Run the Build job on a small ephemeral oracle-vm shape (4cpu/16gb x86-64) instead, so each build gets its own VM and its own buildkit -- the cross-job contention is severed by construction. make build is serial (one image at a time), so the small shape suffices; it relies on the warm mode=max cache (#2938) to stay under the 30-min wall. The `debug` label still routes to self-hosted for the breakpoint path. Phase 0.5 of #2937. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The ephemeral oracle-vm image doesn't carry the helm unittest plugin that the persistent self-hosted runner had installed, so 'make unit-tests' fails immediately with 'unknown command "unittest" for "helm"'. Install it if absent before the test step (idempotent; the debug/self-hosted path already has it). Stopgap only -- the durable fix is a container-based builder (separate PR) that bakes the full toolchain into an image rather than depending on host-accreted tools. Part of #2937. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The installer's image-packages target runs 'flux push artifact' to publish the cozystack-packages OCI artifact, whose digest is written into the installer values.yaml (platformSourceUrl/Ref) and pulled by cozystack-operator as its PackageSource. flux is absent on the ephemeral oracle-vm image, so 'make build' fails at image-packages with 'flux: not found'. Install it if absent (idempotent), alongside the helm-unittest plugin. Stopgap -- the durable fix is to bake both tools into the oracle-vm runner image and drop this step. Part of #2937. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
478509d to
e1781ac
Compare
36dfcaa to
b8c886c
Compare
## What Consolidated landing branch for a set of interdependent platform/CI stabilization fixes. It began as the source PRs in the table below (in dependency order) and has since grown with review-driven fixes and a few production-behavior changes — the full current inventory is under **Scope** below. Landing them as one unit lets CI run against the real combined state instead of a fragile bottom-up merge train. ## Source PRs (dependency order) | # | Commit | Source PR | Addresses | |---|--------|-----------|-----------| | 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi crashloop during cert provisioning | | 2 | test(e2e): bucket.bats port-forward + S3 client reliability | #2944 | flaky bucket test | | 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop volumeSizeLimitMB | #2943 | missing bucket/access classes | | 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps) | #2874 | cilium "IP already in use" leak | | 5 | refactor(build): mode=max registry cache + main-only warmer | #2938 | build cache | | 6 | ci(build): isolate each PR build on its own ephemeral runner VM | #2939 | shared-buildkit contention | | 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline | #2928 | LINSTOR provisioning timeouts | | 8 | test(metallb): assert digest-pinned image form, not version literal | #2873 | brittle metallb assertion | ## Why consolidated These form a dependency DAG (verified from CI logs): e.g. 2943 needs 2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested bottom-up, each lower PR runs with none of the fixes above it and can't go green alone. This branch carries all of them, so CI runs against the real combined state. ## Verification status CI is green on the latest head (`5653c30`): full-suite E2E passes end-to-end and is reproducible (2 of 3 runs on this SHA green). The one red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the sandbox, unrelated to the diff — a different environmental subsystem failed each run (details in the comments below). The earlier 3-hour crust-gather snapshot hang is fixed in `5653c30`. ## Scope beyond the original 8 The branch has grown past the 8 source PRs above with review-driven fixes and a few production-behavior changes. For reviewer transparency, the full set: **Review fixes (@lexfrei review):** - **B1 (blocker)** — `8f41910` converts the SeaweedFS `-lock`/`-readonly` BucketClasses and the s3 service-name override into `patches/`, wired into `make update`, so a re-vendor no longer drops them. - **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods (both delete remedies were RBAC-forbidden). - **FU3** — `014de3f` adds a unit test for the HelmRelease update conflict-retry path. - **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any non-terminal owner phase, not just `Running`. - **FU7** — `f3914d2` sets `persist-credentials: false` on the build-cache checkout (SHA-pinning is handled repo-wide in #2849). **Production-behavior changes (not in the table above):** - VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring (`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops install-time eviction churn. - cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease update path (`728f676`). **E2E robustness follow-ups:** per-test crust-gather snapshots (`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget (`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin (`95f42aa`). **Not addressed (by decision):** FU2 — the s3 Service keeps `seaweedfs.name`-based naming via patch (rendered output unchanged; cozystack renders `fullname == seaweedfs`, so it matches the `componentName`/`fullname` siblings in practice). FU6 — fork PR build push is handled by the existing mirror-to-same-repo-branch workflow. ## Relates to #2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those can be closed; otherwise they remain the granular per-PR review path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added CI build cache warming for main branch builds. * Added a best-effort in-cluster Cilium endpoint self-heal watchdog for e2e installs. * Extended SeaweedFS COSI with object-lock support plus separate readonly access. * **Improvements** * Reduced e2e install flakiness with a shared readiness deadline, safer waits, and improved cleanup/diagnostics capture. * Prevented install-time churn by switching VPA update mode to **Initial**. * Added container startup probes and updated SeaweedFS volume sizing. * **Tests** * Improved e2e robustness (etcd backup schedule waiting, digest-pinned image checks, and corrected S3 port-forwarding). <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…stack#2939) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch diverged landed a competing single-VM build restructure (#2939) and test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build supersedes #2939: - .github/workflows/pull-requests.yaml: keep #2983's plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact interface (pr-patch + talos-image) is identical on both sides, so e2e depends on finalize/build-talos with no input rewiring. main's top-level permissions and SHA-pinned action refs applied across all jobs. - All other files merged cleanly (only pull-requests.yaml conflicted). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

What this PR does
Runs the PR
Buildjob on an ephemeral one-job-per-VM runner (oracle-vm-4cpu-16gb-x86-64) instead of the shared self-hosted runner. Thedebuglabel still routes toself-hostedfor the interactive breakpoint path (mirrors the existing e2e job idiom).Why. The shared self-hosted runner runs several runner agents against a single dockerd-embedded BuildKit. Concurrent build jobs serialize on BuildKit's single-writer bbolt cache-metadata lock + exporter mutex and stall to the 30-minute job timeout (root-caused via a live goroutine dump — full analysis in #2937). A standalone build of an image is ~2 min; under concurrent multi-PR load the same image hangs ~30 min while CPU/RAM/disk stay idle. It's the sharing, not resource exhaustion.
One VM per job ⇒ one BuildKit per job ⇒ the cross-job lock contention is severed by construction.
Sizing.
make buildis serial (one image at a time, no-j), so it does not need the e2e job's 24-CPU shape — a 4cpu/16gb VM is enough. It relies on the warmmode=maxregistry cache (#2938, the base of this PR) to stay well under the 30-min wall on cold VMs.This is Phase 0.5 of #2937: the interim contention fix is sequenced with the cache layer (#2938), since the cache is what isolated/ephemeral builders need to start hot — it is not itself the hang fix.
Stacked on #2938
Base branch is
ci/build-cache-mode-max(#2938). Merge #2938 first (or merge this into it); ephemeral builders depend on #2938's cache to start hot.Verification gates (CI-observable, settle before merge)
oracle-vm-4cpu-16gb-x86-64is genuinely one-job-per-VM (if VMs are reused/co-hosted, contention returns)timeout-minutes: 30(bump the shape or the timeout if it does not)Release note
Part of #2937.