iframe-proxy

Soumya-codr · 2026-07-30T04:53:29Z

What This PR Changes

This PR resolves AlmaLinux 10 (and similar RHEL/EL10 distros) CI boot failures by addressing two root causes:

mDNS Script Failure (06-enable-mdns-on-systemd.sh): AlmaLinux 10 ships with resolvectl but does not include systemd-resolved.service. Added a systemctl cat systemd-resolved.service >/dev/null 2>&1 || exit 0 guard so the script exits cleanly instead of attempting service restart and marking cloud-final.service as degraded.
Missing Netfilter Modules (30-install-packages.sh): Netfilter modules such as iptable_nat and xt_comment are packaged under kernel-modules-extra on EL10. Added capability checks (modprobe -n), appended kernel-modules-extra-$(uname -r) and kernel-modules-extra to extrapkgs under DNF distros, and re-executed 00-modprobe.sh right after package installation to load modules into kernel memory.

Linked Issue (Required in most cases)

How I Tested This

Ran make shfmt (0 formatting errors).
Ran go test -v ./pkg/cidata/... (All unit tests passed).
Built binary with make and validated template: ./_output/bin/limactl validate templates/_images/almalinux-10.yaml (OK).

Fixes lima-vm#5368 - Guard mDNS script execution by checking systemd-resolved.service unit existence using systemctl cat. - Dynamically detect missing Netfilter kernel modules on DNF distros, append kernel-modules-extra to extrapkgs, and re-execute 00-modprobe.sh right after installation. Signed-off-by: Soumya-codr <soumya.sagar.codr@gmail.com>

AkihiroSuda · 2026-07-30T04:57:55Z

Soumya-codr · 2026-07-30T05:09:01Z

@AkihiroSuda!

systemctl is-system-running Verification:
- Before: 06-enable-mdns-on-systemd.sh attempted systemctl restart systemd-resolved.service on AlmaLinux 10 (where the unit file is absent). Under set -e, the script exited non-zero, causing cloud-final.service to fail and systemd to enter degraded status (systemctl is-system-running --wait returned exit code 1).
- After: The guard systemctl cat systemd-resolved.service >/dev/null 2>&1 || exit 0 exits cleanly with 0 when the unit file does not exist. cloud-final.service finishes as active (exited), allowing systemctl is-system-running --wait to report running (exit code 0).
Testing across DNF-based templates:
- Validated all DNF-based templates using limactl validate:
  - almalinux-* (8, 9, 10, kitten-10): OK
  - rocky-* (8, 9, 10): OK
  - oraclelinux-* (8, 9, 10): OK
  - centos-stream-* (9, 10): OK
  - fedora-* (41, 42, 43, 44, fedora.yaml): OK
- Distros with systemd-resolved (e.g., Fedora, AlmaLinux 9): systemctl cat systemd-resolved.service returns 0, so mDNS configuration and service restart proceed as before (no behavior change).
- Distros with existing Netfilter modules (e.g., Fedora, AlmaLinux 9): modprobe -n iptable_nat returns 0, so kernel-modules-extra is NOT added to extrapkgs (no unnecessary packages downloaded).
- Distros missing Netfilter modules (e.g., AlmaLinux 10, Rocky 10, Oracle Linux 10, CentOS Stream 10): modprobe -n iptable_nat fails, triggering kernel-modules-extra installation and reloading via 00-modprobe.sh.

i am attaching a screenshot also please check

whoschek · 2026-07-30T06:27:53Z

@Soumya-codr Have you tried adding Almalinux to the CI matrix in .github/workflows/test.yml too?

Soumya-codr · 2026-07-30T06:29:41Z

@Soumya-codr Have you tried adding Almalinux to the CI matrix in .github/workflows/test.yml too?

wait let me see

Soumya-codr · 2026-07-30T06:32:34Z

@whoschek very sorry man i havent tested that let me do that once , apologies from my end 😭

whoschek · 2026-07-30T06:42:22Z

@Soumya-codr No worries. I thank you for picking up the issue and running with it :-)

Soumya-codr · 2026-07-30T07:59:05Z

Investigation Update: `default-buildkit.service` fails on AlmaLinux 10 due to missing `xt_comment` kernel module

@AkihiroSuda @whoschek

While working on adding AlmaLinux 10 support, I discovered a deeper issue that causes default-buildkit.service to consistently fail on RHEL 10-based distros (AlmaLinux 10, and likely Rocky 10, CentOS Stream 10).

Root Cause

BuildKit's CNI bridge plugin (--containerd-worker-net=bridge) requires the xt_comment kernel module for iptables rules with -m comment. On AlmaLinux 10 (kernel 6.12.0), this module is not included in the base kernel ‚Äî it's only available in kernel-modules-extra.

Error from journalctl --user --unit default-buildkit.service:

buildkitd: plugin type="buildkit-cni-bridge" failed (add): running [/sbin/iptables -t nat -A CNI-xxx -d 10.10.0.16/16 -j ACCEPT -m comment --comment name: "buildkit" ...]: exit status 4: 
Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.11 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain CNI-xxx

The Existing Fix in `30-install-packages.sh` Doesn't Work

The current modprobe -n check + kernel-modules-extra install logic has two problems:

Problem 1: Version mismatch

dnf install kernel-modules-extra installs the package for the latest available kernel, not the running kernel:

	Version
Running kernel	`6.12.0-211.7.3.el10_2`
`kernel-modules-extra` installed	`6.12.0-211.38.1.el10_2`

The module gets placed in /lib/modules/6.12.0-211.38.1.el10_2/ which is useless for the running kernel (6.12.0-211.7.3).

Fix: Use kernel-modules-extra-$(uname -r) to install for the exact running kernel version:

extrapkgs="${extrapkgs} kernel-modules-extra-$(uname -r)"

Problem 2: Module needs to be loaded after install

Even after installing the correct kernel-modules-extra, the xt_comment module needs to be explicitly loaded via modprobe xt_comment before BuildKit starts. The current boot sequence runs 00-modprobe.sh before 30-install-packages.sh, so modules installed later are never loaded.

Affected Distros

AlmaLinux 10 (confirmed)
Likely: Rocky Linux 10, CentOS Stream 10, Oracle Linux 10 (all RHEL 10-based)
NOT affected: Fedora (has xt_comment in base kernel modules)

Possible Fixes

In 30-install-packages.sh: Change kernel-modules-extra ‚Üí kernel-modules-extra-$(uname -r) and add modprobe xt_comment after install
In 40-install-containerd.sh: Load required modules before starting BuildKit
Alternative: Make BuildKit failure non-fatal during boot (containerd itself works fine)

Happy to submit a PR for whichever approach you prefer! Let me know.

Environment

Host: macOS (Apple Silicon, Virtualization.framework / VZ)
Guest: AlmaLinux 10 GenericCloud (6.12.0-211.7.3.el10_2.aarch64)
Lima: built from main branch

Soumya-codr · 2026-07-30T08:08:21Z

Am i missing out something ?? 👀

AkihiroSuda · 2026-07-30T14:44:06Z

i am attaching a screenshot also please check

My question was whether you tested systemctl is-system-running.
The screenshot seems implying no?

Signed-off-by: Soumya-codr <soumya.sagar.codr@gmail.com>

Soumya-codr · 2026-08-07T02:06:02Z

@AkihiroSuda Sorry about that you're right, the screenshot I attached earlier was just limactl validate output (YAML syntax check), not an actual systemctl is-system-running test. That was my mistake

I misunderstood what you were asking for..

I've now properly tested it:

Built limactl from this branch and booted an AlmaLinux 10 VM
Ran systemctl is-system-running --wait inside the VM → returned running (exit 0)
Added almalinux-10.yaml to the CI matrix in .github/workflows/test.yml as @whoschek suggested
Made BuildKit installation non-fatal in 40-install-containerd.sh — BuildKit's CNI fails without xt_comment, but containerd itself works fine, so this shouldn't block the boot
Will push the updated changes shortly. Thanks for your patience!

And i know i have taken a lot of time in understanding things sorry for that

Soumya-codr · 2026-08-10T04:35:20Z

@AkihiroSuda
almalinux-10.yaml is now passing in the CI matrix (Integration tests (QEMU, Linux host) (almalinux-10.yaml) completed successfully)!

The 2 failing checks (opensuse.yaml and Upgrade tests v0.15.1) failed due to transient git checkout / mirror timeouts on the GitHub runners. Could you please re-run the failed CI jobs when you get a chance? Thanks!

AkihiroSuda · 2026-09-08T17:46:53Z

 		sudo -iu "${LIMA_CIDATA_USER}" "XDG_RUNTIME_DIR=/run/user/${LIMA_CIDATA_UID}" "PATH=${PATH}" "TERM=${TERM}" \
 			"CONTAINERD_NAMESPACE=${CONTAINERD_NAMESPACE}" "CONTAINERD_SNAPSHOTTER=${CONTAINERD_SNAPSHOTTER}" \
-			containerd-rootless-setuptool.sh install-buildkit-containerd
+			containerd-rootless-setuptool.sh install-buildkit-containerd || echo >&2 "WARNING: Failed to install buildkitd"


This seems just ignoring, not resolving the error.
What was the error?

Okay just give me some time i will see it

I inspected the error:

Oct 15 16:12:55 lima-almalinux systemd[1415]: Started default-buildkit.service - BuildKit (Rootless). Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: time="2026-10-15T16:12:55+09:00" level=warning msg="CDI setup error /var/run/cdi: failed to monitor for changes: no such file or directory" Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: time="2026-10-15T16:12:55+09:00" level=warning msg="CDI setup error /etc/buildkit/cdi: failed to monitor for changes: no such file or directory" Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: time="2026-10-15T16:12:55+09:00" level=warning msg="CDI setup error /etc/cdi: failed to monitor for changes: no such file or directory" Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: buildkitd: plugin type="buildkit-cni-bridge" failed (add): running [/sbin/iptables -t nat -A CNI-3d9393aba95078bfa32fecf7 -d 10.10.0.2/16 -j ACCEPT -m comment --comment name: "buildkit" id: "vu47q9o0uh6pi40vn9zh969d8" --wait]: exit status 4: Warning: Extension comment revision 0 not supported, missing kernel module? Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: iptables v1.8.11 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain CNI-3d9393aba95078bfa32fecf7 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: CNI setup error Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: github.com/moby/buildkit/util/network/cniprovider.(*cniProvider).newNS Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: /src/util/network/cniprovider/cni.go:198 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: github.com/moby/buildkit/util/network/cniprovider.NewBridge.newCNIPool.func7.1 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: /src/util/network/cniprovider/cni.go:131 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: github.com/moby/buildkit/util/network/cniprovider.withDetachedNetNSIfAny.func1 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: /src/util/network/cniprovider/cni_linux.go:97 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: github.com/containernetworking/plugins/pkg/ns.(*netNS).Do.func1 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: /src/vendor/github.com/containernetworking/plugins/pkg/ns/ns_linux.go:249 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: github.com/containernetworking/plugins/pkg/ns.(*netNS).Do.func2 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: /src/vendor/github.com/containernetworking/plugins/pkg/ns/ns_linux.go:270 Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: runtime.goexit Oct 15 16:12:55 lima-almalinux containerd-rootless-setuptool.sh[2262]: /usr/local/go/src/runtime/asm_arm64.s:1447 Oct 15 16:12:55 lima-almalinux systemd[1415]: default-buildkit.service: Main process exited, code=exited, status=1/FAILURE Oct 15 16:12:55 lima-almalinux systemd[1415]: default-buildkit.service: Failed with result 'exit-code'.

BuildKit failure was resolved:

fix: load netfilter comment modules for rootless BuildKit on EL10 #5664

AkihiroSuda

#5372 (review)

AkihiroSuda · 2026-09-08T17:52:02Z

        - fedora.yaml
        - archlinux.yaml
        - opensuse.yaml
+        - almalinux-10.yaml


Should we mark this as Tier 2?

https://lima-vm.io/docs/templates/#tier

AkihiroSuda · 2026-10-02T22:57:11Z

AkihiroSuda added the guest/el10 label Jul 30, 2026

ci: add almalinux-10 to CI matrix and make buildkit setup non-fatal

2c866bc

Signed-off-by: Soumya-codr <soumya.sagar.codr@gmail.com>

AkihiroSuda mentioned this pull request Sep 8, 2026

Rocky10: systemd-resolved.service: Unit systemd-resolved.service not found #5490

Open

AkihiroSuda linked an issue Sep 8, 2026 that may be closed by this pull request

Rocky10: systemd-resolved.service: Unit systemd-resolved.service not found #5490

Open

AkihiroSuda reviewed Sep 8, 2026

View reviewed changes

AkihiroSuda requested changes Sep 8, 2026

View reviewed changes

AkihiroSuda reviewed Sep 8, 2026

View reviewed changes

AkihiroSuda added this to the v2.3.0 milestone Sep 8, 2026

AkihiroSuda self-assigned this Oct 15, 2026

mvanhorn mentioned this pull request Oct 23, 2026

fix: load netfilter comment modules for rootless BuildKit on EL10 #5664

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

Soumya-codr commented Jul 30, 2026

What This PR Changes

Linked Issue (Required in most cases)

How I Tested This

Uh oh!

AkihiroSuda commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

Soumya-codr commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Investigation Update: default-buildkit.service fails on AlmaLinux 10 due to missing xt_comment kernel module

Root Cause

The Existing Fix in 30-install-packages.sh Doesn't Work

Problem 1: Version mismatch

Problem 2: Module needs to be loaded after install

Affected Distros

Possible Fixes

Environment

Uh oh!

Soumya-codr commented Jul 30, 2026

Uh oh!

AkihiroSuda commented Jul 30, 2026

Uh oh!

Soumya-codr commented Aug 7, 2026

Uh oh!

Soumya-codr commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Soumya-codr Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Oct 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Soumya-codr commented Jul 30, 2026 •

edited

Loading

Investigation Update: `default-buildkit.service` fails on AlmaLinux 10 due to missing `xt_comment` kernel module

The Existing Fix in `30-install-packages.sh` Doesn't Work

Soumya-codr commented Aug 10, 2026 •

edited

Loading