I've been interested in using gVisor on my k8s cluster for a while, most particularly since the vulnpocalypse has gone underway. But the lack of SELinux support is quite tricky to get around, because there's no equivalent there for --security-opt label=disable. You can set the context to spc_t for a conceptual equivalent...but that's still filling in the fields in the spec, so gVisor sees it and bails.

This led me to an experiment where I created this horrid Python wrapper script that would check if a container was using spc_t and, if so, set that to the exec context, clear it from the spec file, and run gVisor. This "worked" because, again, spc_t is effectively unprivileged, so runsc could just run like normal.

But then I wondered...can runsc support SELinux? No no I don't mean inside the container (that would be cool but is definitely far beyond my current abilities!). Rather, can we apply the given label to runsc itself, so that the sentry and gofer are both confined within the label?

And thus, we have this RFC. Although it's very WIP, but it does work, well enough for me to test it out on my actual cluster. There are some...notable quirks here:

• The extent of my testing is make docker-tests and some manual smoke tests.
• In order to get docker-tests to run, I automatically rewrite all mounts to use binds instead with :Z (because the mount API doesn't support relabeling), with some manual exclusions. Obviously very ugly, but I'm not sure what the desired result would look like.
• CI isn't exactly running the normal tests on an SELinux-enabled setup. That would require external work on the CI configs, I presume, to e.g. run inside an SELinux-enabled VM.
• ...and additionally, there are no automated tests yet to make sure confinement actually works. It's all been manual testing (e.g. checking that not-relabeled files are not accessible, verifying /proc/PID/attr/current uses container_t).
• I have not tested things such as GPU access or checkpointing (beyond what the integration tests do, at least).
  • For checkpointing, my understanding is that as-is, if you try to pass fds pointing to checkpoint files over a socket into a confined process, and those files were relabeled (e.g. by copying them), it won't work? This is probably trivially fixable, but I didn't really want to mess too much with something I couldn't reliably test out yet (see above point )
• Right now, this requires a very small custom policy, in order to let runsc create the control socket in a way such that the container can listen on it. This is actually quite trivial, and I expect I could get it into upstream container-selinux, but I didn't want to commit (pun intended?) to that just yet.
• Once the processes are confined, they can no longer write to log-fd, since we don't really have any control over who created that and what it's pointing to. The ideal procedure here would probably be for the unconfined outer layer to install an intermediary that forwards the logs? I'm not sure if there's a better place for that.
• If you try to mount a path without relabeling it (:Z/:z), with directfs enabled, runsc will fail to start. This appears to be the existing behavior of runsc when -v has a path it does not have read access to, but it's a bit weird in the SELinux case, since it's more likely someone will accidentally pass a path without relabeling it. lisafs does not have this problem. I'm not really sure what the ideal solution would be here.
  • Relatedly, if you mount a path where the outer folder is relabeled, but the contents are not, it appears empty. This is because access to the parent directory lets you list the filenames, and gVisor swallows access denials that occur during the actual stat. IMO even w/o SELinux this should actually be showing the file as 000 anyway, so it's clearer to the container that something's wrong...
• There are some parts of the code where the existing semantics are a bit unclear to me, see the location of the why is it significant that the binary is opened now comment in particular, as well as the semantics around restarting (there's just a lot of edge cases there, so it's hard to tell if I missed one).

Is SELinux support necessary? gVisor does of course have its own quite durable sandbox, but I think having more defense-in-depth is always useful, and this would make it pretty hard for even a sandbox hole to turn into e.g. more arbitrary forms of filesystem access, as well as limit the ability for any arbitrary-code-execution vulns to use sketchy syscalls that somehow get through seccomp (e.g. SELinux can block AF_ALG system-wide, without any seccomp changes).

That leads me to: does the project want this? I outlined many gaps above, but these are all very fixable things. I just don't want to start committing to making this production-ready if it's wholly undesired. But personally I do think it would be useful! (If not deemed useful enough, there probably should be a better story in using gVisor in an SELinux-enabled cluster, e.g. just only allowing spc_t or similar, but that's for another day I guess.)

If anyone wants to test this, you can just use the binaries and chcon system_u:object_r:container_runtime_exec_t:s0 on runsc itself + the sidecar binaries (the containerd shim doesn't need it), like the makefile does. It's unclear to me if I broke CI somewhere here, though (I didn't really look too hard for extra linters and such that I was supposed to be running), so getting the binaries might require building yourself.

Assisted-by: Kimi K3