feat: add sysbox extension support - #1171
Conversation
| # sysbox-fs mounts its per-container FUSE filesystem by executing | ||
| # fusermount3, which it looks up in PATH | ||
| - url: https://github.com/libfuse/libfuse/releases/download/fuse-{{ .FUSE3_SYSBOX_VERSION }}/fuse-{{ .FUSE3_SYSBOX_VERSION }}.tar.gz | ||
| destination: fuse.tar.gz |
There was a problem hiding this comment.
fuse should be already available as a separate extension, is there a reason to dupliate?
There was a problem hiding this comment.
It needs it in a particular location and I couldn't quite figure out how to do it effectively
Should I document the dependency and use ln or a similar concept ?
There was a problem hiding this comment.
Oh and also, sysbox uses fuse3 in its sysbox-fs process which runs in the static pod container which would need binding with Talos' libc since the fuse3 extension is not statically built
There was a problem hiding this comment.
I'm lost on this, if it runs it on the host, host has the libs; if it runs within the pod, why does it need on the host (in the extension). Neither makes sense to me.
There was a problem hiding this comment.
Because no image is built for the pod, the pod is just there to be in the same PID namespace as the kubelet which I didn't figure out how to do with a normal Talos service
There was a problem hiding this comment.
I'm not sure what is a pod there in this description. It either runs within the host, or not. Also we have gvisor already which works without any problem.
There was a problem hiding this comment.
It's multiple things:
gvisor doesn't have long-running daemons though in its model it adds additional processes to the sandboxes of containers and the CRI talks to them directly since it originated them.
sysbox-runc on the other hand relies on sysbox-mgr and sysbox-fs to be preexisting processes and communicates with them. When sysbox-runc creates a sandbox, it gets hosts PIDs and communicates those to sysbox-mgr and sysbox-fs.
Talos runs extension services in a private PID namespace, but sysbox-mgr AND sysbox-fs are handed host PIDs by sysbox-runc and they do calls like pidfd_open(2) and also rely on seccomp notifications on those PIDs.
Since those PIDs are hosts PIDs, sysbox-mgr AND sysbox-fs need to be on the host PID namespace, which is not possible through Talos' services. That's where the decision of having static pods comes from.
The pods have no images because the build process doesn't have a way of building images for those, so that's why it's relying on the rootfs to have the necessary binaries to run properly.
There was a problem hiding this comment.
Talos 1.14 does CRI sandboxing, if you mean that, but that is optional.
Static pods is not a great ideas, as it requires e.g. kubelet to be running, so in theory kubelet can't use a new container runtime.
All in all, idk is it too painful and hacky? Does it have enough value to fit in between e.g. gvisor and kata? Can't you just use kata instead? Yes, it has bigger overhead, but less hacks, and truly good isolation?
There was a problem hiding this comment.
No I do not mean CRI sandboxing, but yeah CRI sandboxing will pose another challenge to run sysbox.
Also Kata's problem is not overhead, Kata cannot run every workload the way Sysbox can.
For example Kata's PID 1 inside the microVM is kata-manager (or something like that) meaning you cannot run systemd inside a container, this is something you can easily do using Sysbox because of fs emulation and partial kernel emulation. Same for dockerd, overlayfs, iptables and cgroups issues are all handled by those features.
Supposedly, vnode would also solve the same problem, but it probably will be following the same licensing as vCluster
|
|
||
| # the sysbox packages ship statically linked binaries (they are built | ||
| # with `make sysbox-static`), so they can be used as-is on Talos | ||
| ar p sysbox-ce.deb data.tar.xz | tar -xJ -C sysbox-ce \ |
There was a problem hiding this comment.
is there any reason we can't build sysbox from source? any problem with the toolchain?
There was a problem hiding this comment.
No I kinda went the easy way since their pipeline used to be more annoying, but it wasn't that bad, it's better now though
This commit aims to add sysbox extension to the catalog. Signed-off-by: Yewolf <yyewolf@gmail.com>
4 participants
This is an attempt to resolve the following issue: #431
Here's my use case for sysbox:
I self-host CI at home for various projects (private and public) and have a need to be able to run exotic workload during those jobs (notably kind in kubernetes)
I tried to do using Kata or gvisor but the restrictions (especially on networking) are making the test environment flakier and also unrealistic
This is only a draft and a request to make this real, however I am unsure about the static pod approach to run the sysbox components.
I successfully run this exact setup on my cluster without any issues and this is the kind of stuff I'm trying to run: https://woodpecker.yewolf.fr/repos/5/pipeline/316/5 (this run is on talos with the extension)