rootless

[ibuildthecloud]

This is a POC to get k3s working rootless. Using this PR you can run

env "PATH=$(pwd)/bin:$PATH" bash -x ./scripts/dev-server.sh --rootless --data-dir $(pwd)/

And it will start. You need slirp4netns, nsenter, and socat on your path. It will eventually fail when containerd goes to start the pod with

E0308 10:05:01.440867   20864 pod_workers.go:190] Error syncing pod 3b161e3e-41bc-11e9-a740-4e928cc79e65 ("coredns-7748f7f6df-gxdf5_kube-system(3b161e3e-41bc-11e9-a740-4e928cc79e65)"), skipping: failed to "CreatePodSandbox" for "coredns-7748f7f6df-gxdf5_kube-system(3b161e3e-41bc-11e9-a740-4e928cc79e65)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-7748f7f6df-gxdf5_kube-system(3b161e3e-41bc-11e9-a740-4e928cc79e65)\" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:344: starting container process caused \"process_linux.go:275: applying cgroup configuration for process caused \\\"mkdir /sys/fs/cgroup/cpuset/kubepods: permission denied\\\"\": unknown"

EDIT:

This PR works now, but still going to do more development to make the networking not suck so bad. Right now a new network namespace is created for the process but no ports are bound to the host. So you can't do anything with the launched process unless you do nsenter -n -t ${PID}. I'm going to write a controller that watches servicelb ports and binds ports through rootlesskit's API. Will be cool.

[AkihiroSuda]

👍

Currently containerd needs disable_cgroup = true: https://github.com/rootless-containers/usernetes/blob/7946d5cdfb9b4dc7b3907ca3248008dea07ec60b/boot/containerd.sh#L16

For supporting cgroup, we need to wait for runc to support cgroup v2 with nsdelegate (or we could use pam_cgfs.so with existing cgroup v1, but it is not adopted in Fedora/RHEL due to a security concern: containers/podman#1429)

[AkihiroSuda]

Tracker for bringing cgroup2 to OCI is here: opencontainers/runtime-spec#1002

[ibuildthecloud]

@AkihiroSuda I was looking through all the usernetes scripts but somehow I glanced over that you disabled cgroups. Thanks!

[ibuildthecloud]

@AkihiroSuda Thanks, that was it. I got it work now. Incredibly magical!!!

k3s server --rootless or k3s agent --rooless ...

[dur3x]

@ibuildthecloud Does this option fix this problem I got on certain devices ? (see below)

q201:/data/data/berserker.android.apps.sshdroid/home # ./k3s-arm64 server
INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/a76eac31f2b7bd3c07a844671a9405198e33f3c29e3c2f441bc4866bf476421f 
INFO[0000] error extracting tarball into /var/lib/rancher/k3s/data/a76eac31f2b7bd3c07a844671a9405198e33f3c29e3c2f441bc4866bf476421f-tmp after 0 files, 0 dirs, 4.367833ms: mkdir /var: read-only file system 
FATA[0000] extracting data: mkdir /var: read-only file system 

[ibuildthecloud]

@dur3x it could, but it currently doesn't. If running in rootless mode /var needs to exist on the host. Do you have no /var at all? Is this andriod? Right now I need /var, /etc/, /tmp to exist on the host. You don't need any privileges to them, just exist.

[erikwilson]

@dur3x Are you running on a rooted android device? It looks like you are running as root and are trying to write to /var which is mounted read-only. You will need to remount /var as read-write, or try using the --data-dir flag to specify a writable directory. It may not matter tho unless your kernel supports user namespaces, if ls /proc/self/ns/user works then k3s will probably work.

[AkihiroSuda]

snapshotter needs to be set to vfs native except on Ubuntu kernel

[AkihiroSuda]

os.Exit?

[giuseppe]

What storage drivers are supported in rootless mode? Is there any plan for using something like fuse-overlayfs?

[AkihiroSuda]

Only overlay and native (and maybe btrfs depending on the configuration - untested) atm, but I have a plan to add support for fuse-overlayfs as well.

[giuseppe]

That is great. fuse-overlayfs should be already usable with CRI-O as the underlying storage library is the same as Podman and Buildah, but I have not tried that yet

[dur3x]

@dur3x it could, but it currently doesn't. If running in rootless mode /var needs to exist on the host. Do you have no /var at all? Is this andriod? Right now I need /var, /etc/, /tmp to exist on the host. You don't need any privileges to them, just exist.

No /var at all.

@dur3x Are you running on a rooted android device? It looks like you are running as root and are trying to write to /var which is mounted read-only. You will need to remount /var as read-write, or try using the --data-dir flag to specify a writable directory. It may not matter tho unless your kernel supports user namespaces, if ls /proc/self/ns/user works then k3s will probably work.

ls /proc/self/ns/user doesn't work :-(
Indeed it's a rooted android device. By remounting some partitions as read-write the process goes further but it still not working completely:

WARN[2019-03-10T13:59:25.563442726Z] Failed to find cpuset cgroup, you may need to add "cgroup_enable=cpuset" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi) 
ERRO[2019-03-10T13:59:25.563571142Z] Failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)
FATA[2019-03-10T13:59:25.563653767Z] failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)

But this is another thing and is the same as #192

[borg286]

@dur3x it could, but it currently doesn't. If running in rootless mode /var needs to exist on the host. Do you have no /var at all? Is this andriod? Right now I need /var, /etc/, /tmp to exist on the host. You don't need any privileges to them, just exist.

No /var at all.

@dur3x Are you running on a rooted android device? It looks like you are running as root and are trying to write to /var which is mounted read-only. You will need to remount /var as read-write, or try using the --data-dir flag to specify a writable directory. It may not matter tho unless your kernel supports user namespaces, if ls /proc/self/ns/user works then k3s will probably work.

ls /proc/self/ns/user doesn't work :-(
Indeed it's a rooted android device. By remounting some partitions as read-write the process goes further but it still not working completely:

WARN[2019-03-10T13:59:25.563442726Z] Failed to find cpuset cgroup, you may need to add "cgroup_enable=cpuset" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi) 
ERRO[2019-03-10T13:59:25.563571142Z] Failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)
FATA[2019-03-10T13:59:25.563653767Z] failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)

But this is another thing and is the same as #192

For the Android environment, is it enough to assume one is running it inside of the UserLand app? Or are you trying to make an app out of this and want to restrict yourself to only the directories natively available?

[dur3x]

For the Android environment, is it enough to assume one is running it inside of the UserLand app? Or are you trying to make an app out of this and want to restrict yourself to only the directories natively available?

For me yes it's enough. My first goal is to be able to run the k3s server (or at least the agent part).

[AkihiroSuda]

Let me know if I can be of any help to get this merged

[ibuildthecloud]

I'm going to merge this although there really are a lot of usability issues. I'll document the issues.