Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

kata-runtime leaves mounts stranded after cgroup startup failure #2108

Closed
crobinso opened this issue Oct 4, 2019 · 4 comments · Fixed by #2127
Closed

kata-runtime leaves mounts stranded after cgroup startup failure #2108

crobinso opened this issue Oct 4, 2019 · 4 comments · Fixed by #2127
Assignees
@devimc
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.

Comments

@crobinso
Copy link

crobinso commented Oct 4, 2019

When kata-runtime hits issues setting up cgroups, it leaves kata mounts stranded on the host. I don't know enough about the container stack to say for certain this is kata-runtime and not podman or something else, so feedback appreciated.

This is with current kata-runtime git, b3aa770

The initial error is tracked in #2107

# No kata mounts present
$ mount | grep kata
$

# podman run hits cgroup error, expected
$ sudo podman run --security-opt label=disable --runtime=$KATAGIT/kata-runtime --rm -it alpine sh
Error: Could not create cgroup for /vc/kata_machine.slice:libpod:7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2: cgroups: cgroup mountpoint does not exist: OCI runtime error

# but now host is polluted with kata mounts
$ mount | grep kata
overlay on /run/kata-containers/shared/sandboxes/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/rootfs type overlay (rw,nodev,relatime,context="system_u:object_r:container_file_t:s0:c348,c694",lowerdir=/var/lib/containers/storage/overlay/l/2ZCHUQTSUTVAYGOEFFSXG7FN3Z,upperdir=/var/lib/containers/storage/overlay/abd8df4bcd5d2958967ab9965a2c2810e56b55f77f8a5b068acaa4742aee067b/diff,workdir=/var/lib/containers/storage/overlay/abd8df4bcd5d2958967ab9965a2c2810e56b55f77f8a5b068acaa4742aee067b/work,metacopy=on)
tmpfs on /run/kata-containers/shared/sandboxes/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2-5710b68ade218590-secrets type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run/kata-containers/shared/sandboxes/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2-acedc29709b485b2-resolv.conf type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run/kata-containers/shared/sandboxes/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2-4b427d5d3a1426cf-hosts type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run/kata-containers/shared/sandboxes/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2-dbc0cf4809dfda42-hostname type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run/kata-containers/shared/sandboxes/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2/7e6b8f9aaf4161685325a0b9930bf763f8bea079a300c352b5ac62968a142ea2-f2959f3c2ac16040-.containerenv type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
@crobinso crobinso added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels Oct 4, 2019
@devimc
Copy link

devimc commented Oct 10, 2019

@crobinso thanks for raising this, I can reproduce it and I'll be working on a patch

@devimc devimc self-assigned this Oct 10, 2019
devimc pushed a commit to devimc/kata-runtime that referenced this issue Oct 10, 2019
virtcontainers: unmount host mounts if container can be created
ed14667 
Mount points, like `resolv.conf` and `hostname` are left in the
host when the cgroup creation fails.
Use `unmountHostMounts()` in the rollback function that is called
when container's creation fails.

fixes kata-containers#2108

Signed-off-by: Julio Montes <[email protected]>
@devimc devimc mentioned this issue Oct 10, 2019
Merged
devimc pushed a commit to devimc/kata-runtime that referenced this issue Oct 10, 2019
virtcontainers: unmount host mounts if container can't be created
d992a16 
Mount points, like `resolv.conf` and `hostname` are left in the
host when the cgroup creation fails.
Use `unmountHostMounts()` in the rollback function that is called
when container's creation fails.

fixes kata-containers#2108

Signed-off-by: Julio Montes <[email protected]>
devimc pushed a commit to devimc/kata-runtime that referenced this issue Oct 11, 2019
virtcontainers: unmount host mounts if container can't be created
eca7bd2 
Mount points, like `resolv.conf` and `hostname` are left in the
host when the cgroup creation fails.
Use `unmountHostMounts()` and `bindUnmountContainerRootfs()` in the rollback
function that is called when container's creation fails.

fixes kata-containers#2108

Signed-off-by: Julio Montes <[email protected]>
@crobinso
Copy link
Author

Thanks! I confirmed the latest version fixes my issues

@jodh-intel
Copy link
Contributor

Hi @crobinso - good to hear! Could you possibly paste the version your are using here and then close this issue?

@crobinso
Copy link
Author

@jodh-intel oops sorry that comment was meant for the PR, I tested the latest commits there. so better to let this autoclose when the PR is pushed

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@devimc devimc

Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

virtcontainers: unmount host mounts if container can't be created devimc/kata-runtime
3 participants
@crobinso @devimc @jodh-intel