Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

kata-runtime cannot kill container #2959

Closed
keloyang opened this issue Sep 16, 2020 · 0 comments · Fixed by #2960
Closed

kata-runtime cannot kill container #2959

keloyang opened this issue Sep 16, 2020 · 0 comments · Fixed by #2960
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.

Comments

@keloyang
Copy link
Contributor

kata-runtime can't kill container
how to reproduce ?
just do an exception test, e.g. kill qemu-system-x86_64.

and then we can' see kata-runtime log,

-- Logs begin at Tue 2020-09-15 17:10:13 CST. --
Sep 16 20:21:43 centos1 kata-runtime[275798]: time="2020-09-16T20:21:43.308832594+08:00" level=info arch=amd64 arguments="\"kill a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 9\"" command=kill commit=911a49585b41828944b8768e789e1c3fbbcbfccf name=kata-runtime pid=275798 source=runtime version=1.12.0-alpha1
Sep 16 20:21:43 centos1 kata-runtime[275798]: time="2020-09-16T20:21:43.308927331+08:00" level=error msg="open /run/vc/sbs/177612d40b7e2b574c5bb0f0beea9353495cad86bb52850e5587ea3198c1bb3c: no such file or directory" arch=amd64 command=kill container=a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 name=kata-runtime pid=275798 source=runtime
Sep 16 20:21:54 centos1 kata-runtime[275861]: time="2020-09-16T20:21:54.244563048+08:00" level=info msg="loaded configuration" arch=amd64 command=kill file=/etc/kata-containers/configuration.toml format=TOML name=kata-runtime pid=275861 source=katautils
Sep 16 20:21:54 centos1 kata-runtime[275861]: time="2020-09-16T20:21:54.244660403+08:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=kill name=kata-runtime pid=275861 source=katautils
Sep 16 20:21:54 centos1 kata-runtime[275861]: time="2020-09-16T20:21:54.244716332+08:00" level=info arch=amd64 arguments="\"kill a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 15\"" command=kill commit=911a49585b41828944b8768e789e1c3fbbcbfccf name=kata-runtime pid=275861 source=runtime version=1.12.0-alpha1
Sep 16 20:21:54 centos1 kata-runtime[275861]: time="2020-09-16T20:21:54.24480912+08:00" level=error msg="open /run/vc/sbs/177612d40b7e2b574c5bb0f0beea9353495cad86bb52850e5587ea3198c1bb3c: no such file or directory" arch=amd64 command=kill container=a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 name=kata-runtime pid=275861 source=runtime
Sep 16 20:21:54 centos1 kata-runtime[275869]: time="2020-09-16T20:21:54.290173354+08:00" level=info msg="loaded configuration" arch=amd64 command=kill file=/etc/kata-containers/configuration.toml format=TOML name=kata-runtime pid=275869 source=katautils
Sep 16 20:21:54 centos1 kata-runtime[275869]: time="2020-09-16T20:21:54.290264233+08:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=kill name=kata-runtime pid=275869 source=katautils
Sep 16 20:21:54 centos1 kata-runtime[275869]: time="2020-09-16T20:21:54.290306643+08:00" level=info arch=amd64 arguments="\"kill a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 9\"" command=kill commit=911a49585b41828944b8768e789e1c3fbbcbfccf name=kata-runtime pid=275869 source=runtime version=1.12.0-alpha1
Sep 16 20:21:54 centos1 kata-runtime[275869]: time="2020-09-16T20:21:54.29040511+08:00" level=error msg="open /run/vc/sbs/177612d40b7e2b574c5bb0f0beea9353495cad86bb52850e5587ea3198c1bb3c: no such file or directory" arch=amd64 command=kill container=a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 name=kata-runtime pid=275869 source=runtime
Sep 16 20:22:05 centos1 kata-runtime[275937]: time="2020-09-16T20:22:05.291439068+08:00" level=info msg="loaded configuration" arch=amd64 command=kill file=/etc/kata-containers/configuration.toml format=TOML name=kata-runtime pid=275937 source=katautils
Sep 16 20:22:05 centos1 kata-runtime[275937]: time="2020-09-16T20:22:05.294147336+08:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=kill name=kata-runtime pid=275937 source=katautils
Sep 16 20:22:05 centos1 kata-runtime[275937]: time="2020-09-16T20:22:05.294352761+08:00" level=info arch=amd64 arguments="\"kill a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 15\"" command=kill commit=911a49585b41828944b8768e789e1c3fbbcbfccf name=kata-runtime pid=275937 source=runtime version=1.12.0-alpha1
Sep 16 20:22:05 centos1 kata-runtime[275937]: time="2020-09-16T20:22:05.296014011+08:00" level=error msg="open /run/vc/sbs/177612d40b7e2b574c5bb0f0beea9353495cad86bb52850e5587ea3198c1bb3c: no such file or directory" arch=amd64 command=kill container=a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 name=kata-runtime pid=275937 source=runtime
Sep 16 20:22:05 centos1 kata-runtime[275945]: time="2020-09-16T20:22:05.394575916+08:00" level=info msg="loaded configuration" arch=amd64 command=kill file=/etc/kata-containers/configuration.toml format=TOML name=kata-runtime pid=275945 source=katautils
Sep 16 20:22:05 centos1 kata-runtime[275945]: time="2020-09-16T20:22:05.394685885+08:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=kill name=kata-runtime pid=275945 source=katautils
Sep 16 20:22:05 centos1 kata-runtime[275945]: time="2020-09-16T20:22:05.394784371+08:00" level=info arch=amd64 arguments="\"kill a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 9\"" command=kill commit=911a49585b41828944b8768e789e1c3fbbcbfccf name=kata-runtime pid=275945 source=runtime version=1.12.0-alpha1
Sep 16 20:22:05 centos1 kata-runtime[275945]: time="2020-09-16T20:22:05.394905086+08:00" level=error msg="open /run/vc/sbs/177612d40b7e2b574c5bb0f0beea9353495cad86bb52850e5587ea3198c1bb3c: no such file or directory" arch=amd64 command=kill container=a6eec310c9a3d23a2eb85713f6b76d5a41020182366bd813de4b4d7b2241cc58 name=kata-runtime pid=275945 source=runtime

kata-containers is exit but kubelet still think kata-container is ok,

[root@centos0 ~]# kubectl get pod;date
NAME        READY   STATUS    RESTARTS   AGE
samplepod   1/1     Running   0          40m
Wed Sep 16 19:20:14 CST 2020
...

[root@centos0 ~]# kubectl get pod;date
NAME        READY   STATUS    RESTARTS   AGE
samplepod   1/1     Running   0          40m
Wed Sep 16 20:29:44 CST 2020
@keloyang keloyang added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels Sep 16, 2020
@keloyang keloyang mentioned this issue Sep 16, 2020
Merged
@jodh-intel jodh-intel changed the title kata-runtime can' kill container kata-runtime cannot kill container Sep 17, 2020
amshinde pushed a commit to amshinde/kata-runtime that referenced this issue Nov 10, 2020
@keloyang @amshinde
runtime: Ignore ENOENT in kill/delete
b71211c 
If sandbox/container's dir is not exist, the kill/delete will
always fail,and this make kubelet/container delete it repeatedly
but fail always. In some kind of abnormal
situation(e.g. kill -9 $pidofqemu),
kata-runtime may kill/delete sandbox first, this make container'dir
not exist, so kata-runtime should skip this error.

Fixes: kata-containers#2959.

Signed-off-by: Shukui Yang <[email protected]>
(cherry picked from commit 120e616)
amshinde pushed a commit to amshinde/kata-runtime that referenced this issue Nov 10, 2020
@keloyang @amshinde
runtime: Ignore ENOENT in kill/delete
915411a 
If sandbox/container's dir is not exist, the kill/delete will
always fail,and this make kubelet/container delete it repeatedly
but fail always. In some kind of abnormal
situation(e.g. kill -9 $pidofqemu),
kata-runtime may kill/delete sandbox first, this make container'dir
not exist, so kata-runtime should skip this error.

Fixes: kata-containers#2959.

Signed-off-by: Shukui Yang <[email protected]>
(cherry picked from commit 120e616)
amshinde pushed a commit to amshinde/kata-runtime that referenced this issue Nov 11, 2020
@keloyang @amshinde
runtime: Ignore ENOENT in kill/delete
951302f 
If sandbox/container's dir is not exist, the kill/delete will
always fail,and this make kubelet/container delete it repeatedly
but fail always. In some kind of abnormal
situation(e.g. kill -9 $pidofqemu),
kata-runtime may kill/delete sandbox first, this make container'dir
not exist, so kata-runtime should skip this error.

Fixes: kata-containers#2959.

Signed-off-by: Shukui Yang <[email protected]>
(cherry picked from commit 120e616)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

runtime: Ignore ENOENT in kill/delete
1 participant
@keloyang