failed to remove paused container

[devimc]

Description of problem

$ docker run --name foo -dti debian bash && docker pause foo && docker rm -f foo
c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1
foo
Error response from daemon: Could not kill running container c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1, cannot remove - Cannot kill container c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1: unknown error after kill: /usr/bin/kata-runtime did not terminate sucessfully: Container c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1 not ready or running, cannot send a signal
: unknown

Expected result

no errors

Actual result

Error response from daemon: Could not kill running container c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1, cannot remove - Cannot kill container c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1: unknown error after kill: /usr/bin/kata-runtime did not terminate sucessfully: Container c6510b66d61f7cfff9a65bab8a08f583086a9fe46d4f4903bbf72b03e33f95d1 not ready or running, cannot send a signal

[sboeuf]

We need to be careful here about how we plan to fix this. I think docker pause invoke kata-runtime pause, which means it actually pause the entire VM. Maybe this is not the way we should handle it and instead we should create a pause/resume command through the agent in order to leverage libcontainer implementation.

Because what happens here is that we're trying to kill the container but the container is not in the right state because the VM is paused. And even if we bypass this, the runtime won't be able to communicate with the agent to actually kill the container because the VM is paused. Seems like pause.go should be implemented differently IMO.

[jshachm]

One case is that, if a sandbox in other word a pod has two service containers besides pause container. And we just want to pause the second container and leaving other container running as normal.Now we simply pause the entire vm. Maybe it's not the right solution. Should we support this case?

@devimc @sboeuf

[devimc]

@jshachm yes, and I think we should do what @sboeuf suggests.
@kata-containers/runtime @egernst @bergwolf @gnawux @jodh-intel @jcvenegas @WeiZhang555 @jingxiaolu what do you think?

[jingxiaolu]

There are three scenarios：

1. pause all apps in one container;
2. pause all apps in whole sandbox;
3. pause entire vm (including qemu)

Currently kata-runtime pause is scenario 3, and @sboeuf and @jshachm is suggesting scenario 1 (correct me if I am wrong). What about scenario 2?

[bergwolf]

pause/resume is a docker use case, so I agree with @sboeuf that we should do it on a per container basis. And we can also optimize to pause the entire vm when pausing the last container of a sandbox.

OTOH, point 3, pause entire vm, is also used internally by vm templating. If we decide to change the virtcontainers API, please keep the internal implementation.

Another thought on the original problem reported in this issue, we know that the vm is paused, so we should be able to just delete it by removing relevant resources. It's just about simulating runc API, we do not have to follow runc internal details.