create docker ramdisk test

[jodh-intel]

Once kata-containers/runtime#412 lands, we need a test that checks to ensure a container can be run in a ramdisk.

[GabyCT]

@jodh-intel , I can create a test where I set the environment variable of DOCKER_RAMDISK=true and then run a container, do you think this is the best approach?

[jodh-intel]

Sounds fine to me. Maybe we could get some input from @wwq19920405 on the sorts of test that we could run here?

We need to make sure we unset the DOCKER_RAMDISK variable after the ramdisk-specific test(s) are complete of course.

[wwq2333]

background

In our production environment, the OS is running on the memory. We must start dockerd with the DOCKER_RAMDISK env. Otherwise, there will be a mistake as follow.

#docker run --rm --runtime runc busybox uname -a
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:107: jailing process inside rootfs caused \\\"pivot_root invalid argument\\\"\"": unknown.

According to the annotation in docker, they will use chroot instead of pivot_root.

// NoPivotRoot will use MS_MOVE and a chroot to jail the process into the container's rootfs
// This is a common option when the container is running in ramdisk

The kata-runtime is based on qemu, i'm not sure the difference between chroot and pivot_root in kata-runtime. During the Hackthon in Beijing, i consulted @WeiZhang555 and @gnawux about this question. The kata-runtime don't need to think about chroot or pivot_root, so this PR( kata-containers/runtime#423 ) just add the no-pivot flag without more logic.

recent situation

Due to some limits, we still use runc in the production environment, and there is no more detailed test data about kata-runtime. Can you (@jodh-intel) provide some suggestions about the test, i would like to test and give a feedback.

In the latest version, we can use the runc and kata-runtime together with DOCKER_RAMDISK.

# docker run --rm --runtime kata-runtime busybox uname -a
Linux a0e8d4a5e8da 4.14.51-132.container #1 SMP Tue Jul 3 17:15:59 UTC 2018 x86_64 GNU/Linux

# docker run --rm --runtime runc busybox uname -a
Linux cc85fe17ee34 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 GNU/Linux

Without DOCKER_RAMDISK.

# docker run --rm --runtime kata-runtime busybox uname -a
Linux 08a571f58aaf 4.14.51-132.container #1 SMP Tue Jul 3 17:15:59 UTC 2018 x86_64 GNU/Linux

# docker run --rm --runtime runc busybox uname -a
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:107: jailing process inside rootfs caused \\\"pivot_root invalid argument\\\"\"": unknown.

[jodh-intel]

Hi @wwq19920405 - sorry, I'm confused. It doesn't appear that the docker ramdisk feature is well documented (although maybe I've just been unlucky finding info on it). Can you explain how you setup docker for use with a ramdisk? I read something that suggested you run dockerd -g, but that doesn't appear to be valid (any more?)

I retract my "sounds fine to me" comment in #429 (comment) since if we just create a test with the DOCKER_RAMDISK variable set, we don't know if we're testing the environment correctly (as I noted on #446).

For a good test, we need to ensure our test environment is setup correctly to run docker using a ramdisk (just for this test). But we also need to check that a ramdisk is being used somehow.

[jodh-intel]

Ping @wwq19920405.

[grahamwhaley]

@wwq19920405 @jodh-intel @GabyCT
Reading this PR and kata-containers/runtime#412, it seems to me that the only change to Kata, when DOCKER_RAMDISK is set, is that we now benignly accept the pivot-root argument to the runtime. Thus, I think/suspect:

• for Kata, there is nothing to be done with --pivot-root
• and thus for this test, the only thing we are checking is that setting DOCKER_RAMDISK now runs (and the kata-runtime accepts the extra argument, and does not error out).

I don't think setting DOCKER_RAMDISK actually has any effect, or is expected or can have any effect, on how kata-runtime actually executes the container.

Can you confirm pls @wwq19920405

Maybe we can separate this test - 'does kata-runtime accept --no-pivot as set by DOCKER_RAMDISK, vs a new test maybe that is 'can kata-runtime run a container when a ramdisk is used on the host' ?

[jodh-intel]

I agree that for our runtime, it's a "nop". But that makes the test arguably useless - we won't know if it ever breaks.

I think a "fuller" test would be to test the four scenarios @wwq19920405 lists in #429 (comment) so that along with the kata-runtime nop test we explicitly test to ensure the runc behaviour running with / without a ramdisk with DOCKER_RAMDISK set and unset. It seems a bit crazy doing this, but that's the only way we'll detect if docker ever remove support, or change the behaviour of the DOCKER_RAMDISK variable I think (which will notify us that we might need to change our "nop" test into something else)?

Alternatively, we could just land #446 and create a separate PR for the runc-specific tests so we get alerted to a change in docker behaviour.

[grahamwhaley]

I don't think the test is totally useless - it is checking that our runtime benignly supports --no-pivot. For that, I think we should go ahead and merge #446 (well, it needs another ack first ;-) ).
I agree with you that, if we want a test that detects if docker ever drops DOCKER_RAMDISK, then we would have to add a runc test here. It is a bit bizarre and backwards, and tbh, if they drop it then our runtime is not going to fail - worst case is we end up benignly supporting an argument that does not exist.

I'll open a new issue, and let's try and land #446.

[jodh-intel]

Yes, it's not totally useless but I'd be concerned that we don't know what it is actually giving us if docker changes. A thought occurs... rather than actually running the docker tests, we could just add the following to our CI to supplement what is already on #446.

$ url=github.com/moby/moby
$ go get -d $url
$ cd $GOPATH/src/$url
$ grep -q 'os.Getenv("DOCKER_RAMDISK")' libcontainerd/client_daemon.go || die "ramdisk code changed"

Yes, it's totally fragile, but that's kind of the point - we want to know when docker change this code.

[grahamwhaley]

We do have a bot type thing (internal to Intel right now?) that checks for a number of things, such as k8s, OCI and runc version and feature updates I think. This feels like something that belongs more there than in the CI to me - it does not need to run on every CI run, but on a reasonably short periodic cycle (daily would be nice).
@chavafg - what do you think to moving the polling check code into Jenkins (or Zuul), and have that on a timed build (which I'm presuming we can do in Jenkins and maybe Zuul). That way we can have all that in the public eye.
The only tricky bit I see is how we get it to feed the info back to us when something changes. Ideally an email to the public mailing list I think - but for that we need it to have access to an MTA. I wonder if there is one available on vexxhost or the OSF we can use for that? @cboylan @mnaser @annabellebertooch

Let's discuss, and then open an Issue if we see this as a good and viable path.

[wwq2333]

hi @jodh-intel, i found DOCKER_RAMDISK in environment-variables.
And this env is worked with dockerd, DOCKER_RAMDISK=true dockerd. Don't need other special args.This env is ineffective with docker-ce 17.x (I forget the versions), and 18.02+ is effective.

You can add a mock-runtime in /etc/docker/daemon.json, just like this

#!/bin/bash

echo $* >> /root/test-runtime.log

And it's easy to find --no-pivot option :

--root /var/run/docker/runtime-test-runtime/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a8fd96986883f0b20c69679f67f03650f1c494184a955b3cc39337444d770aeb/log.json --log-format json create --bundle /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a8fd96986883f0b20c69679f67f03650f1c494184a955b3cc39337444d770aeb --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a8fd96986883f0b20c69679f67f03650f1c494184a955b3cc39337444d770aeb/init.pid a8fd96986883f0b20c69679f67f03650f1c494184a955b3cc39337444d770aeb
--root /var/run/docker/runtime-test-runtime/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ca236dfb90263dbc5f9ca8e159051f9942187dcbb6835b345d235d1980eed2f9/log.json --log-format json create --bundle /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ca236dfb90263dbc5f9ca8e159051f9942187dcbb6835b345d235d1980eed2f9 --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ca236dfb90263dbc5f9ca8e159051f9942187dcbb6835b345d235d1980eed2f9/init.pid --no-pivot ca236dfb90263dbc5f9ca8e159051f9942187dcbb6835b345d235d1980eed2f9

[jodh-intel]

Thanks @wwq19920405.

@GabyCT - could you take a look at this?