annotations: kata annotations not working anymore

[devimc]

cannot enable a debug console using annotations

I'm using the following yaml, custom kernel parameters are useless

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  annotations:
    io.kata-containers.hypervisor.kernel_params: "agent.debug_console"
spec:
  shareProcessNamespace: true
  runtimeClassName: kata
  containers:
  - name: first-test-container
    image: busybox
    command:
      - sh
    stdin: true
    tty: true

cc @amshinde

[amshinde]

@devimc I hadnt tried with this particular option, but had tested with others.
Should'nt this be agent.debug_console=true ?
Can you try with that and let me know if it still does not work.

[devimc]

@amshinde same issue

$ kubectl attach  -ti pod busybox -c first-test-container
If you don't see a command prompt, try pressing enter.
/ # cat /proc/cmdline 
tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 \
i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off \
cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 \
rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 debug \
systemd.show_status=true systemd.log_level=debug panic=1 nr_cpus=8 agent.use_vsock=false \
systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service \
systemd.mask=systemd-networkd.socket agent.log=debug agent.log=debug

[devimc]

@amshinde I'm facing the same issue with the kernel_modules annotation

[amshinde]

@devimc Are you using containerd or crio?

[devimc]

@amshinde I'm using containerd, going try with crio

[devimc]

@amshinde same issue with CRIO 1.15.0, I updated the annotation to use:

io.katacontainers.hypervisor.kernel_params: "agent.debug_console"

After digging in the logs, I found that the annotation is there but it's not passed to QEMU

Oct 10 17:47:18 ubuntu-fc kata-runtime[24440]: time="2019-10-10T17:47:18.133714273Z" level=debug msg="sending request" arch=amd64 command=create container=03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863 name=grpc.CreateContainerRequest pid=24440 req="container_id:\"03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863\" exec_id:\"03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863\" storages:<driver:\"scsi\" source:\"0:0\" fstype:\"xfs\" options:\"nouuid\" mount_point:\"/run/kata-containers/shared/containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863\" > OCI:<Version:\"1.0.1-dev\" Process:<User:<> Args:\"/pause\" Env:\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\" Env:\"TERM=xterm\" Cwd:\"/\" Capabilities:<Bounding:\"CAP_CHOWN\" Bounding:\"CAP_DAC_OVERRIDE\" Bounding:\"CAP_FSETID\" Bounding:\"CAP_FOWNER\" Bounding:\"CAP_NET_RAW\" Bounding:\"CAP_SETGID\" Bounding:\"CAP_SETUID\" Bounding:\"CAP_SETPCAP\" Bounding:\"CAP_NET_BIND_SERVICE\" Bounding:\"CAP_SYS_CHROOT\" Bounding:\"CAP_KILL\" Effective:\"CAP_CHOWN\" Effective:\"CAP_DAC_OVERRIDE\" Effective:\"CAP_FSETID\" Effective:\"CAP_FOWNER\" Effective:\"CAP_NET_RAW\" Effective:\"CAP_SETGID\" Effective:\"CAP_SETUID\" Effective:\"CAP_SETPCAP\" Effective:\"CAP_NET_BIND_SERVICE\" Effective:\"CAP_SYS_CHROOT\" Effective:\"CAP_KILL\" Inheritable:\"CAP_CHOWN\" Inheritable:\"CAP_DAC_OVERRIDE\" Inheritable:\"CAP_FSETID\" Inheritable:\"CAP_FOWNER\" Inheritable:\"CAP_NET_RAW\" Inheritable:\"CAP_SETGID\" Inheritable:\"CAP_SETUID\" Inheritable:\"CAP_SETPCAP\" Inheritable:\"CAP_NET_BIND_SERVICE\" Inheritable:\"CAP_SYS_CHROOT\" Inheritable:\"CAP_KILL\" Permitted:\"CAP_CHOWN\" Permitted:\"CAP_DAC_OVERRIDE\" Permitted:\"CAP_FSETID\" Permitted:\"CAP_FOWNER\" Permitted:\"CAP_NET_RAW\" Permitted:\"CAP_SETGID\" Permitted:\"CAP_SETUID\" Permitted:\"CAP_SETPCAP\" Permitted:\"CAP_NET_BIND_SERVICE\" Permitted:\"CAP_SYS_CHROOT\" Permitted:\"CAP_KILL\" > OOMScoreAdj:-998 > Root:<Path:\"/run/kata-containers/shared/containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863/rootfs\" Readonly:true > Hostname:\"busybox\" Mounts:<destination:\"/proc\" source:\"proc\" type:\"proc\" options:\"nosuid\" options:\"noexec\" options:\"nodev\" > Mounts:<destination:\"/dev\" source:\"tmpfs\" type:\"tmpfs\" options:\"nosuid\" options:\"strictatime\" options:\"mode=755\" options:\"size=65536k\" > Mounts:<destination:\"/dev/pts\" source:\"devpts\" type:\"devpts\" options:\"nosuid\" options:\"noexec\" options:\"newinstance\" options:\"ptmxmode=0666\" options:\"mode=0620\" options:\"gid=5\" > Mounts:<destination:\"/dev/mqueue\" source:\"mqueue\" type:\"mqueue\" options:\"nosuid\" options:\"noexec\" options:\"nodev\" > Mounts:<destination:\"/sys\" source:\"sysfs\" type:\"sysfs\" options:\"nosuid\" options:\"noexec\" options:\"nodev\" options:\"ro\" > Mounts:<destination:\"/etc/resolv.conf\" source:\"/run/kata-containers/shared/containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863-6fdf983c54bc9098-resolv.conf\" type:\"bind\" options:\"ro\" options:\"bind\" options:\"nodev\" options:\"nosuid\" options:\"noexec\" > Mounts:<destination:\"/dev/shm\" source:\"/run/kata-containers/sandbox/shm\" type:\"bind\" options:\"rbind\" > Mounts:<destination:\"/etc/hostname\" source:\"/run/kata-containers/shared/containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863-2e6ad27dde467e92-hostname\" type:\"bind\" options:\"ro\" options:\"bind\" options:\"nodev\" options:\"nosuid\" options:\"noexec\" > Annotations:<key:\"io.katacontainers.hypervisor.kernel_params\" value:\"agent.debug_console\" > Annotations:<key:\"io.kubernetes.container.name\" value:\"POD\" > Annotations:<key:\"io.kubernetes.cri-o.Annotations\" value:\"{\\\"io.katacontainers.hypervisor.kernel_params\\\":\\\"agent.debug_console\\\",\\\"kubectl.kubernetes.io/last-applied-configuration\\\":\\\"{\\\\\\\"apiVersion\\\\\\\":\\\\\\\"v1\\\\\\\",\\\\\\\"kind\\\\\\\":\\\\\\\"Pod\\\\\\\",\\\\\\\"metadata\\\\\\\":{\\\\\\\"annotations\\\\\\\":{\\\\\\\"io.katacontainers.hypervisor.kernel_params\\\\\\\":\\\\\\\"agent.debug_console\\\\\\\"},\\\\\\\"name\\\\\\\":\\\\\\\"busybox\\\\\\\",\\\\\\\"namespace\\\\\\\":\\\\\\\"default\\\\\\\"},\\\\\\\"spec\\\\\\\":{\\\\\\\"containers\\\\\\\":[{\\\\\\\"command\\\\\\\":[\\\\\\\"sh\\\\\\\"],\\\\\\\"image\\\\\\\":\\\\\\\"busybox\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"c1\\\\\\\",\\\\\\\"stdin\\\\\\\":true,\\\\\\\"tty\\\\\\\":true}],\\\\\\\"runtimeClassName\\\\\\\":\\\\\\\"kata\\\\\\\",\\\\\\\"shareProcessNamespace\\\\\\\":true}}\\\\n\\\",\\\"kubernetes.io/config.seen\\\":\\\"2019-10-10T17:47:14.354030382Z\\\",\\\"kubernetes.io/config.source\\\":\\\"api\\\"}\" > Annotations:<key:\"io.kubernetes.cri-o.CNIResult\" value:\"Interfaces:[{Name:eth0 Mac:ae:ac:af:7e:d4:03 Sandbox:/var/run/netns/k8s_busybox_default_fd065f6c-64f1-4d1d-b204-6d1775203544_0-2dd59cd1}], IP:[{Version:4 Interface:0xc00087dc70 Address:{IP:10.244.0.13 Mask:ffffff00} Gateway:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]}\" > Annotations:<key:\"io.kubernetes.cri-o.CgroupParent\" value:\"/kubepods/besteffort/podfd065f6c-64f1-4d1d-b204-6d1775203544\" > Annotations:<key:\"io.kubernetes.cri-o.ContainerID\" value:\"03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863\" > Annotations:<key:\"io.kubernetes.cri-o.ContainerName\" value:\"k8s_POD_busybox_default_fd065f6c-64f1-4d1d-b204-6d1775203544_0\" > Annotations:<key:\"io.kubernetes.cri-o.ContainerType\" value:\"sandbox\" > Annotations:<key:\"io.kubernetes.cri-o.Created\" value:\"2019-10-10T17:47:14.826042498Z\" > Annotations:<key:\"io.kubernetes.cri-o.HostName\" value:\"busybox\" > Annotations:<key:\"io.kubernetes.cri-o.HostNetwork\" value:\"false\" > Annotations:<key:\"io.kubernetes.cri-o.HostnamePath\" value:\"/var/run/containers/storage/devicemapper-containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863/userdata/hostname\" > Annotations:<key:\"io.kubernetes.cri-o.IP\" value:\"10.244.0.13\" > Annotations:<key:\"io.kubernetes.cri-o.KubeName\" value:\"busybox\" > Annotations:<key:\"io.kubernetes.cri-o.Labels\" value:\"{\\\"io.kubernetes.container.name\\\":\\\"POD\\\",\\\"io.kubernetes.pod.name\\\":\\\"busybox\\\",\\\"io.kubernetes.pod.namespace\\\":\\\"default\\\",\\\"io.kubernetes.pod.uid\\\":\\\"fd065f6c-64f1-4d1d-b204-6d1775203544\\\"}\" > Annotations:<key:\"io.kubernetes.cri-o.LogPath\" value:\"/var/log/pods/default_busybox_fd065f6c-64f1-4d1d-b204-6d1775203544/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863.log\" > Annotations:<key:\"io.kubernetes.cri-o.Metadata\" value:\"{\\\"name\\\":\\\"busybox\\\",\\\"uid\\\":\\\"fd065f6c-64f1-4d1d-b204-6d1775203544\\\",\\\"namespace\\\":\\\"default\\\"}\" > Annotations:<key:\"io.kubernetes.cri-o.MountPoint\" value:\"/var/lib/containers/storage/devicemapper/mnt/47c8ec441f6b8fa203d969ff410159b7e59d380c7abf84e2b630e68454b7e04e/rootfs\" > Annotations:<key:\"io.kubernetes.cri-o.Name\" value:\"k8s_busybox_default_fd065f6c-64f1-4d1d-b204-6d1775203544_0\" > Annotations:<key:\"io.kubernetes.cri-o.Namespace\" value:\"default\" > Annotations:<key:\"io.kubernetes.cri-o.NamespaceOptions\" value:\"{}\" > Annotations:<key:\"io.kubernetes.cri-o.PortMappings\" value:\"[]\" > Annotations:<key:\"io.kubernetes.cri-o.PrivilegedRuntime\" value:\"false\" > Annotations:<key:\"io.kubernetes.cri-o.ResolvPath\" value:\"/var/run/containers/storage/devicemapper-containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863/userdata/resolv.conf\" > Annotations:<key:\"io.kubernetes.cri-o.RuntimeHandler\" value:\"kata\" > Annotations:<key:\"io.kubernetes.cri-o.SandboxID\" value:\"03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863\" > Annotations:<key:\"io.kubernetes.cri-o.SeccompProfilePath\" value:\"\" > Annotations:<key:\"io.kubernetes.cri-o.ShmPath\" value:\"/var/run/containers/storage/devicemapper-containers/03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863/userdata/shm\" > Annotations:<key:\"io.kubernetes.pod.name\" value:\"busybox\" > Annotations:<key:\"io.kubernetes.pod.namespace\" value:\"default\" > Annotations:<key:\"io.kubernetes.pod.uid\" value:\"fd065f6c-64f1-4d1d-b204-6d1775203544\" > Annotations:<key:\"kubectl.kubernetes.io/last-applied-configuration\" value:\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Pod\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"io.katacontainers.hypervisor.kernel_params\\\":\\\"agent.debug_console\\\"},\\\"name\\\":\\\"busybox\\\",\\\"namespace\\\":\\\"default\\\"},\\\"spec\\\":{\\\"containers\\\":[{\\\"command\\\":[\\\"sh\\\"],\\\"image\\\":\\\"busybox\\\",\\\"name\\\":\\\"c1\\\",\\\"stdin\\\":true,\\\"tty\\\":true}],\\\"runtimeClassName\\\":\\\"kata\\\",\\\"shareProcessNamespace\\\":true}}\\n\" > Annotations:<key:\"kubernetes.io/config.seen\" value:\"2019-10-10T17:47:14.354030382Z\" > Annotations:<key:\"kubernetes.io/config.source\" value:\"api\" > Linux:<Resources:<CPU:<Shares:2 > > CgroupsPath:\"/kubepods/besteffort/podfd065f6c-64f1-4d1d-b204-6d1775203544/crio-03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863\" Namespaces:<Type:\"ipc\" > Namespaces:<Type:\"uts\" > Namespaces:<Type:\"mount\" > > > " source=virtcontainers subsystem=kata_agent
Oct 10 17:47:18 ubuntu-fc kata-runtime[24440]: time="2019-10-10T17:47:18.135634047Z" level=debug msg="reading guest console" arch=amd64 command=create container=03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863 name=kata-runtime pid=24440 sandbox=03b14cac9c1c456e447b608f581724703164e7f0f859d22a4ddbe7feb909f863 source=virtcontainers subsystem=kata_agent vmconsole="time=\"2019-10-10T17:47:18.117510674Z\" level=info msg=\"Received add uevent\" debug_console=false name=kata-agent pid=80 source=agent subsystem=udevlistener uevent-action=add uevent-devname=\"bsg/0:0:0:0\" uevent-devpath=\"/devices/pci0000:00/0000:00:04.0/virtio1/host0/target0:0:0/0:0:0:0/bsg/0:0:0:0\" uevent-seqnum=1288 uevent-subsystem=bsg

any thoughts ?

[amshinde]

Looks like there is a missing "config" there. Can you try with

annotations:
    io.katacontainers.config.hypervisor.kernel_params: "agent.debug_console"

[devimc]

@amshinde thanks, that was the issue.
Closing issue since it will be fixed here #2100