Pod annotations for selecting a custom hypervisor binary not working

[gkunz]

Problem

We have a use case in which we'd like to select custom components of the the Kata sandbox (kernel, image, hypervisor) using pod annotations. Following the corresponding documentation [1], setting the kernel and image via pod annotations works fine, but the custom hypervisor binary gets ignored and instead the default hypervisor as configured in the config file is used.

Example workload definition:

  apiVersion: v1
  kind: Pod
  metadata:
    name: ubuntu-untrusted-custom-kernel
    annotations:
      # works
      io.katacontainers.config.hypervisor.kernel: "/tmp/kata_custom_sandbox/kernel/vmlinuz.container"
      # works
      io.katacontainers.config.hypervisor.image: "/tmp/kata_custom_sandbox/image/kata-containers.img"
      # gets ignored
      io.katacontainers.config.hypervisor.path: /tmp/kata_custom_sandbox/hypervisor/qemu-virtiofs-system-x86_64
  spec:
    runtimeClassName: kata-qemu
    containers:
      - name: ubuntu
        image: ubuntu
        command: ['bash', '-c', 'sleep infinity']

I am running Kata 1.12-alpha as deployed (some time ago) by kata-deploy.

Expected result

The sandbox should be run by the custom hypervisor specified by the pod annotation.

Actual result

The sandbox is being run by the default hypervisor according to the runtime class definition.

Further information

Looking at the code, it seems that the custom "hypervisor asset" is not considered here [2] and here [3] and so the pod annotations are not passed down to sandbox. Those links point to the Kata 1.x repos, but it seems the same situation is true in the 2.0 repo.

[1] https://github.com/kata-containers/documentation/blob/master/how-to/how-to-set-sandbox-config-kata.md
[2] https://github.com/kata-containers/runtime/blob/master/virtcontainers/pkg/oci/utils.go#L353
[3] https://github.com/kata-containers/runtime/blob/master/virtcontainers/sandbox.go#L441

[jodh-intel]

Thanks for raising @gkunz. I can confirm this behaviour.

[c3d]

I had noticed the discrepancy while working on #3005. I should have asked, I assumed there was something intentional in the list given in addAssetAnnotations.