Errors in running posix filesystem conformance tests

[GabyCT]

While running the posix filesystem conformance tests (https://github.com/pjd/pjdfstest), we are getting the following errors

Test Summary Report
-------------------
./tests/chmod/12.t          (Wstat: 0 Tests: 14 Failed: 6)
  Failed tests:  3-4, 7-8, 11-12
./tests/link/00.t           (Wstat: 0 Tests: 202 Failed: 10)
  Failed tests:  135-136, 142-143, 149-150, 156-157, 163-164
./tests/mkdir/00.t          (Wstat: 0 Tests: 36 Failed: 2)
  Failed tests:  33-34
./tests/mkfifo/00.t         (Wstat: 0 Tests: 36 Failed: 2)
  Failed tests:  33-34
./tests/mknod/00.t          (Wstat: 0 Tests: 36 Failed: 2)
  Failed tests:  33-34
./tests/mknod/11.t          (Wstat: 0 Tests: 28 Failed: 4)
  Failed tests:  12-13, 25-26
./tests/open/00.t           (Wstat: 0 Tests: 47 Failed: 2)
  Failed tests:  33-34
Files=232, Tests=8789, 704 wallclock secs ( 1.69 usr  1.17 sys + 21.37 cusr 276.72 csys = 300.95 CPU)
Result: FAIL

This is the setup that I am using:

# `kata-env`

Output of "`/usr/local/bin/kata-runtime kata-env`":
```toml
[Meta]
  Version = "1.0.18"

[Runtime]
  Debug = true
  Path = "/usr/local/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.3.0"
    Commit = "d00742f43f3fb1177e74d864e47a1bae697c2d73"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = true
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.0_agent_042c3ebd71c.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67-13"
  Parameters = "agent.log=debug"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.0-6ddb006ad3f709cab018af9dc0bf9e756c3ce2cd"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.0-5fbf1f0919ce0bb1f2b7e85692cdf3058023926f"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.15.0-1023-azure"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Ubuntu"
    Version = "16.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz"

[Netmon]
  Version = "kata-netmon version 1.3.0"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = true
  Enable = false

[grahamwhaley]

Thanks for the report @GabyCT - are you going to dig into those to work out why they fail?

[GabyCT]

@grahamwhaley yes, I will take a look

[GabyCT]

While running the conformance tests with runc no failures were found. However, if we run with kata but using devicemapper we only have 2 failures

Test Summary Report
-------------------
./tests/chown/00.t          (Wstat: 0 Tests: 1323 Failed: 0)
  TODO passed:   693, 697, 708-709, 714-715, 729, 733
./tests/symlink/03.t        (Wstat: 0 Tests: 6 Failed: 2)
  Failed tests:  1-2
Files=232, Tests=8789, 181 wallclock secs ( 1.48 usr  0.36 sys + 21.71 cusr 10.53 csys = 34.08 CPU)
Result: FAIL

[grahamwhaley]

Just to be clear then, we are noting that 9pfs has a lot more failures than devicemapper, yes?
(which, is not unexpected I think, and possibly not something we can do too much about ;-) )

[GabyCT]

Yes, overlay2 has 34 failures and devicemapper only 2 failures

[jodh-intel]

Hi @GabyCT - a few questions:

• Could we run these tests as part of the CI?
  • How long do they take to run?
• If we could run them under the CI in a reasonable amount of time, could we ensure that these two tests fail (assert that behaviour) and that the others pass?

[GabyCT]

@jodh-intel , I already submitted a PR kata-containers/tests#815 :)

[jodh-intel]

@GabyCT - thanks!