Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

Root user can't have write access in Kata android container #1748

Closed
zhiminghufighting opened this issue May 30, 2019 · 15 comments
Closed

Root user can't have write access in Kata android container #1748

zhiminghufighting opened this issue May 30, 2019 · 15 comments

Comments

@zhiminghufighting
Copy link

Description of problem

After launched an android container based on kata, root user can't write/create file even the file owner is root with "drwxr-x--x" access right;

Expected result

root user can own the access right of whom the file owner is root;

Actual result

  1. enter android container:
    31bc2e1d785a:/ # ls
    acct init.environ.rc property_contexts
    android-entry init.houdini.rc root
    bugreports init.logs.rc sbin
    cache init.multidroid.rc sdcard
    charger init.multidroid_cloud.rc seapp_contexts
    config init.multidroid_cloud.usb.rc selinux_version
    d init.rc sepolicy
    data init.usb.configfs.rc service_contexts
    default.prop init.usb.rc storage
    dev init.zygote32.rc sys
    etc init.zygote64_32.rc system
    file_contexts.bin ipc ueventd.multidroid_cloud.rc
    fstab.multidroid_cloud mnt ueventd.rc
    init oem vendor
    init.crashlogd.rc proc
    31bc2e1d785a:/ #

  2. list the own/group owner/other user access right by "ls -l"
    31bc2e1d785a:/ # ls -l
    total 6136
    drwxr-xr-x 1 root root 17 2019-05-29 13:42 acct
    -rwxr-xr-x 1 root root 1266408 2018-11-15 07:24 android-entry
    lrwxrwxrwx 1 root root 50 2018-11-15 07:24 bugreports -> /data/user_de/0/com.android.shell/files/bugreports
    drwxr-xr-x 2 root root 6 2018-11-15 07:24 cache
    lrwxrwxrwx 1 root root 13 2018-11-15 07:24 charger -> /sbin/healthd
    drwxr-xr-x 2 root root 6 2018-11-15 07:24 config
    lrwxrwxrwx 1 root root 17 2018-11-15 07:24 d -> /sys/kernel/debug
    drwxr-x--x 34 root root 4096 2019-05-29 09:13 data
    -rw-r--r-- 1 root root 1160 2018-11-15 07:24 default.prop
    drwxr-xr-x 15 root root 2980 2019-05-29 13:42 dev
    drwxr-xr-x 1 root root 66 2019-05-29 13:42 etc
    -rw-r--r-- 1 root root 77090 2018-11-15 07:24 file_contexts.bin
    -rw-r--r-- 1 root root 113 2018-11-15 07:24 fstab.multidroid_cloud
    -rwxr-xr-x 1 root root 1468488 2018-11-15 07:24 init
    -rw-r--r-- 1 root root 647 2018-11-15 07:24 init.crashlogd.rc
    -rw-r--r-- 1 root root 887 2018-11-15 07:24 init.environ.rc
    -rw-r--r-- 1 root root 541 2018-11-15 07:24 init.houdini.rc
    -rw-r--r-- 1 root root 1557 2018-11-15 07:24 init.logs.rc
    -rwxr-xr-x 1 root root 1506 2018-11-15 07:24 init.multidroid.rc
    -rwxr-xr-x 1 root root 71 2018-11-15 07:24 init.multidroid_cloud.rc
    -rw-r--r-- 1 root root 4106 2018-11-15 07:24 init.multidroid_cloud.usb.rc
    -rwxr-xr-x 1 root root 27158 2018-11-15 07:24 init.rc
    -rw-r--r-- 1 root root 9283 2018-11-15 07:24 init.usb.configfs.rc
    -rwxr-xr-x 1 root root 5715 2018-11-15 07:24 init.usb.rc
    -rw-r--r-- 1 root root 411 2018-11-15 07:24 init.zygote32.rc
    -rw-r--r-- 1 root root 684 2018-11-15 07:24 init.zygote64_32.rc
    drwxr-xr-x 3 root root 20 2019-05-29 09:12 ipc
    drwxr-xr-x 10 root system 220 2019-05-29 13:42 mnt
    drwxr-xr-x 2 root root 6 2018-11-15 07:24 oem
    dr-xr-xr-x 84 root root 0 2019-05-29 13:42 proc
    -rw-r--r-- 1 root root 4745 2018-11-15 07:24 property_contexts
    drwxr-xr-x 2 root root 6 2018-11-15 07:24 root
    drwxr-xr-x 2 root root 65 2018-11-15 07:24 sbin
    lrwxrwxrwx 1 root root 21 2018-11-15 07:24 sdcard -> /storage/self/primary
    -rw-r--r-- 1 root root 758 2018-11-15 07:24 seapp_contexts
    -rw-r--r-- 1 root root 71 2018-11-15 07:24 selinux_version
    -rw-r--r-- 1 root root 178142 2018-11-15 07:24 sepolicy
    -rw-r--r-- 1 root root 11162 2018-11-15 07:24 service_contexts
    drwxr-xr-x 3 root root 60 2019-05-29 13:42 storage
    dr-xr-xr-x 11 root root 0 2019-05-29 13:42 sys
    drwxr-xr-x 17 root root 224 2018-11-15 07:24 system
    -rw-r--r-- 1 root root 272 2018-11-15 07:24 ueventd.multidroid_cloud.rc
    -rwxr-xr-x 1 root root 4853 2018-11-15 07:24 ueventd.rc
    lrwxrwxrwx 1 root root 14 2018-11-15 07:24 vendor -> /system/vendor

  3. try to modify a file and report the error "sh: can't create test.txt: Read-only file system"
    31bc2e1d785a:/data # ls
    adb app-private drm misc_ce security user
    anr backup local misc_de ss user_de
    app bootchart lost+found ota system
    app-asec cache media ota_package system_ce
    app-ephemeral dalvik-cache mediadrm property system_de
    app-lib data misc resource-cache tombstones
    31bc2e1d785a:/data # echo 111 > test.txt
    sh: can't create test.txt: Read-only file system
    1|31bc2e1d785a:/data #
    1|31bc2e1d785a:/data # ls -l
    total 24
    drwx------ 2 root root 6 2019-05-29 09:13 adb
    drwxrwxr-x 2 root root 52 2019-05-29 09:13 anr
    drwxrwx--x 4 root root 63 2019-05-29 09:13 app
    drwx------ 2 root root 6 2019-05-29 09:13 app-asec
    drwxrwx--x 2 root root 6 2019-05-29 09:12 app-ephemeral
    drwxrwx--x 2 root root 6 2019-05-29 09:13 app-lib
    drwxrwx--x 2 root root 6 2019-05-29 09:13 app-private
    drwx------ 3 root root 62 2019-05-29 09:12 backup
    drwxr-xr-x 2 root root 6 2019-05-29 09:13 bootchart
    drwxrwx--- 5 root root 56 2019-05-29 09:13 cache
    drwxrwx--x 4 root root 31 2019-05-29 09:12 dalvik-cache
    drwxr-x--x 59 root root 4096 2019-05-29 09:13 data
    drwxrwx--- 2 root root 6 2019-05-29 09:13 drm
    drwxr-x--x 3 root root 17 2019-05-29 09:12 local
    drwxrwx--- 2 root root 6 2019-05-29 09:12 lost+found
    drwxrwx--- 4 root root 26 2019-05-29 09:13 media
    drwxrwx--- 2 root root 6 2019-05-29 09:13 mediadrm
    drwxrwx--x 30 root root 4096 2019-05-29 09:13 misc
    drwxrwx--x 3 root root 15 2019-05-29 09:13 misc_ce
    drwxrwx--x 3 root root 15 2019-05-29 09:13 misc_de
    drwxrwx--x 2 root root 6 2019-05-29 09:12 ota
    drwxrwx--- 2 root root 6 2019-05-29 09:13 ota_package
    drwx------ 2 root root 241 2019-05-29 09:13 property
    drwxrwx--x 2 root root 6 2019-05-29 09:13 resource-cache
    drwx--x--x 2 root root 6 2019-05-29 09:13 security
    drwx------ 2 root root 6 2019-05-29 09:13 ss
    drwxrwxr-x 14 root root 4096 2019-05-29 09:12 system
    drwxrwx--- 3 root root 15 2019-05-29 09:13 system_ce
    drwxrwx--- 3 root root 15 2019-05-29 09:12 system_de
    drwxrwxr-x 2 root root 206 2019-05-29 09:12 tombstones
    drwx--x--x 2 root root 6 2019-05-29 09:12 user
    drwx--x--x 3 root root 15 2019-05-29 09:12 user_de

By the way, if i run the same android container image based on runc or run other Linux container image based on kata, there is no such access issue.
I am not sure if there is related with 9pfs shared ro or build container rootfs to change it into read-only mode.
I will check and verify this issue by code debugging.

Kata version :1.7.0
@grahamwhaley
Copy link
Contributor

@zhiminghufighting - seeing the output of mount inside the container might give some clues as well. thx.

@devimc
Copy link

devimc commented May 30, 2019

@zhiminghufighting for security reasons, in kata containers the rootfs is mounted as a read-only filesystem, hence you can't modify it. Fortunately you can re-mount it running mount -o remount,rw / /

@zhiminghufighting
Copy link
Author

@grahamwhaley yes, i also think it is related with 'mount' operations.
@devimc i have questions about "kata containers rootfs is mounted as a read-only filesystem:

  1. per my understanding, the 'mounted' actions is implemented by agent in kata guest VM, am i right?
  2. so i need to do the 'mount -o remount,rm / /' in the Kata guest VM shell, am i right?
  3. i see there are different properties for kataShared mount as below:
    A) in a centos container shell:
    Here the kataShared based on 9p is "rw".
    kataShared on / type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)

[root@localhost ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9cb629eda3ce android:NAe000032 "/android-entry -c 0…" 4 hours ago Up 4 hours 0.0.0.0:5555->5555/tcp android0
2eaa35d9fd64 centos "/bin/bash" 4 hours ago Up 4 hours musing_mahavira
93a0d1fb64e0 centos "/bin/bash" 5 hours ago Up 5 hours hopeful_chandrasekhar
5ceff5ce0746 centos "/bin/bash" 24 hours ago Up 24 hours cocky_easley
[root@localhost ~]# docker exec -ti 2eaa sh
sh-4.2#
sh-4.2#
sh-4.2#
sh-4.2# ls
anaconda-post.log dev home lib64 mnt proc run srv tmp var
bin etc lib media opt root sbin sys usr
sh-4.2# mount
kataShared on / type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
kataShared on /etc/resolv.conf type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/hostname type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/hosts type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
sh-4.2#

B) in an android container shell: kataShared on / type 9p (ro,dirsync,relatime,mmap,access=client,trans=virtio)
why it is "ro" instead of "rw" but they both are launched by the same command and kata settings?

[root@localhost ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9cb629eda3ce android:NAe000032 "/android-entry -c 0…" 4 hours ago Up 4 hours 0.0.0.0:5555->5555/tcp android0
2eaa35d9fd64 centos "/bin/bash" 4 hours ago Up 4 hours musing_mahavira
93a0d1fb64e0 centos "/bin/bash" 5 hours ago Up 5 hours hopeful_chandrasekhar
5ceff5ce0746 centos "/bin/bash" 24 hours ago Up 24 hours cocky_easley
[root@localhost ~]# docker exec -ti 9cb6 sh
9cb629eda3ce:/ #
9cb629eda3ce:/ # ls
acct init.environ.rc property_contexts
android-entry init.houdini.rc root
bugreports init.logs.rc sbin
cache init.multidroid.rc sdcard
charger init.multidroid_cloud.rc seapp_contexts
config init.multidroid_cloud.usb.rc selinux_version
d init.rc sepolicy
data init.usb.configfs.rc service_contexts
default.prop init.usb.rc storage
dev init.zygote32.rc sys
etc init.zygote64_32.rc system
file_contexts.bin ipc ueventd.multidroid_cloud.rc
fstab.multidroid_cloud mnt ueventd.rc
init oem vendor
init.crashlogd.rc proc
9cb629eda3ce:/ # mount
kataShared on / type 9p (ro,dirsync,relatime,mmap,access=client,trans=virtio)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
kataShared on /data type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)
kataShared on /ipc type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)
kataShared on /etc/resolv.conf type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)
kataShared on /etc/hostname type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)
kataShared on /etc/hosts type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /mnt type tmpfs (rw,relatime,mode=755,gid=1000)
none on /dev/memcg type cgroup (rw,relatime,memory)
none on /dev/cpuctl type cgroup (rw,relatime,cpu,cpuacct)
none on /dev/cpuset type cgroup (rw,relatime,cpuset)
tmpfs on /storage type tmpfs (rw,relatime,mode=755,gid=1000)

why they have different behavior with the same kata setting and same launching command?

@devimc
Copy link

devimc commented May 30, 2019

@zhiminghufighting

per my understanding, the 'mounted' actions is implemented by agent in kata guest VM, am i right?

no, the guest kernel mounts the rootfs as a ro fs, you can check it in /proc/cmdline

so i need to do the 'mount -o remount,rm / /' in the Kata guest VM shell, am i right?

Yes

B) in an android container shell: kataShared on / type 9p (ro,dirsync,relatime,mmap,access=client,trans=virtio)
why it is "ro" instead of "rw" but they both are launched by the same command and kata settings?

that's weird, are you using a custom kernel?

why they have different behavior with the same kata setting and same launching command?

good question ... 🤔

@zhiminghufighting
Copy link
Author

@devimc yes, i used customized kata guest kernel, but i just change the kernel config for some android depends on modular and never touch kernel cmdline.

@grahamwhaley
Copy link
Contributor

@zhiminghufighting @devimc - you are trying to write to /data, yes - and from the mount output, we see:

kataShared on /data type 9p (ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio)

So, that /data' is an ro9p mount. I guess the question is, how is/data` being requested/passed in to the container - on the command line as a volume etc. etc. ?

@zhiminghufighting
Copy link
Author

@devimc @grahamwhaley
i setup debug console in the way by add "agent.debug" in cmdline and enter the cmd(mount -o remount,rw / )to try to remount kata guest rootfs as "rw" and get errors as below:

[root@localhost multidroid_cloud-aic-NAe000032]#
[root@localhost multidroid_cloud-aic-NAe000032]#
[root@localhost multidroid_cloud-aic-NAe000032]#
[root@localhost multidroid_cloud-aic-NAe000032]# sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"
[ 471.210563] init: couldn't write 2626 to /dev/cpuset/camera-daemon/tasks: No such file or directory
[ 476.315887] init: couldn't write 2655 to /dev/cpuset/camera-daemon/tasks: No such file or directory
[ 481.404290] init: couldn't write 2684 to /dev/cpuset/camera-daemon/tasks: No such file or directory
bin dev home lib64 media opt root sbin sys usr
boot etc lib lost+found mnt proc run srv tmp var
bash-4.2# [ 487.306680] init: cannot execve('/system/bin/mediaserver'): Exec format error
bash-4.2# mount -o remount,rw /
mount: 'kataShared' not user mountable in fstab
1|cloud:/ $ [ 492.385424] init: cannot execve('/system/bin/mediaserver'): Exec format error
[ 506.331266] init: couldn't write 2830 to /dev/cpuset/camera-daemon/tasks: No such file or directory
[ 511.416619] init: couldn't write 2859 to /dev/cpuset/camera-daemon/tasks: No such file or directory

And kata guest filesystem is still read-only and the cmd doesn't work for this issue.

As the comparison , i try the same cmd in the kata guest shell of centos or ubuntu container and they can work.

@zhiminghufighting
Copy link
Author

@devimc @grahamwhaley @jodh-intel
i check all the possibile code in kata-runtime for bindmount & kernel cmd paramter and make sure there is no 'ro' can be set in host side bindmount.
The cmd(mount -o remount,rw /) doesn't work in kata guest VM shell too.

Is there any other ways to enable 'write' right for root user in a container?
Is there possibility that 'ro' property is set by agent in guest VM?

Thanks in advance!

@devimc
Copy link

devimc commented May 31, 2019

@zhiminghufighting

mount -o remount,rw /
mount: 'kataShared' not user mountable in fstab

you forgot the last /, try again mount -o remount,rw / /

@zhiminghufighting
Copy link
Author

@grahamwhaley /data is passed into container by '-v' for some dependent lib of android app.
@devimc thank a lot! I try again the cmd and find new error as below.

bash-4.2# bash-4.2# mount -o remount,rw / /
mount: '/' not in /proc/mounts

127|cloud:/ $ bash-4.2# [95713.216696] init: cannot execve('/system/bin/drmserver'): Exec format error
[95713.218291] init: cannot execve('/system/bin/debuggerd'): Exec format error
bash-4.2# cat /proc/mounts
kataShared / 9p ro,dirsync,relatime,mmap,access=client,trans=virtio 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
kataShared /ipc 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0
kataShared /data 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0
kataShared /etc/resolv.conf 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0
kataShared /etc/hostname 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0
kataShared /etc/hosts 9p ro,dirsync,nodev,relatime,mmap,access=client,trans=virtio 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
tmpfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
tmpfs /mnt tmpfs rw,relatime,mode=755,gid=1000 0 0
none /dev/memcg cgroup rw,relatime,memory 0 0
none /dev/cpuctl cgroup rw,relatime,cpu,cpuacct 0 0
none /dev/cpuset cgroup rw,relatime,cpuset 0 0
tmpfs /storage tmpfs rw,relatime,mode=755,gid=1000 0 0

It is weird, there is no /dev/root / in /proc/mounts file.

@jodh-intel
Copy link
Contributor

@zhiminghufighting - That looks very odd - I think you are seeing a mixture of the Android init system and systemd there. The "exec format error" suggests a mismatch between binaries and the host architecture (either bitness or totally different architectures).

It would really help us to help you if you could explain how we can recreate this issue with publically available images.

@zhiminghufighting
Copy link
Author

zhiminghufighting commented Jun 4, 2019 •
edited
Loading

@jodh-intel yes, it is weird. i agree with your analysis it is high possibility of mixture of android init system and system there.
And i wonder why 9p mount share point is "ro" instead of "rw".

I have this image and related kata kernel image to reproduce such issue. Do you need to access my environment or i copy the android contaienr image + kata guest image to any test bed?
Then you can reproduce it easily.

@jodh-intel
Copy link
Contributor

@zhiminghufighting - I think what would be best would be instructions on this issue explaining how to create the sort of environment you are using:

  • details of custom kernel
  • details of custom image
  • details of docker image you are using.
  • full Kata config (or just output of sudo kata-collect-data.sh, after you have reviewed it to make sure there is no sensitive / secret / personal information included in it).
  • details of the command you are running.

I'm guessing it doesn't have to be exactly the same as the environment you are using (as that might have sensitive / customer information in it?) but I don't think anyone has tried running Kata with Android so the problems you are seeing are probably generic issues.

@zhiminghufighting
Copy link
Author

@jodh-intel, there are many customization for this case.

  1. kata guest kata is reconfigure to enable vga;
  2. kata runtime is reconfigure to enable vga;
  3. kata runtime to patch for floppy device filtering error for some specific host os;
  4. kata guest luanching paramter is changed for GUI display;
  5. i rebuild an andriod container image from a new base and no such "ro" issue in the new image and the suusgestion provided by @devimc run well.
    A. old andriod image + docker runc is OK without "ro" issue;
    B. new based andriod image + docker runc & kata is OK without "ro" issue;
    C. old andriod image + kata is with "ro" issue;
    So i wonder why the same image is Ok in runc but there is "ro" issue in kata?
    And this is the only one image being found such weird issue.
    I dig into it and guess there must be some potential nasty flaw in somewhere related with contianer image build or kata code.

Thanks you all for great support!
i will keep to dig in and need to get more usefull info then to discuss with you guys.

@egernst egernst changed the title Root user can't have right access in Kata android container Root user can't have write access in Kata android container Jun 10, 2019
@zhiminghufighting
Copy link
Author

@devimc @jodh-intel @egernst @grahamwhaley I get help from PRC local android team and find the root cause of this issue: android container will remount rootfs as ro in this container and create some folder and assign there different access right according to different directories for security purpose. That beyond of the coverage of kata code and docker file.
So i will close this issue and thank you all!

zklei pushed a commit to zklei/runtime that referenced this issue Jun 13, 2019
@bergwolf
vendor: update grpc-go dependency
cd03e0c 
This updates grpc-go vendor package to v1.11.3 release, to fix server.Stop()
handling so that server.Serve() does not wait blindly.

Full commit list:
d11072e (tag: v1.11.3) Change version to 1.11.3
d06e756 clientconn: add support for unix network in DialContext. (kata-containers#1883)
452c2a7 Change version to 1.11.3-dev
d89cded (tag: v1.11.2) Change version to 1.11.2
98ac976 server: add grpc.Method function for extracting method from context (kata-containers#1961)
0f5fa28 Change version to 1.11.2-dev
1e2570b (tag: v1.11.1) Change version to 1.11.1
d28faca client: Fix race when using both client-side default CallOptions and per-call CallOptions (kata-containers#1948)
48b7669 Change version to 1.11.1-dev
afc05b9 (tag: v1.11.0) Change version to 1.11.0
f2620c3 resolver: keep full unparsed target string if scheme in parsed target is not registered (kata-containers#1943)
9d2250f status: rename Status to GRPCStatus to avoid name conflicts (kata-containers#1944)
2756956 status: Allow external packages to produce status-compatible errors (kata-containers#1927)
0ff1b76 routeguide: reimplement distance calculation
dfbefc6 service reflection can lookup enum, enum val, oneof, and field symbols (kata-containers#1910)
32d9ffa Documentation: Fix broken link in rpc-errors.md (kata-containers#1935)
d5126f9 Correct Go 1.6 support policy (kata-containers#1934)
5415d18 Add documentation and example of adding details to errors (kata-containers#1915)
57640c0 Allow storing alternate transport.ServerStream implementations in context (kata-containers#1904)
031ee13 Fix Test: Update the deadline since small deadlines are prone to flakes on Travis. (kata-containers#1932)
2249df6 gzip: Add ability to set compression level (kata-containers#1891)
8124abf credentials/alts: Remove the enable_untrusted_alts flag (kata-containers#1931)
b96718f metadata: Fix bug where AppendToOutgoingContext could modify another context's metadata (kata-containers#1930)
738eb6b fix minor typos and remove grpc.Codec related code in TestInterceptorCanAccessCallOptions (kata-containers#1929)
211a7b7 credentials/alts: Update ALTS "New" APIs (kata-containers#1921)
fa28bef client: export types implementing CallOptions for access by interceptors (kata-containers#1902)
ec9275b travis: add Go 1.10 and run vet there instead of 1.9 (kata-containers#1913)
13975c0 stream: split per-attempt data from clientStream (kata-containers#1900)
2c2d834 stats: add BeginTime to stats.End (kata-containers#1907)
3a9e1ba Reset ping strike counter right before sending out data. (kata-containers#1905)
90dca43 resolver: always fall back to default resolver when target does not follow URI scheme (kata-containers#1889)
9aba044 server: Convert all non-status errors to codes.Unknown (kata-containers#1881)
efcc755 credentials/alts: change ALTS protos to match the golden version (kata-containers#1908)
0843fd0 credentials/alts: fix infinite recursion bug [in custom error type] (kata-containers#1906)
207e276 Fix test race: Atomically access minConnecTimout in testing environment. (kata-containers#1897)
3ae2a61 interop: Add use_alts flag to client and server binaries (kata-containers#1896)
5190b06 ALTS: Simplify "New" APIs (kata-containers#1895)
7c5299d Fix flaky test: TestCloseConnectionWhenServerPrefaceNotReceived (kata-containers#1870)
f0a1202 examples: Replace context.Background with context.WithTimeout (kata-containers#1877)
a1de3b2 alts: Change ALTS proto package name (kata-containers#1886)
2e7e633 Add ALTS code (kata-containers#1865)
583a630 Expunge error codes that shouldn't be returned from library (kata-containers#1875)
2759199 Small spelling fixes (unknow -> unknown) (kata-containers#1868)
12da026 clientconn: fix a typo in GetMethodConfig documentation (kata-containers#1867)
dfa1834 Change version to 1.11.0-dev (kata-containers#1863)
46fd263 benchmarks: add flag to benchmain to use bufconn instead of network (kata-containers#1837)
3926816 addrConn: Report underlying connection error in RPC error (kata-containers#1855)
445b728 Fix data race in TestServerGoAwayPendingRPC (kata-containers#1862)
e014063 addrConn: keep retrying even on non-temporary errors (kata-containers#1856)
484b3eb transport: fix race causing flow control discrepancy when sending messages over server limit (kata-containers#1859)
6c48c7f interop test: Expect io.EOF from stream.Send() (kata-containers#1858)
08d6261 metadata: provide AppendToOutgoingContext interface (kata-containers#1794)
d50734d Add status.Convert convenience function (kata-containers#1848)
365770f streams: Stop cleaning up after orphaned streams (kata-containers#1854)
7646b53 transport: support stats.Handler in serverHandlerTransport (kata-containers#1840)
104054a Fix connection drain error message (kata-containers#1844)
d09ec43 Implement unary functionality using streams (kata-containers#1835)
37346e3 Revert "Add WithResolverUserOptions for custom resolver build options" (kata-containers#1839)
424e3e9 Stream: do not cancel ctx created with service config timeout (kata-containers#1838)
f9628db Fix lint error and typo (kata-containers#1843)
0bd008f stats: Fix bug causing trailers-only responses to be reported as headers (kata-containers#1817)
5769e02 transport: remove unnecessary rstReceived (kata-containers#1834)
0848a09 transport: remove redundant check of stream state in Write (kata-containers#1833)
c22018a client: send RST_STREAM on client-side errors to prevent server from blocking (kata-containers#1823)
82e9f61 Use keyed fields for struct initializers (kata-containers#1829)
5ba054b encoding: Introduce new method for registering and choosing codecs (kata-containers#1813)
4f7a2c7 compare atomic and mutex performance in case of contention. (kata-containers#1788)
b71aced transport: Fix a data race when headers are received while the stream is being closed (kata-containers#1814)
46bef23 Write should fail when the stream was done but context wasn't cancelled. (kata-containers#1792)
10598f3 Explain target format in DialContext's documentation (kata-containers#1785)
08b7bd3 gzip: add Name const to avoid typos in usage (kata-containers#1804)
8b02d69 remove .please-update (kata-containers#1800)
1cd2346 Documentation: update broken wire.html link in metadata package. (kata-containers#1791)
6913ad5 Document that all errors from RPCs are status errors (kata-containers#1782)
8a8ac82 update const order (kata-containers#1770)
e975017 Don't set reconnect parameters when the server has already responded. (kata-containers#1779)
7aea499 credentials: return Unavailable instead of Internal for per-RPC creds errors (kata-containers#1776)
c998149 Avoid copying headers/trailers in unary RPCs unless requested by CallOptions (kata-containers#1775)
8246210 Update version to 1.10.0-dev (kata-containers#1777)
17c6e90 compare atomic and mutex performance for incrementing/storing one variable (kata-containers#1757)
65c901e Fix flakey test. (kata-containers#1771)
7f2472b grpclb: Remove duplicate init() (kata-containers#1764)
09fc336 server: fix bug preventing Serve from exiting when Listener is closed (kata-containers#1765)
035eb47 Fix TestGracefulStop flakiness (kata-containers#1767)
2720857 server: fix race between GracefulStop and new incoming connections (kata-containers#1745)
0547980 Notify parent ClientConn to re-resolve in grpclb (kata-containers#1699)
e6549e6 Add dial option to set balancer (kata-containers#1697)
6610f9a Fix test: Data race while resetting global var. (kata-containers#1748)
f4b5237 status: add Code convenience function (kata-containers#1754)
47bddd7 vet: run golint on _string files (kata-containers#1749)
45088c2 examples: fix concurrent map accesses in route_guide server (kata-containers#1752)
4e393e0 grpc: fix deprecation comments to conform to standard (kata-containers#1691)
0b24825 Adjust keepalive paramenters in the test such that scheduling delays don't cause false failures too often. (kata-containers#1730)
f9390a7 fix typo (kata-containers#1746)
6ef45d3 fix stats flaky test (kata-containers#1740)
98b17f2 relocate check for shutdown in ac.tearDown() (kata-containers#1723)
5ff10c3 fix flaky TestPickfirstOneAddressRemoval (kata-containers#1731)
2625f03 bufconn: allow readers to receive data after writers close (kata-containers#1739)
b0e0950 After sending second goaway close conn if idle. (kata-containers#1736)
b8cf13e Make sure all goroutines have ended before restoring global vars. (kata-containers#1732)
4742c42 client: fix race between server response and stream context cancellation (kata-containers#1729)
8fba5fc In gracefull stop close server transport only after flushing status of the last stream. (kata-containers#1734)
d1fc8fa Deflake tests that rely on Stop() then Dial() not reconnecting (kata-containers#1728)
dba60db Switch balancer to grpclb when at least one address is grpclb address (kata-containers#1692)
ca1b23b Update CONTRIBUTING.md to CNCF CLA
2941ee1 codes: Add UnmarshalJSON support to Code type (kata-containers#1720)
ec61302 naming: Fix build constraints for go1.6 and go1.7 (kata-containers#1718)
b8191e5 remove stringer and go generate (kata-containers#1715)
ff1be3f Add WithResolverUserOptions for custom resolver build options (kata-containers#1711)
580defa Fix grpc basics link in route_guide example (kata-containers#1713)
b7dc71e Optimize codes.String() method using a switch instead of a slice of indexes (kata-containers#1712)
1fc873d Disable ccBalancerWrapper when it is closed (kata-containers#1698)
bf35f1b Refactor roundrobin to support custom picker (kata-containers#1707)
4308342 Change parseTimeout to not handle non-second durations (kata-containers#1706)
be07790 make load balancing policy name string case-insensitive (kata-containers#1708)
cd563b8 protoCodec: avoid buffer allocations if proto.Marshaler/Unmarshaler (kata-containers#1689)
61c6740 Add comments to ClientConn/SubConn interfaces to indicate new methods may be added (kata-containers#1680)
ddbb27e client: backoff before reconnecting if an HTTP2 server preface was not received (kata-containers#1648)
a4bf341 use the request context with net/http handler (kata-containers#1696)
c6b4608 transport: fix race sending RPC status that could lead to a panic (kata-containers#1687)
00383af Fix misleading default resolver scheme comments (kata-containers#1703)
a62701e Eliminate data race in ccBalancerWrapper (kata-containers#1688)
1e1a47f Re-resolve target when one connection becomes TransientFailure (kata-containers#1679)
2ef021f New grpclb implementation (kata-containers#1558)
10873b3 Fix panics on balancer and resolver updates (kata-containers#1684)
646f701 Change version to 1.9.0-dev (kata-containers#1682)

Fixes: kata-containers#307

Signed-off-by: Peng Tao <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@grahamwhaley @devimc @jodh-intel @zhiminghufighting