Unable to get rootless Podman to work with NFS storage. Is fuse-overlayfs the solution?

[Deliveranc3]

I've been pulling my hair out for days on this. I have a Synology NAS with an NFS share that I want to use as storage for rootless Podman containers on a Fedora VM.

I've mounted the NFS share on the Fedora VM and created a directory owned by my rootless user. This is all fine, except when I run a container that tries to chmod files to a uid within its user namespace, the NFS server denies the operation which causes the container to fail. I'm using a bind volume mount for the container.

I came across this article which includes the following:

One prominent feature we have added is support for storing file security attributes in extended file attributes (xattrs). We need this to support NFS home directories. NFS servers block the use of containers with more than one UID within the user namespace. ... When a process running in this mode creates a file with a different UID, fuse-overlay intercepts the UID creation, creates the file with the mounters UID, and stores the different UID in an xattr.

This sounds like exactly what I need, so I created ~/.config/containers/storage.conf and changed mount_program to /usr/bin/fuse-overlayfs. I did a podman system reset then verified that overlay.mount_program is set to fuse-overlayfs in podman info. I don't see any difference in behavior, however. I still get permission denied with bind mounts. I tried to create and mount a podman volume as described here, but I still get permission denied. I don't see any indication that podman is actually using fuse. Am I doing something wrong?

I'm on Fedora 38, kernel 6.7.4-100.fc38.x86_64, and podman 4.8.3

[Deliveranc3]

Update: I've had some mixed success using a combination of:

• --userns=keep-id
• ignore_chmod_errors = "true" in storage.conf
• Setting environment variables on containers to run as "root" in the container (i.e. my rootless user)

I wouldn't call this solved or ideal though. It isn't very clean and I'm sure not all containers will work like this

[eriksjolund]

I did a test on Fedora CoreOS with

• fuse-overlayfs with options xattr_permissions=2,threaded=1
• podman volume create with --driver=image and an empty container image

It seems to work.

(In the example I didn't use NFS but maybe the same method could be used if
your NFS server supports XATTR).

Example

1. sudo useradd test
2. sudo machinectl shell --uid=test
3. mkdir -p ~/.config/containers
4. Create the file ~/.config/containers/storage.conf with the contents

   [storage]
   driver="overlay"
   [storage.options]
   mount_program = "/usr/bin/fuse-overlayfs"
   [storage.options.overlay]
   mountopt="xattr_permissions=2,threaded=1"
   

5. mkdir ~/src
6. Create the file ~/src/Containerfile with the contents

   FROM scratch
   

7. Build the container image

   podman build -t empty ~/src/
   

8. Create a volume

   podman volume create vol1 --driver=image  --opt image=localhost/empty
   

9. Create files in the volume vol1

    podman run --rm -v vol1:/vol docker.io/library/alpine \
       sh -c "touch /vol/myfile1 ; touch /vol/myfile2 ; chown 1:1 /vol/myfile2"
   

10. List files in the volume vol1. Run the command

    podman run --rm -v vol1:/vol docker.io/library/alpine \
        ls -l /vol1
    

    The following output is printed

    total 0
    -rw-r--r--    1 root     root             0 Mar  10 11:25 myfile1
    -rw-r--r--    1 bin      bin              0 Mar  10 11:25 myfile2
    

11. List files with numeric UID and GID in the volume vol1. Run the command

    podman run --rm -v vol1:/vol docker.io/library/alpine \
        ls -ln /vol1
    

    The following output is printed

    total 0
    -rw-r--r--    1 0         0                  0 Mar  10 11:25 myfile1
    -rw-r--r--    1 1         1                  0 Mar  10 11:25 myfile2
    

12. Check XATTR of the files myfile1 and myfile2 in the file system on the host

myfile1:

test@localhost:~$ find ~ | grep myfile1
/var/home/test/.local/share/containers/storage/overlay/f1e93eccfd0179e429b82dd4f6e1a0cb552fc7480cd7614eeafbcc1e1bffafdd/diff/myfile1
test@localhost:~$ getfattr -d -- $(find ~ | grep myfile1)
getfattr: Removing leading '/' from absolute path names
# file: var/home/test/.local/share/containers/storage/overlay/f1e93eccfd0179e429b82dd4f6e1a0cb552fc7480cd7614eeafbcc1e1bffafdd/diff/myfile1
user.fuseoverlayfs.override_stat="0:0:100755"
test@localhost:~$ ls -l $(find ~ | grep myfile1)
-rwxr-xr-x. 1 test test 0 Mar 10 11:25 /var/home/test/.local/share/containers/storage/overlay/f1e93eccfd0179e429b82dd4f6e1a0cb552fc7480cd7614eeafbcc1e1bffafdd/diff/myfile1

myfile2:

test@localhost:~$ find ~ | grep myfile2
/var/home/test/.local/share/containers/storage/overlay/f1e93eccfd0179e429b82dd4f6e1a0cb552fc7480cd7614eeafbcc1e1bffafdd/diff/myfile2
test@localhost:~$ getfattr -d -- $(find ~ | grep myfile2)
getfattr: Removing leading '/' from absolute path names
# file: var/home/test/.local/share/containers/storage/overlay/f1e93eccfd0179e429b82dd4f6e1a0cb552fc7480cd7614eeafbcc1e1bffafdd/diff/myfile2
user.fuseoverlayfs.override_stat="1:1:100755"
test@localhost:~$ ls -l $(find ~ | grep myfile2)
-rwxr-xr-x. 1 test test 0 Mar 10 11:25 /var/home/test/.local/share/containers/storage/overlay/f1e93eccfd0179e429b82dd4f6e1a0cb552fc7480cd7614eeafbcc1e1bffafdd/diff/myfile1

Result: It seems to work. The files myfile1 and myfile2 both have the same ownership as the regular user on the host when looking directly in the host file system. The XATTR user.fuseoverlayfs.override_stat="0:0:100755" has been set by fuse-overlayfs to store the ownership 0:0 for myfile1. The XATTR user.fuseoverlayfs.override_stat="1:1:100755" has been set by fuse-overlayfs to store the ownership 1:1 for myfile2.

[eriksjolund]

Seeing this line

fuse-overlayfs: read xattr `user.fuseoverlayfs.override_stat` from upperdir: Operation not supported

I would guess that your NFS mount does not support xattr.
Have you tried setting xattr directly on a file (without any use of podman)?

https://man7.org/linux/man-pages/man1/setfattr.1.html

I did a google search for linux,nfs,xattr and found this web page
https://doc.owncloud.com/ocis/next/prerequisites/xattrs.html

It seems Linux kernel version 5.9 or later is required on both the client and the server.
Also running mount.nfs -V on the client should show a version 2.6.1 or later.
That web page also states that NFSv4 is needed.

[eriksjolund]

A side note: If you could use a bind mount instead, then you would not need to use fuse-overlayfs. Sometimes you need to add the option --userns keep-id:uid=$uid,gid=$gid which maps a container user to your regular user on the host.

I usually try out a few alternatives:

• --userns keep-id:uid=$uid,gid=$gid
• --userns keep-id:uid=$uid,gid=$gid --user ${uid}:${gid}
• --userns keep-id:uid=$uid,gid=$gid --user 0:0

[Deliveranc3]

Bind mounting would actually be preferable for my use case. I only wound up going down the named volumes path in order to try and work with fuse-overlayfs but it's looking like that may not be the route I need to go. I did try bind mounting previously with --userns keep-id but not passing it the uid/gid explicitly like that. I'll give that a shot later.

[Deliveranc3]

Unfortunately similar results with --userns keep-id and --user. With some containers, this seems to work fine, but not for others (i.e. the official Nextcloud container image). I suspect this is because some containers switch user or launch additional processes under separate UIDs as part of their entrypoint scripts. I'm sure I could dig into it and make a custom image which removes that logic but it certainly doesn't seem worth the effort. Unfortunately at this point I don't think it's going to be possible to make NFS work universally and smoothly - it seems like block storage with a proper filesystem is my best bet.

[eriksjolund]

In November 2023 I tried to set up Quadlets for running Nextcloud with rootless Podman and have all written files in
the bind-mounts be owned by the regular user on the host:

https://github.com/eriksjolund/nextcloud-podman

keep-id option was used at these lines

https://github.com/eriksjolund/nextcloud-podman/blob/f98db004ca299ddfa01e88343dea25f7cf37ffe1/nextcloud.container#L15
https://github.com/eriksjolund/nextcloud-podman/blob/main/mariadb.container#L16
https://github.com/eriksjolund/nextcloud-podman/blob/f98db004ca299ddfa01e88343dea25f7cf37ffe1/redis.container#L11

I just tried this out quickly back then. I logged in once in the Nextcloud web interface and afterwards I verified that the files in the bind-mount directories had the correct ownership (regular user on the host). More testing is needed before it's possible to say that it's working.

[savely-krasovsky]

My NFS share support xattrs, but I still struggle to use mounted share. Permission denied, Operation not permitted, etc.

[savely-krasovsky]

@rhatdan thanks for replying! I tried to ask question in Podman matrix chat, but due to Matrix bugs I don't think anyone see my message (at least I don't see anyone else messages), so I will copy my long message here:

Can someone clarify whether it is possible to use an already mounted NFS-backed directory as a volume in rootless Podman? I found very little information about this—most discussions focus on NFS home directories (not sure who even uses those, maybe some corporate setups?), but not on simply mounting an NFS directory.

From what I understand, I need fuse-overlayfs, BUT it should already be the default in rootless mode. Also, my NFS probably needs to support xattrs (which it does—I checked multiple times using getfattr/setfattr) and pass mountopt = "xattr_permissions=2", but I'm not sure about the last part.

If I manually run sudo chown -R 525287:525287 app_data, the user inside the container correctly maps UID 525287 to UID 1000. However, I don't understand why it can't create files with those UID:GID automatically. I also don't quite understand how to handle containers that use chown. I assume they lack the necessary permissions, since changing ownership from, say, 525287 to 525288 requires root privileges, but the container runs as a rootless user and therefore can't do it. Does this mean it's only possible in rootful mode in such cases?

Finally, I don't see fuse-overlayfs working at all. I expect it to set xattrs so that files remain owned by 1000:1000 while storing UID:GID mappings in xattrs. However, I don't see any xattrs at all. I even tried setting graphroot to NFS, and only then I did Podman detected NFS and started setting xattrs—but only for its own files in graphroot, not the mounted volume. What am I missing?

Regarding SELinux: it's already set to permissive.

[rhatdan]

The chown will not be allowed, because it is inforced on the NFS Server side. So a rootless container would never be allowed to chown a file to a different UID. As rootful it might work.

Not being able to create a file in a certain directory, are you talking about a world writeable directory from the NFS side of a directory on the NFS that is owned by 525287:525287, the latter should be writable by the user in the container. You can play around with this outside of a the container BTW via podman unshare and then CD into the NFS directory as the particular user.

[savely-krasovsky]

@rhatdan I have a simple demonstration of my problem:

$ podman unshare chown 1000:1000 /home/core/test/ # podman changes rights to 100999:100999 successfully (underlying volume uses XFS).
$ podman unshare chown 1000:1000 /mnt/docker/app_data # podman cannot change rights to 100999:100999 (underlying volume uses NFS).
chown: changing ownership of '/mnt/docker/app_data': Operation not permitted

Basically I want my containers able to create files on NFS share using their internal UID=1000 mapped into host UID=100999. It works for any block FS, but doesn't for NFS. I guess it's kernel limitation? I don't have any squashing at NFS server, it should accept any UID/GID IMO.

[rhatdan]

Yes from the servers point of view, it sees the real users UID say 1000 attempting to chown a file from 1000 to 100999 or something like that. The NFS server does not understand user namespace so this looks like a dangerous thing to do, so it blocks it. With XFS you are dealling with the local kernel, and it understands the user namespace, so it checks to make sure that the process in the user namespace has CAP_DAC_OVERRIDE for the user namespace and the UID it is chowning the file to a UID within the user namespace.

[savely-krasovsky]

@rhatdan thanks, finally got a proof of what is really going on. Are there any workarounds for cases there container uses multiple users and chown? I understand that proper solution is just not using chown so I could probably run it myself once for example (or just map container uid=1000 into host uid=1000), but I don't think it's possible in my case, it's a proprietary image.