Using rootless podman on HPC cluster, where many concurrent jobs on different nodes access same imagestore

[bbimber]

Hello,

We are trying to run rootless podman-based jobs on a high-performance computing cluster. As-is, the cluster is configured where each user has a folder defined as the graphroot (set in storage.conf), where image and container information is stored. Under this pattern, independent podman jobs that run on different nodes are concurrently accessing this one central store of images.

I am seeing lots of errors that I assume are due to multiple jobs butting against each other when trying to interact with this one central image store. This includes podman-run errors like 'Error: beginning transaction: database is locked', or cryptic errors about 'container not found'.

Are there ways to configure the podman image store that work well in a cluster-like environment?

If we can reasonably do so, I would like to share images across jobs (avoiding re-download); however, we do not want containers to persist after a job. I would be perfectly happy using a temporary job-specific folder for these data; however, would prefer not to duplicate the base image layers if we can avoid it.

I am looking at ways to configure runroot (https://docs.podman.io/en/stable/markdown/podman.1.html), or similar options, but it wasnt quite sure what combination of options fits this situation best.

Thank you for any ideas or suggestions.

[optiz0r]

We're seeing similar issues here, even without having a central store shared across multiple hosts. graphRoot is set to the default rootless location of ~/.local/share/containers/storage, and large number of podman jobs are being run on single hosts concurrently and we're seeing a high rate of failures due to Error: beginning transaction: database is locked. So the problem does not relate to an image store on a shared location, just concurrent access, even from podman commands running locally on a single host.

This is on podman 5.2.2 shipped with Rocky Linux 9.5.