Unable to authenticate in compose with redirected registry #25410

cheewee-cartology · 2025-02-27T03:53:09Z

Issue Description

Compose is unable to authenticate with a private remote mirror when docker.io redirected to it. Using podman cli directly works perfectly. Compose without the indirection works too.

Steps to reproduce the issue

Create registry.conf to redirect docker.io

cat /etc/containers/registries.conf.d/002-proxy.conf 
[[registry]]
prefix = "docker.io"
insecure = false
location = "us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror"

Create compose file

$ cat compose.yml 
services:
  works:
    image: us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69
    command: echo "Hello, World!"
  failed:
    image: docker.io/rclone/rclone:1.69
      #image: us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/
    command: echo "Hello, World!"

Authenticate with podman login as necessary
Symlink auth.json as necessary
Start podman service and set docker host as necessary
Pull the images

docker-compose pull failed
docker-compose pull works

Describe the results you received

Compose pull the mirror url works. Compose pull indirected fails. Podman pull indirected works.

$ docker-compose pull failed
[+] Pulling 1/1
 ✘ failed Error {"message":"Requesting bearer token: invalid status code from registry 403 (Forbidden)"}                                                              4.3s 
Error response from daemon: {"message":"Requesting bearer token: invalid status code from registry 403 (Forbidden)"}
$ docker-compose pull works
[+] Pulling 7/7
 ✔ works Pulled                                                                                                                                                       9.8s 
   ✔ f27ed7a90d4a Download complete                                                                                                                                   0.0s 
   ✔ 06487e735056 Download complete                                                                                                                                   0.0s 
   ✔ 1f3e46996e29 Download complete                                                                                                                                   0.8s 
   ✔ 3ba39bb8e169 Download complete                                                                                                                                   0.5s 
   ✔ d12573aef0cf Download complete                                                                                                                                   2.0s 
   ✔ c461d84251fb Download complete 
$ podman pull docker.io/rclone/rclone:1.69
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull docker.io/rclone/rclone:1.69) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver                           
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
INFO[0000] [graphdriver] using prior storage driver: overlay 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image docker.io/rclone/rclone:1.69 (policy: always) 
DEBU[0000] Looking up image "docker.io/rclone/rclone:1.69" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/rclone/rclone:1.69" ...    
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/rclone/rclone:1.69" ...    
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/rclone/rclone:1.69" ...    
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-proxy.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/rclone/rclone:1.69 for docker.io/rclone/rclone:1.69 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69" 
Trying to pull docker.io/rclone/rclone:1.69...
DEBU[0000] Copying source image //rclone/rclone:1.69 to destination image [overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69" 
DEBU[0000] Found credentials for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone in credential helper containers-auth.json in file /run/containers/0/auth.json 
DEBU[0000]  No signature storage configuration found for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69, using built-in default file:///var/lib/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/us-docker.pkg.dev 
DEBU[0000] GET https://us-docker.pkg.dev/v2/            
DEBU[0001] Ping https://us-docker.pkg.dev/v2/ status 401 
DEBU[0001] GET https://us-docker.pkg.dev/v2/token?account=oauth2accesstoken&scope=repository%3Acart-repos-shared-15f6%2Fcart-docker-mirror%2Frclone%2Frclone%3Apull 
DEBU[0003] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/manifests/1.69 
DEBU[0003] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[0003] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0003] Source is a manifest list; copying (only) instance sha256:7fe21a46326000f54b0321949d34a58d7503cac1cffcc4b4e284513cfb76f1de for current system 
DEBU[0003] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/manifests/sha256:7fe21a46326000f54b0321949d34a58d7503cac1cffcc4b4e284513cfb76f1de 
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0004] IsRunningImageAllowed for image docker:docker.io/rclone/rclone:1.69 
DEBU[0004]  Using default policy section                
DEBU[0004]  Requirement 0: allowed                      
DEBU[0004] Overall: allowed                             
DEBU[0004] Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5 
DEBU[0004] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5 
Getting image source signatures
DEBU[0005] Reading /var/lib/containers/sigstore/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone@sha256=7fe21a46326000f54b0321949d34a58d7503cac1cffcc4b4e284513cfb76f1de/signature-1 
DEBU[0005] Not looking for sigstore attachments: disabled by configuration 
DEBU[0005] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0005] ... will first try using the original manifest unmodified 
DEBU[0005] Checking if we can reuse blob sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0005] Checking if we can reuse blob sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0005] Checking if we can reuse blob sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0005] Checking if we can reuse blob sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0005] Checking if we can reuse blob sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0005] Failed to retrieve partial blob: convert_images not configured 
DEBU[0005] Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b 
DEBU[0005] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b 
DEBU[0005] Failed to retrieve partial blob: convert_images not configured 
DEBU[0005] Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0 
DEBU[0005] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0 
DEBU[0005] Failed to retrieve partial blob: convert_images not configured 
DEBU[0005] Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22 
DEBU[0005] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22 
DEBU[0005] Failed to retrieve partial blob: convert_images not configured 
DEBU[0005] Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e 
DEBU[0005] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e 
DEBU[0005] Failed to retrieve partial blob: convert_images not configured 
DEBU[0005] Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09 
DEBU[0005] GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09 
Copying blob f27ed7a90d4a [--------------------------------------] 0.0b / 94.0b | 0.0 b/s
Copying blob 1f3e46996e29 [--------------------------------------] 0.0b / 3.5MiB | 0.0 b/s
Copying blob f27ed7a90d4a done   | 
Copying blob 1f3e46996e29 [--------------------------------------] 0.0b / 3.5MiB | 0.0 b/s
Copying blob f27ed7a90d4a done   | 
Copying blob 1f3e46996e29 [--------------------------------------] 0.0b / 3.5MiB | 0.0 b/s
Copying blob d12573aef0cf [--------------------------------------] 0.0b / 20.8MiB | 0.0 b/s
Copying blob 3ba39bb8e169 [--------------------------------------] 0.0b / 532.2KiB | 0.0 b/s
Copying blob f27ed7a90d4a done   | 
Copying blob 1f3e46996e29 [--------------------------------------] 26.0KiB / 3.5MiB | 735.2 KiB/s
Copying blob f27ed7a90d4a done   | 
Copying blob 1f3e46996e29 done   | 
Copying blob d12573aef0cf [>-------------------------------------] 656.5KiB / 20.8MiB | 7.6 MiB/s
Copying blob f27ed7a90d4a done   | 
Copying blob f27ed7a90d4a done   | 
Copying blob f27ed7a90d4a done   | 
Copying blob 1f3e46996e29 done   | 
Copying blob d12573aef0cf done   | 
Copying blob 3ba39bb8e169 done   | 
Copying blob 06487e735056 done   | 
DEBU[0010] Applying tar in /var/lib/containers/storage/overlay/cb06561ad00a0b1fce65813322a374f7d2a7a2a0ed7cbea48ad81aa212a5d377/diff 
DEBU[0010] No compression detected                      
DEBU[0010] Compression change for blob sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5 ("application/vnd.oci.image.config.v1+json") not supported 
DEBU[0010] Using original blob without modification     
Copying config c461d84251 done   | 
Writing manifest to image destination
DEBU[0010] setting image creation date to 2025-02-14 15:19:30.479365755 +0000 UTC 
DEBU[0010] created new image ID "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" with metadata "{}" 
DEBU[0010] added name "docker.io/rclone/rclone:1.69" to image "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" 
DEBU[0010] Pulled candidate docker.io/rclone/rclone:1.69 successfully 
DEBU[0010] Looking up image "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" in local containers storage 
DEBU[0010] Trying "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" ... 
DEBU[0010] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" 
DEBU[0010] Found image "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" as "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" in local containers storage 
DEBU[0010] Found image "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" as "c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5) 
DEBU[0010] exporting opaque data as blob "sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5" 
c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5
DEBU[0010] Called pull.PersistentPostRunE(podman --log-level debug pull docker.io/rclone/rclone:1.69) 
DEBU[0010] Shutting down engines                        
INFO[0010] Received shutdown.Stop(), terminating!        PID=1475

Logs from podman system service

time="2025-02-27T03:51:17Z" level=info msg="podman filtering at log level debug"
time="2025-02-27T03:51:17Z" level=debug msg="Called service.PersistentPreRunE(podman --log-level debug system service --time=0 unix:///run/podman/podman.sock)"
time="2025-02-27T03:51:17Z" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2025-02-27T03:51:17Z" level=info msg="Using sqlite as database backend"
time="2025-02-27T03:51:17Z" level=debug msg="Using graph driver "
time="2025-02-27T03:51:17Z" level=debug msg="Using graph root /var/lib/containers/storage"
time="2025-02-27T03:51:17Z" level=debug msg="Using run root /run/containers/storage"
time="2025-02-27T03:51:17Z" level=debug msg="Using static dir /var/lib/containers/storage/libpod"
time="2025-02-27T03:51:17Z" level=debug msg="Using tmp dir /run/libpod"
time="2025-02-27T03:51:17Z" level=debug msg="Using volume path /var/lib/containers/storage/volumes"
time="2025-02-27T03:51:17Z" level=debug msg="Using transient store: false"
time="2025-02-27T03:51:17Z" level=debug msg="overlay: test mount with multiple lowers succeeded"
time="2025-02-27T03:51:17Z" level=debug msg="Cached value indicated that overlay is supported"
time="2025-02-27T03:51:17Z" level=debug msg="overlay: test mount indicated that metacopy is not being used"
time="2025-02-27T03:51:17Z" level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false"
time="2025-02-27T03:51:17Z" level=info msg="[graphdriver] using prior storage driver: overlay"
time="2025-02-27T03:51:17Z" level=debug msg="Initializing event backend file"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2025-02-27T03:51:17Z" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2025-02-27T03:51:17Z" level=debug msg="Podman detected system restart - performing state refresh"
time="2025-02-27T03:51:17Z" level=info msg="Setting parallel job count to 25"
time="2025-02-27T03:51:17Z" level=debug msg="registered SIGHUP watcher for config"
time="2025-02-27T03:51:17Z" level=info msg="API service listening on \"/run/podman/podman.sock\". URI: \"unix:///run/podman/podman.sock\""
time="2025-02-27T03:51:17Z" level=debug msg="CORS Headers were not set"
time="2025-02-27T03:51:17Z" level=debug msg="waiting for SIGHUP to reload configuration"
time="2025-02-27T03:51:17Z" level=debug msg="API service(s) shutting down, idle for 0s"
time="2025-02-27T03:51:17Z" level=debug msg="API service shutdown request ignored as timeout Duration is UnlimitedService"
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:new 0m+0h/0t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:active 0m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
@ - - [27/Feb/2025:03:51:20 +0000] "HEAD /_ping HTTP/1.1" 200 0 "" "Docker-Client/unknown-version (linux)"
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:idle 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:active 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="overlay: test mount indicated that volatile is being used"
time="2025-02-27T03:51:20Z" level=debug msg="Check for idmapped mounts support create mapped mount: operation not permitted"
time="2025-02-27T03:51:20Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2025-02-27T03:51:20Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/002-proxy.conf\""
time="2025-02-27T03:51:20Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/shortnames.conf\""
@ - - [27/Feb/2025:03:51:20 +0000] "GET /v1.41/info HTTP/1.1" 200 2516 "" "compose/v2.33.1"
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:idle 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:active 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="Looking up image \"docker.io/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:20Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="Looking up image \"docker.io/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:20Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=info msg="Request Failed(Not Found): failed to find image docker.io/rclone/rclone:1.69: docker.io/rclone/rclone:1.69: No such image"
@ - - [27/Feb/2025:03:51:20 +0000] "GET /v1.41/images/docker.io/rclone/rclone:1.69/json HTTP/1.1" 404 229 "" "compose/v2.33.1"
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:idle 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="IdleTracker:active 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:20Z" level=debug msg="Looking up image \"rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:20Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"localhost/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]localhost/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="Pulling image docker.io/rclone/rclone:1.69 (policy: always)"
time="2025-02-27T03:51:20Z" level=debug msg="Looking up image \"docker.io/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:20Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:20Z" level=debug msg="Trying \"docker.io/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:20Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:20Z" level=debug msg="Attempting to pull candidate docker.io/rclone/rclone:1.69 for docker.io/rclone/rclone:1.69"
time="2025-02-27T03:51:20Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69\""
Trying to pull docker.io/rclone/rclone:1.69...
time="2025-02-27T03:51:20Z" level=debug msg="Copying source image //rclone/rclone:1.69 to destination image [overlay@/var/lib/containers/storage+/run/containers/storage]docker.io/rclone/rclone:1.69"
time="2025-02-27T03:51:20Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2025-02-27T03:51:20Z" level=debug msg="Trying to access \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\""
time="2025-02-27T03:51:20Z" level=debug msg="No credentials matching us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone found in /var/tmp/auth.json.1277445654"
time="2025-02-27T03:51:20Z" level=debug msg="No credentials for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone found"
time="2025-02-27T03:51:20Z" level=debug msg=" No signature storage configuration found for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69, using built-in default file:///var/lib/containers/sigstore"
time="2025-02-27T03:51:20Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/us-docker.pkg.dev"
time="2025-02-27T03:51:20Z" level=debug msg="GET https://us-docker.pkg.dev/v2/"
time="2025-02-27T03:51:22Z" level=debug msg="Ping https://us-docker.pkg.dev/v2/ status 401"
time="2025-02-27T03:51:22Z" level=debug msg="GET https://us-docker.pkg.dev/v2/token?scope=repository%3Acart-repos-shared-15f6%2Fcart-docker-mirror%2Frclone%2Frclone%3Apull"
time="2025-02-27T03:51:24Z" level=debug msg="Accessing \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" failed: Requesting bearer token: invalid status code from registry 403 (Forbidden)"
time="2025-02-27T03:51:24Z" level=debug msg="Error pulling candidate docker.io/rclone/rclone:1.69: initializing source docker://rclone/rclone:1.69: Requesting bearer token: invalid status code from registry 403 (Forbidden)"
@ - - [27/Feb/2025:03:51:20 +0000] "POST /v1.41/images/create?fromImage=rclone%2Frclone&tag=1.69 HTTP/1.1" 500 89 "" "compose/v2.33.1"
time="2025-02-27T03:51:24Z" level=debug msg="IdleTracker:idle 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:24Z" level=debug msg="IdleTracker:closed 1m+0h/1t connection(s)" X-Reference-Id=0xc000081e18
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:new 0m+0h/1t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:active 0m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
@ - - [27/Feb/2025:03:51:33 +0000] "HEAD /_ping HTTP/1.1" 200 0 "" "Docker-Client/unknown-version (linux)"
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:idle 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:active 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
@ - - [27/Feb/2025:03:51:33 +0000] "GET /v1.41/info HTTP/1.1" 200 2515 "" "compose/v2.33.1"
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:idle 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:active 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:33Z" level=debug msg="Looking up image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="Looking up image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=info msg="Request Failed(Not Found): failed to find image us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69: us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69: No such image"
@ - - [27/Feb/2025:03:51:33 +0000] "GET /v1.41/images/us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69/json HTTP/1.1" 404 429 "" "compose/v2.33.1"
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:idle 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:33Z" level=debug msg="IdleTracker:active 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:33Z" level=debug msg="Looking up image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="Pulling image us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69 (policy: always)"
time="2025-02-27T03:51:33Z" level=debug msg="Looking up image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" does not resolve to an image ID"
time="2025-02-27T03:51:33Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:33Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:33Z" level=debug msg="Attempting to pull candidate us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69 for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69"
time="2025-02-27T03:51:33Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\""
Trying to pull us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69...
time="2025-02-27T03:51:33Z" level=debug msg="Copying source image //us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69 to destination image [overlay@/var/lib/containers/storage+/run/containers/storage]us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69"
time="2025-02-27T03:51:33Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2025-02-27T03:51:33Z" level=debug msg="Trying to access \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\""
time="2025-02-27T03:51:33Z" level=debug msg="Returning credentials for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone from DockerAuthConfig"
time="2025-02-27T03:51:33Z" level=debug msg=" No signature storage configuration found for us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69, using built-in default file:///var/lib/containers/sigstore"
time="2025-02-27T03:51:33Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/us-docker.pkg.dev"
time="2025-02-27T03:51:33Z" level=debug msg="GET https://us-docker.pkg.dev/v2/"
time="2025-02-27T03:51:35Z" level=debug msg="Ping https://us-docker.pkg.dev/v2/ status 401"
time="2025-02-27T03:51:35Z" level=debug msg="GET https://us-docker.pkg.dev/v2/token?account=oauth2accesstoken&scope=repository%3Acart-repos-shared-15f6%2Fcart-docker-mirror%2Frclone%2Frclone%3Apull"
time="2025-02-27T03:51:37Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/manifests/1.69"
time="2025-02-27T03:51:38Z" level=debug msg="Content-Type from manifest GET is \"application/vnd.oci.image.index.v1+json\""
time="2025-02-27T03:51:38Z" level=debug msg="Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite"
time="2025-02-27T03:51:38Z" level=debug msg="Source is a manifest list; copying (only) instance sha256:7fe21a46326000f54b0321949d34a58d7503cac1cffcc4b4e284513cfb76f1de for current system"
time="2025-02-27T03:51:38Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/manifests/sha256:7fe21a46326000f54b0321949d34a58d7503cac1cffcc4b4e284513cfb76f1de"
time="2025-02-27T03:51:39Z" level=debug msg="Content-Type from manifest GET is \"application/vnd.oci.image.manifest.v1+json\""
time="2025-02-27T03:51:39Z" level=debug msg="IsRunningImageAllowed for image docker:us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69"
time="2025-02-27T03:51:39Z" level=debug msg=" Using default policy section"
time="2025-02-27T03:51:39Z" level=debug msg=" Requirement 0: allowed"
time="2025-02-27T03:51:39Z" level=debug msg="Overall: allowed"
time="2025-02-27T03:51:39Z" level=debug msg="Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5"
time="2025-02-27T03:51:39Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5"
Getting image source signatures
time="2025-02-27T03:51:40Z" level=debug msg="Reading /var/lib/containers/sigstore/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone@sha256=7fe21a46326000f54b0321949d34a58d7503cac1cffcc4b4e284513cfb76f1de/signature-1"
time="2025-02-27T03:51:40Z" level=debug msg="Not looking for sigstore attachments: disabled by configuration"
time="2025-02-27T03:51:40Z" level=debug msg="Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json]"
time="2025-02-27T03:51:40Z" level=debug msg="... will first try using the original manifest unmodified"
Copying blob sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b
time="2025-02-27T03:51:40Z" level=debug msg="Checking if we can reuse blob sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar+gzip\" = true"
Copying blob sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22
time="2025-02-27T03:51:40Z" level=debug msg="Checking if we can reuse blob sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar+gzip\" = true"
Copying blob sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09
time="2025-02-27T03:51:40Z" level=debug msg="Checking if we can reuse blob sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar+gzip\" = true"
Copying blob sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e
time="2025-02-27T03:51:40Z" level=debug msg="Checking if we can reuse blob sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar+gzip\" = true"
Copying blob sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0
time="2025-02-27T03:51:40Z" level=debug msg="Checking if we can reuse blob sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar+gzip\" = true"
time="2025-02-27T03:51:40Z" level=debug msg="Failed to retrieve partial blob: convert_images not configured"
time="2025-02-27T03:51:40Z" level=debug msg="Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b"
time="2025-02-27T03:51:40Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:f27ed7a90d4aa60e6105c4ccbf3ddb993fa2e99d961f81d978547c84c9ff031b"
time="2025-02-27T03:51:40Z" level=debug msg="Failed to retrieve partial blob: convert_images not configured"
time="2025-02-27T03:51:40Z" level=debug msg="Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22"
time="2025-02-27T03:51:40Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:d12573aef0cf7086fcea1be041e531e660eb0c5e85e947837d107baa4beeba22"
time="2025-02-27T03:51:40Z" level=debug msg="Failed to retrieve partial blob: convert_images not configured"
time="2025-02-27T03:51:40Z" level=debug msg="Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09"
time="2025-02-27T03:51:40Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:06487e735056858f0f47803456163b65180dd1fa23675cd20b2baa80170f8d09"
time="2025-02-27T03:51:40Z" level=debug msg="Failed to retrieve partial blob: convert_images not configured"
time="2025-02-27T03:51:40Z" level=debug msg="Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e"
time="2025-02-27T03:51:40Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:3ba39bb8e169a497387a605bf67894944e0596e227a0c5351aad93fedec3ba9e"
time="2025-02-27T03:51:40Z" level=debug msg="Failed to retrieve partial blob: convert_images not configured"
time="2025-02-27T03:51:40Z" level=debug msg="Downloading /v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0"
time="2025-02-27T03:51:40Z" level=debug msg="GET https://us-docker.pkg.dev/v2/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone/blobs/sha256:1f3e46996e2966e4faa5846e56e76e3748b7315e2ded61476c24403d592134f0"
time="2025-02-27T03:51:40Z" level=debug msg="Detected compression format gzip"
time="2025-02-27T03:51:40Z" level=debug msg="Using original blob without modification"
time="2025-02-27T03:51:42Z" level=debug msg="Detected compression format gzip"
time="2025-02-27T03:51:42Z" level=debug msg="Using original blob without modification"
time="2025-02-27T03:51:42Z" level=debug msg="Detected compression format gzip"
time="2025-02-27T03:51:42Z" level=debug msg="Using original blob without modification"
time="2025-02-27T03:51:42Z" level=debug msg="Detected compression format gzip"
time="2025-02-27T03:51:42Z" level=debug msg="Using original blob without modification"
time="2025-02-27T03:51:43Z" level=debug msg="Detected compression format gzip"
time="2025-02-27T03:51:43Z" level=debug msg="Using original blob without modification"
time="2025-02-27T03:51:44Z" level=debug msg="Applying tar in /var/lib/containers/storage/overlay/a0904247e36a7726c03c71ee48f3e64462021c88dafeb13f37fdaf613b27f11c/diff"
time="2025-02-27T03:51:44Z" level=debug msg="Applying tar in /var/lib/containers/storage/overlay/09206d0f222e40994fb73296763114ad344442dfc815599cf9037531e80d8d31/diff"
time="2025-02-27T03:51:45Z" level=debug msg="Applying tar in /var/lib/containers/storage/overlay/705bc69039a1f0d0bad245623c7ba8c8f6c82debe7c867d597b7803f2743c37a/diff"
time="2025-02-27T03:51:45Z" level=debug msg="Applying tar in /var/lib/containers/storage/overlay/0aa5d9d0b8f67e75d245272b0c9018acd73adc35e7173137dddffcbe10ba3ee9/diff"
time="2025-02-27T03:51:45Z" level=debug msg="Applying tar in /var/lib/containers/storage/overlay/cb06561ad00a0b1fce65813322a374f7d2a7a2a0ed7cbea48ad81aa212a5d377/diff"
Copying config sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5
time="2025-02-27T03:51:45Z" level=debug msg="No compression detected"
time="2025-02-27T03:51:45Z" level=debug msg="Compression change for blob sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5 (\"application/vnd.oci.image.config.v1+json\") not supported"
time="2025-02-27T03:51:45Z" level=debug msg="Using original blob without modification"
Writing manifest to image destination
time="2025-02-27T03:51:45Z" level=debug msg="setting image creation date to 2025-02-14 15:19:30.479365755 +0000 UTC"
time="2025-02-27T03:51:45Z" level=debug msg="created new image ID \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" with metadata \"{}\""
time="2025-02-27T03:51:45Z" level=debug msg="added name \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" to image \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="Pulled candidate us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69 successfully"
time="2025-02-27T03:51:45Z" level=debug msg="Looking up image \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" in local containers storage"
time="2025-02-27T03:51:45Z" level=debug msg="Trying \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" ..."
time="2025-02-27T03:51:45Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="Found image \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" as \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" in local containers storage"
time="2025-02-27T03:51:45Z" level=debug msg="Found image \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" as \"c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5)"
time="2025-02-27T03:51:45Z" level=debug msg="exporting opaque data as blob \"sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
@ - - [27/Feb/2025:03:51:33 +0000] "POST /v1.41/images/create?fromImage=us-docker.pkg.dev%2Fcart-repos-shared-15f6%2Fcart-docker-mirror%2Frclone%2Frclone&tag=1.69 HTTP/1.1" 200 1294 "" "compose/v2.33.1"
time="2025-02-27T03:51:45Z" level=debug msg="IdleTracker:idle 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:45Z" level=debug msg="IdleTracker:active 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:45Z" level=debug msg="Looking up image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:45Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:45Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:45Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="Found image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" as \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:45Z" level=debug msg="Found image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" as \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5)"
time="2025-02-27T03:51:45Z" level=debug msg="exporting opaque data as blob \"sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="Looking up image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:45Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-02-27T03:51:45Z" level=debug msg="Trying \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" ..."
time="2025-02-27T03:51:45Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="Found image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" as \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage"
time="2025-02-27T03:51:45Z" level=debug msg="Found image \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" as \"us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69\" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5)"
time="2025-02-27T03:51:45Z" level=debug msg="exporting opaque data as blob \"sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="Inspecting image c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5"
time="2025-02-27T03:51:45Z" level=debug msg="exporting opaque data as blob \"sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
time="2025-02-27T03:51:45Z" level=debug msg="exporting opaque data as blob \"sha256:c461d84251fb0cd58350147be3f8639eadb413399579a30b0b33dbfb722eabe5\""
@ - - [27/Feb/2025:03:51:45 +0000] "GET /v1.41/images/us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror/rclone/rclone:1.69/json HTTP/1.1" 200 3291 "" "compose/v2.33.1"
time="2025-02-27T03:51:45Z" level=debug msg="IdleTracker:idle 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0
time="2025-02-27T03:51:45Z" level=debug msg="IdleTracker:closed 1m+0h/2t connection(s)" X-Reference-Id=0xc0006260a0

Describe the results you expected

Compose pull the mirror url works. Compose pull indirected works. Podman pull indirected works.

podman info output

$ docker-compose --version
Docker Compose version v2.33.1
$ podman version
Client:       Podman Engine
Version:      5.3.2
API Version:  5.3.2
Go Version:   go1.23.5
Built:        Mon Jan 27 11:57:58 2025
OS/Arch:      linux/amd64
root ➜ /work $ podman info
host:
  arch: amd64
  buildahVersion: 1.38.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-3_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 92.67
    systemPercent: 1.38
    userPercent: 5.95
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: unknown
  eventLogger: file
  freeLocks: 2048
  hostname: 52a48b82d7ef
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.13.4-arch1-1
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 767709184
  memTotal: 16248848384
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.12.2-1_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark_1.12.1-3_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun_1.18-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.18
      commit: 8656b2548509fcc69ea7e8823a870564360a57a1
      rundir: /tmp/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240906.6b38f07-1_amd64
    version: |
      pasta 0.0~git20240906.6b38f07-1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 15031246848
  swapTotal: 17179865088
  uptime: 28h 50m 46.00s (Approximately 1.17 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: us-docker.pkg.dev/cart-repos-shared-15f6/cart-docker-mirror
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    PullFromMirror: ""
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 485464547328
  graphRootUsed: 307144765440
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.3.2
  Built: 1737979078
  BuiltTime: Mon Jan 27 11:57:58 2025
  GitCommit: ""
  GoVersion: go1.23.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.2

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Tried in GCP as root and local root in container.

Additional information

Latest 5.4.0 does not seem to have podman system service to create socket for compose

The text was updated successfully, but these errors were encountered:

cheewee-cartology added the kind/bug Categorizes issue or PR as related to a bug. label Feb 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to authenticate in compose with redirected registry #25410

Unable to authenticate in compose with redirected registry #25410

cheewee-cartology commented Feb 27, 2025 •

edited

Loading

Unable to authenticate in compose with redirected registry #25410

Unable to authenticate in compose with redirected registry #25410

Comments

cheewee-cartology commented Feb 27, 2025 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

cheewee-cartology commented Feb 27, 2025 •

edited

Loading