Support for Docker compose files

[greysteil]

From @armin-joellenbeck on December 23, 2017 9:21

Knowing when new Docker images are published would be helpful when the are used in a Docker compose file too.

Just like #20, with the file docker-compose.yml instead of Dockerfile.

Copied from original issue: dependabot/feedback#66

[greysteil]

👍 for this, and should be relatively straightforward. A couple of things I want to get to first, but I'm definitely game for adding this to Dependabot at some point!

[victorcerutti]

Hi,
This would be a really good addition, is this still going to happen someday ?

Thanks in advance

[greysteil]

I hope so, yes! This is another one that @hmarr owns on our side, but he's very busy scaling Dependabot up to 100m repos!

[bkaid]

@stalebot please leave this open - hoping it gets implemented

[gegoune]

@robaiken It works better now, thanks. Unfortunately it is still impossible to tell what's going on.

I don't think registries allow searching by digest (at least Docker Hub doesn't seem to), meaning that it's impossible or at least very difficult to tell which version PR updates to with just:

build(deps): bump postgres from `a35ec42` to `0321e22`

Can human readable versions be used instead?

Also, to bring my other point from previous comment. It's common practice to keep human-readable version string in comment after digest. I would like to see that string updated as well, otherwise we run into issue as described in first paragraph again.

So,

image: postgres@sha256:a35ec42526e3c522eb13b4d82eddaee875d0ac6ca9eb5cc5607e412854478c71 # 16.7

to be updated to

image: postgres@sha256:{new hash} # 16.8

(theoretical version)

[robaiken]

@gegoune it's definitely a limitation with how things work right now. The version readability stuff is a bit outside what we can tackle in this implementation.

Mind opening up a feature request for this? Would be great to track things like showing readable versions and keeping those comments in sync. It'll help us plan this out properly for the future.

[Aaron-Ritter]

Appears to be failing for me 🤔 Attaching a snippet with a stacktrace I just saw in GH Actions:

Dependabot encountered '1' error(s) during execution, please check the logs for more details.

For what it's worth, some of the images in my docker-compose.yml are "parametrized", e.g. look like this:

services:
api:
image: ${E2E_API_IMAGE_TAG}
pull_policy: missing
# ...

@robaiken FYI in case you are planning to address env param in the future. I had the same issue with a test, which had mostly parameterized image entries with the exception of one "hard-coded". In my example it wouldn't have been able to lookup the ENV as it is generated ad-hoc with test scripts, but my plan was to still use dependabot to update the "hard-coded" ones:
https://github.com/sonderformat-llc/Mailu/actions/runs/13440943004/job/37554951541

Good news, the new version with #11656, works for the "hard-coded" image entry and skips the rest:
https://github.com/sonderformat-llc/Mailu/actions/runs/13462353541/job/37620450101

[Bert-R]

@robaiken Could it be that you only read the first part of #390 (comment) ("Thanks for fixing this! It works for me on a whole set of repositories")? I see a 🎉 reaction, but I also reported an issue ("but it fails for Elasticsearch, Kibana and Filebeat and also for NGINX."). The log is included in that comment. I just retested and it still fails.

The compose file is trivially simple:

services:
  elasticsearch:
    image: "elastic/elasticsearch:8.16.4"
    container_name: "elasticsearch"
    volumes:
      - "/.../data:/usr/share/elasticsearch/data"
      - "/.../config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml"
    ports:
      - "9200:9200"
      - "9300:9300"
    network_mode: "bridge"
    environment:
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
    deploy:
      resources:
        limits:
          memory: 4GB
    restart: "always"

The compose files for Filebeat and Kibana have the same structure and reference the same version number.

For NGINX, this is the compose file:

services:
    nginx:
        image: "nginx:1.27.2"
        container_name: nginx
        volumes:
            - "/.../config/nginx.conf:/etc/nginx/nginx.conf"
            - "/.../config/conf.d:/etc/nginx/conf.d"
            - "/.../config/extra:/etc/nginx/extra"
            - "/.../certs:/etc/ssl/private"
        ports:
            - "80:80"
            - "443:443"
        network_mode: "bridge"
        extra_hosts:
            - "host.docker.internal:host-gateway"
        restart: always

[gmile]

Looks like dependabot will refuse to upgrade an image with the following tag:

service:
  serviceA:
    image: hexpm/elixir:1.18.2-erlang-27.2.2-alpine-3.21.2

However, today the following images are already available:

hexpm/elixir:1.18.2-erlang-27.2.3-alpine-3.21.3
hexpm/elixir:1.18.2-erlang-27.2.4-alpine-3.21.3

For reference, these are official images built & published by Elixir / Hexpm team, see https://github.com/hexpm/bob?tab=readme-ov-file#docker-images

[Aaron-Ritter]

@gmile this could be related to this #10678.

And i had previously similar issues with updates in Dockerfile and k8s files, as soon i used a definition of repository_name:semver-alpine-semver, or for example with mongo-express where it suggested to update 1.0.2-18-alpine3.18 to 1-18-alpine3.18. Plus dependabot does as well some evaluation on where the latest tag is.

These complex tags seem to be generally an issue for dependabot #10385 #8648.

[robaiken]

@Aaron-Ritter Thanks for reporting this! The current implementation focuses only on hardcoded image tags, and I'm glad it's working for those in your setup.

For images using environment variables (like ${E2E_API_IMAGE_TAG}), would you mind opening a feature request? This would be a separate feature requiring its own implementation.

[Aaron-Ritter]

@robaiken sure, will create a issue accordingly once i've updated everything with the official release.

With regards to the earlier reported issue #390 (comment) on images not updating and trowing an error.

This seems to be related to the quotation of the images, i updated my test with single and double quotes as well as without quotes, the one without is now updating propperly:
https://github.com/sonderformat-llc/novu/actions/runs/13496505546/job/37704897458

So the issue reported from @Bert-R seems to be as well related to the quotation as double quotes are used in this example:
#390 (comment)

[robaiken]

@Aaron-Ritter Thanks for the detailed investigation! Yes, you're correct - I've submitted PR #11669 to add support for images with quotes. This should resolve the issues you and @Bert-R encountered with quoted image tags.

[Bert-R]

@robaiken In can confirm it's fixed now. Thanks!

[robaiken]

@gmile Could you share the job logs from one of your failed runs? You can find them following GitHub's guide on viewing Dependabot logs.

[Aaron-Ritter]

@robaiken thanks a lot! I can confirm this now works for all my tests.

https://github.com/sonderformat-llc/novu/actions/runs/13498872364/job/37712208984
https://github.com/sonderformat-llc/novu/actions/runs/13498870760/job/37712205124

[gmile]

@robaiken sure. I get this at the end of logs:

2025-02-24T11:44:24.5799611Z updater | 2025/02/24 11:44:24 INFO <job_969523985> Finished job processing
2025-02-24T11:44:24.5810598Z updater | 2025/02/24 11:44:24 INFO Results:
2025-02-24T11:44:24.5813338Z Dependabot encountered '1' error(s) during execution, please check the logs for more details.
2025-02-24T11:44:24.5815455Z +-----------------------------------------------+
2025-02-24T11:44:24.5815963Z |         Dependencies failed to update         |
2025-02-24T11:44:24.5816455Z +--------------+--------------------------------+
2025-02-24T11:44:24.5816966Z | hexpm/elixir | dependency_file_not_resolvable |
2025-02-24T11:44:24.5817478Z +--------------+--------------------------------+

Can't share the entire log publicly for privacy reasons, but here are Elixir-related lines from it - this is what was logged before dependabot moved on checking another image. To sum up, it seems like its attempts to access registry.hub.docker.com are failing:

2025-02-24T11:43:51.7383374Z updater | 2025/02/24 11:43:51 INFO <job_969523985> Checking if hexpm/elixir 1.18.2-erlang-27.2.2-alpine-3.21.2 needs updating
2025-02-24T11:43:51.7704386Z   proxy | 2025/02/24 11:43:51 [236] GET https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:43:51.8076709Z   proxy | 2025/02/24 11:43:51 [236] 401 https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:43:51.8389915Z   proxy | 2025/02/24 11:43:51 [238] GET https://auth.docker.io:443/token?service=registry.docker.io&scope=repository%3Ahexpm%2Felixir%3Apull&account
2025-02-24T11:43:51.9669235Z   proxy | 2025/02/24 11:43:51 [238] 200 https://auth.docker.io:443/token?service=registry.docker.io&scope=repository%3Ahexpm%2Felixir%3Apull&account
2025-02-24T11:43:51.9979545Z   proxy | 2025/02/24 11:43:51 [240] GET https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:43:57.0077144Z   proxy | 2025/02/24 11:43:57 [242] GET https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:43:59.5757363Z   proxy | 2025/02/24 11:43:59 [240] 200 https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:43:59.5759220Z   proxy | 2025/02/24 11:43:59 [240] WARN: Cannot write TLS response header from mitm'd client: write tcp 172.19.0.2:1080->172.19.0.3:48122: write: broken pipe
2025-02-24T11:44:02.0449152Z   proxy | 2025/02/24 11:44:02 [244] GET https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:44:02.0451276Z 2025/02/24 11:44:02 [244] 401 https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:44:02.0790947Z   proxy | 2025/02/24 11:44:02 [246] GET https://auth.docker.io:443/token?service=registry.docker.io&scope=repository%3Ahexpm%2Felixir%3Apull&account
2025-02-24T11:44:02.0799472Z   proxy | 2025/02/24 11:44:02 [246] 200 https://auth.docker.io:443/token?service=registry.docker.io&scope=repository%3Ahexpm%2Felixir%3Apull&account
2025-02-24T11:44:02.1109714Z   proxy | 2025/02/24 11:44:02 [248] GET https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:44:02.1112752Z 2025/02/24 11:44:02 [248] 200 https://registry.hub.docker.com:443/v2/hexpm/elixir/tags/list
2025-02-24T11:44:02.1122214Z updater | 2025/02/24 11:44:02 INFO <job_969523985> Adding dependencies as handled: (hexpm/elixir).
2025-02-24T11:44:02.1579650Z   proxy | 2025/02/24 11:44:02 [250] POST /update_jobs/969523985/record_update_job_error
2025-02-24T11:44:02.2435527Z   proxy | 2025/02/24 11:44:02 [250] 204 /update_jobs/969523985/record_update_job_error
2025-02-24T11:44:02.2448795Z updater | 2025/02/24 11:44:02 INFO <job_969523985> Handled error whilst updating hexpm/elixir: dependency_file_not_resolvable {:message=>"Error while accessing docker image at registry.hub.docker.com"}

Upd. For what it's worth, the number of image tags contained in the repository is:

curl --silent https://hub.docker.com/v2/repositories/hexpm/elixir/tags | jq '.count'
152536

[robaiken]

@gmile This seems to happen because hexpm/elixir have thousands of tags. The default timeout was 5 seconds when obtaining these tags, I have updated this to 60 seconds #11676

I got a successful run 🥳

updater | 2025/02/25 15:16:27 INFO <job_970512666> Finished job processing
updater | 2025/02/25 15:16:27 INFO Results:
+----------------------------------------------------------------------------------------------------------+
|                                   Changes to Dependabot Pull Requests                                    |
+---------+------------------------------------------------------------------------------------------------+
| created | hexpm/elixir ( from 1.18.2-erlang-27.2.2-alpine-3.21.2 to 1.18.2-erlang-27.2.4-alpine-3.21.3 ) |
+---------+------------------------------------------------------------------------------------------------+
Cleaned up container 34ca6d6b221d9b28edab9dda39699e831e2eea9d8da467652783a023f5a842a2
  proxy | 2025/02/25 15:16:27 0/12 calls cached (0%)
2025/02/25 15:16:27 Posting metrics to remote API endpoint

[gmile]

Awesome, thank you so much! I'll wait until tomorrow morning to verify if this worked, since it doesn't seem like re-triggering dependabot workflow is possible in principle (attempting to do so returns a warning message in the workflow run, but results in a no-op)

[robaiken]

@gmile To manually retrigger a Dependabot job (it's hidden pretty deep in GitHub's UI):

1. Go to your repository
2. Click the 'Insights' tab at the top
3. Select 'Dependency graph' from the left sidebar
4. Click the 'Dependabot' tab
5. Find your specific dependency alert
6. Click 'Recent update jobs'
7. Press the 'Check for updates' button

This path isn't obvious, it is really buried! Hopefully, this helps

[wjam]

I've managed to find an image which isn't getting updated. I've got a private repository which contains the quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z image but dependabot isn't suggesting RELEASE.2025-02-18T16-25-55Z which is the current alias to latest.

There is very little that I can see in the logs aside from these three lines:

2025-02-25T19:39:31.6710531Z updater | 2025/02/25 19:39:31 INFO <job_970622557> Checking if minio/minio RELEASE.2025-01-20T14-49-07Z needs updating
2025-02-25T19:39:31.6714026Z updater | 2025/02/25 19:39:31 INFO <job_970622557> Latest version is RELEASE.2025-01-20T14-49-07Z
2025-02-25T19:39:31.6715237Z 2025/02/25 19:39:31 INFO <job_970622557> No update needed for minio/minio RELEASE.2025-01-20T14-49-07Z

[abdulapopoola]

@wjam , can you please file a new bug?

[wjam]

@wjam , can you please file a new bug?

Raised #11680 for the issue

[wjam]

Also, the GitHub issue labelling currently doesn't understand docker-compose!

[Marinofull]

Dependabot fails to open for private repositories (in this case, also inside the same organization). Is there a way to make the authentication work or my only option is to skip them?

[robaiken]

Hi @Marinofull, I see this issue has already been reported here: #11686
Would you mind adding your specific details to that thread? Particularly helpful would be:

1. The exact package and version you're attempting to update
2. Your run logs (instructions on obtaining these logs can be found here)

Contributing this information helps us better understand the scale of the issue and identify patterns across different environments.