Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to restore cache only #663

Open
ForsakenHarmony opened this issue Jan 10, 2023 · 5 comments
Open

Add option to restore cache only #663

ForsakenHarmony opened this issue Jan 10, 2023 · 5 comments
Labels
feature request New feature or request to improve the current logic

Comments

@ForsakenHarmony
Copy link

ForsakenHarmony commented Jan 10, 2023

Description:
It would be nice to have the option to restore only and not save.

Justification:
In bigger repos, it's easy to hit the 10gb cache limit. Not caching in PRs can help with this.
Right now, I have to manually delete caches to help keep the caches on main alive.

Are you willing to submit a PR?

References
actions/toolkit#1308
Swatinem/rust-cache#95 (has a save-if option https://github.com/Swatinem/rust-cache#example-usage)

@ForsakenHarmony ForsakenHarmony added feature request New feature or request to improve the current logic needs triage labels Jan 10, 2023
@dmitry-shibanov
Copy link
Contributor

Hello @ForsakenHarmony. Thank you for your report. We'll investigate the issue.

@desrosj
Copy link

desrosj commented Oct 25, 2023

Found this searching for an open issue because I wanted to request the same feature. Some context on the specific use case I have.

In workflows triggered with the pull_request_target event, the recommendation in the docs is as follows:

Although the workflow runs in the context of the base of the pull request, you should make sure that you do not check out, build, or run untrusted code from the pull request with this event. Additionally, any caches share the same scope as the base branch. To help prevent cache poisoning, you should not save the cache if there is a possibility that the cache contents were altered.

When using the setup-node action, I'd like to turn off cache saving for runs triggered with this event.

@hugo082
Copy link

hugo082 commented Dec 28, 2023

It can also help when a GitHub matrix is used. On large matrices, we may end with concurrency on cache write, forcing the job to wait (more than 1m on my repo).

Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/... --files-from manifest.txt --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key <KEY>, another job may be creating this cache. More details: Cache already exists. Scope: refs/heads/main, Key: <KEY>, Version: <HASH>

@marcferreiro
Copy link

Hi! Do you have any plan about this? In our case, we're handling the cache with a custom system, so we'd love to save some time in our workflows by disabling the save step.

@maschwenk
Copy link

@ForsakenHarmony agree. In a big monorepo with 500+ open PRs and ~100 commits per day, the cache becomes unusable because there's too many circumstances where the main cache gets evicted because of hundreds of non-main caches. I can't decide if:

  1. I need to control which bits of my pipeline are restore-only
  2. I should automate purging all non-main caches

But it feels like a little of both are necessary

github-merge-queue bot pushed a commit to camunda/camunda that referenced this issue Aug 22, 2024
## Description

This PRs adds a custom action that uses similar logic like [Zeebe has
for
Maven](https://github.com/camunda/camunda/blob/fd2e86a59785408bd39daa1134aa1d292281c390/.github/actions/setup-zeebe/action.yml#L113-L180)
combined with Yarn-specific paths from
https://github.com/actions/setup-node/blob/26961cf329f22f6837d5f54c3efd76b480300ace/src/cache-utils.ts#L39-L66
([setup-node cannot do
it](actions/setup-node#663)) to implement our
[caching
strategy](https://github.com/camunda/camunda/wiki/CI-&-Automation#caching-strategy).

To demonstrate that this causes no problems I changed only 1 job to use
it, see
https://github.com/camunda/camunda/actions/runs/10508214639/job/29111637484?pr=21475
-- remaining jobs will be refactored in
#21427

Old usage:

```yaml
    - name: Setup yarn cache
      uses: actions/setup-node@v4
      with:
        node-version: "20"
        cache: "yarn"
        cache-dependency-path: project-path/yarn.lock
```

New usage after this PR:

```yaml
    - name: Setup yarn cache
      uses: actions/setup-node@v4
      with:
        node-version: "20"
    - uses: ./.github/actions/setup-yarn-cache
      with:
        directory: project-path
```

As a test in
d432446
I replaced all usages like this and indeed no **node** caches were
created on the PR:

![image](https://github.com/user-attachments/assets/4031032a-5f7a-43cd-88f4-d12950ef9c63)

## Checklist

<!--- Please delete options that are not relevant. Boxes should be
checked by reviewer. -->
- [x] for CI changes:
- [x] structural/foundational changes signed off by [CI
DRI](https://github.com/cmur2)
- [x]
[ci.yml](https://github.com/camunda/camunda/blob/main/.github/workflows/ci.yml)
modifications comply with ["Unified CI"
requirements](https://github.com/camunda/camunda/wiki/CI-&-Automation#workflow-inclusion-criteria)

## Related issues

Related to #21424
github-merge-queue bot pushed a commit to camunda/camunda that referenced this issue Aug 22, 2024
## Description

This PRs adds a custom action that uses similar logic like [Zeebe has
for
Maven](https://github.com/camunda/camunda/blob/fd2e86a59785408bd39daa1134aa1d292281c390/.github/actions/setup-zeebe/action.yml#L113-L180)
combined with Yarn-specific paths from
https://github.com/actions/setup-node/blob/26961cf329f22f6837d5f54c3efd76b480300ace/src/cache-utils.ts#L39-L66
([setup-node cannot do
it](actions/setup-node#663)) to implement our
[caching
strategy](https://github.com/camunda/camunda/wiki/CI-&-Automation#caching-strategy).

To demonstrate that this causes no problems I changed only 1 job to use
it, see
https://github.com/camunda/camunda/actions/runs/10508214639/job/29111637484?pr=21475
-- remaining jobs will be refactored in
#21427

Old usage:

```yaml
    - name: Setup yarn cache
      uses: actions/setup-node@v4
      with:
        node-version: "20"
        cache: "yarn"
        cache-dependency-path: project-path/yarn.lock
```

New usage after this PR:

```yaml
    - name: Setup yarn cache
      uses: actions/setup-node@v4
      with:
        node-version: "20"
    - uses: ./.github/actions/setup-yarn-cache
      with:
        directory: project-path
```

As a test in
d432446
I replaced all usages like this and indeed no **node** caches were
created on the PR:

![image](https://github.com/user-attachments/assets/4031032a-5f7a-43cd-88f4-d12950ef9c63)

## Checklist

<!--- Please delete options that are not relevant. Boxes should be
checked by reviewer. -->
- [x] for CI changes:
- [x] structural/foundational changes signed off by [CI
DRI](https://github.com/cmur2)
- [x]
[ci.yml](https://github.com/camunda/camunda/blob/main/.github/workflows/ci.yml)
modifications comply with ["Unified CI"
requirements](https://github.com/camunda/camunda/wiki/CI-&-Automation#workflow-inclusion-criteria)

## Related issues

Related to #21424
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request to improve the current logic
Projects
None yet
Development

No branches or pull requests

8 participants
@desrosj @maschwenk @ForsakenHarmony @hugo082 @dmitry-shibanov @dusan-trickovic @marcferreiro and others