Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use GHA cache for yocto sstate and shared downloads #351

Closed
wants to merge 2 commits into from

Conversation

klutchell
Copy link
Contributor

@klutchell klutchell commented Jun 13, 2024

The current shared NFS cache is insecure as any workflow using our
self-hosted runners has full access to all cached files, including
private device types and sources.

This change drops the NFS in favour of GitHub built-in cache features.

These cache entries are very large, near the maximum 10gb size,
so they don't last very long in testing. A day or two.

However, it means we don't have to maintain a gigantic shared
cache and clean it regularly. But we will still sometimes
get some faster builds with cache-hit.

We could also run a local S3 or NFS instance on the self-hosted
runner stack to get more than 10gb of cache storage, but we will have
the same issue with permissions being too open to all workflows
unless we switch to a public cloud that supports OIDC.

Change-type: minor

@klutchell klutchell temporarily deployed to balena-staging.com June 13, 2024 18:55 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com June 27, 2024 15:54 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com July 2, 2024 13:49 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 2, 2024 15:18 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 2, 2024 15:18 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 2, 2024 15:18 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com July 2, 2024 19:21 — with GitHub Actions Inactive
@klutchell klutchell changed the title Cache yocto sstate and shared downloads via GHA cache Use GHA cache for yocto sstate and shared downloads Jul 2, 2024
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 2, 2024 20:08 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 2, 2024 20:08 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 2, 2024 20:08 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com July 5, 2024 13:35 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 5, 2024 14:21 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 5, 2024 14:21 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to bm.balena-dev.com July 5, 2024 14:21 — with GitHub Actions Inactive
@klutchell
Copy link
Contributor Author

klutchell commented Sep 18, 2024

Last test results: https://github.com/balena-os/balena-yocto-scripts/actions/runs/10927276022/job/30333070956?pr=351

  • Sync downloads cache: 0s (not found)
  • Sync sstate cache: 0s (not found)
  • Build: 1h 19m 25s
  • Sync sstate cache: 2m 27s
    Cache Size: ~4395 MB (4608746647 B)
    Cache saved successfully
    Cache saved with key: yocto-sstate-generic-amd6[4](https://github.com/balena-os/balena-yocto-scripts/actions/runs/10927276022/job/30333070956?pr=351#step:74:4)
    
  • Post Sync downloads cache: 5m 29s
    Cache Size: ~9285 MB (9736044847 B)
    Cache saved successfully
    Cache saved with key: yocto-downloads-generic-amd6[4](https://github.com/balena-os/balena-yocto-scripts/actions/runs/10927276022/job/30333070956?pr=351#step:75:4)
    
  • Total: 1h 31m 11s

The current shared NFS cache is insecure as any workflow using our
self-hosted runners has full access to all cached files, including
private device types and sources.

This change drops the NFS in favour of GitHub built-in cache features.

These cache entries are very large, near the maximum 10gb size,
so they don't last very long in testing. A day or two.

However, it means we don't have to maintain a gigantic shared
cache and clean it regularly. But we will still sometimes
get some faster builds with cache-hit.

We could also run a local S3 or NFS instance on the self-hosted
runner stack to get more than 10gb of cache storage, but we will have
the same issue with permissions being too open to all workflows
unless we switch to a public cloud that supports OIDC.

Change-type: minor
Signed-off-by: Kyle Harding <[email protected]>
@klutchell
Copy link
Contributor Author

Last test results: https://github.com/balena-os/balena-yocto-scripts/actions/runs/11630209275/job/32388667813?pr=351

  • Cache not found for input keys: yocto-downloads
  • Cache not found for input keys: yocto-sstate-genericx86-64-ext
  • Build:
  • Post Sync sstate cache:
  • Post Sync downloads cache:
  • Total:

@klutchell
Copy link
Contributor Author

Superseded by #443

@klutchell klutchell closed this Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant