Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: lower time for fuzz/race tests #15141

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

erikburt
Copy link
Collaborator

@erikburt erikburt commented Nov 6, 2024

Lowers the runtime of CI Core fuzz/race tests during executions triggered by PRs.

Changes

Race Tests:

  • Lower timeouts to 15s, and count to 5 (on PRs only, still at 10m/50 on scheduled runs)
  • Add build cache

Fuzz Tests:

  • Lower fuzz timeout to 4 minutes on PRs, 10m on scheduled runs
  • Calculating --seconds parameter passed to python fuzz script, based on FUZZ_TIMEOUT_MINUTES
  • Better calculation of time elapsed, and individual fuzzer runtimes in python script

Testing

https://github.com/smartcontractkit/chainlink/actions/runs/11734083428?pr=15141

  • Core Integ Tests: 2m25s
  • Deployment Tests: 2m32s
  • Race Tests: 4m39s
  • Core Tests: 5m34s
  • Fuzz Tests: 5m55s

In best case scenario, long pole is now "Clean Go Tidy & Generate"


RE-3205

@erikburt erikburt self-assigned this Nov 6, 2024
@erikburt erikburt requested review from a team as code owners November 6, 2024 21:05
chainchad
chainchad previously approved these changes Nov 6, 2024
momentmaker
momentmaker previously approved these changes Nov 6, 2024
momentmaker
momentmaker previously approved these changes Nov 6, 2024
jmank88
jmank88 previously approved these changes Nov 6, 2024
@erikburt erikburt dismissed stale reviews from jmank88 and momentmaker via c040274 November 6, 2024 23:40
@smartcontractkit smartcontractkit deleted a comment from github-actions bot Nov 6, 2024
@smartcontractkit smartcontractkit deleted a comment from github-actions bot Nov 6, 2024
momentmaker
momentmaker previously approved these changes Nov 7, 2024
@erikburt erikburt marked this pull request as draft November 7, 2024 00:18
@erikburt erikburt force-pushed the chore/lower-race-fuzz-tests branch 2 times, most recently from 9934f37 to 9c9711c Compare November 7, 2024 18:59
@erikburt erikburt marked this pull request as ready for review November 7, 2024 21:07
jmank88
jmank88 previously approved these changes Nov 7, 2024
@jmank88
Copy link
Contributor

jmank88 commented Nov 7, 2024

Looks good, but could be a little faster? 🤷
image

@erikburt
Copy link
Collaborator Author

erikburt commented Nov 7, 2024

Looks good, but could be a little faster? 🤷

Ok I've lowered them even further. I was having trouble lowering race tests at all, seems no matter what it sits around 8 minutes runtime. I'm thinking it might be build time that is the majority of the runtime.

@jmank88
Copy link
Contributor

jmank88 commented Nov 7, 2024

Looks good, but could be a little faster? 🤷

Ok I've lowered them even further. I was having trouble lowering race tests at all, seems no matter what it sits around 8 minutes runtime. I'm thinking it might be build time that is the majority of the runtime.

You could try dropping the -json flag from the race tests. I don't know why that is enabled, but it creates extra noise by enabling -v. Might not matter though 🤷

@erikburt
Copy link
Collaborator Author

erikburt commented Nov 7, 2024

You could try dropping the -json flag from the race tests. I don't know why that is enabled, but it creates extra noise by enabling -v. Might not matter though 🤷

That seem to have helped by about 30 seconds. Down to 7.5min execution on the race test.

@erikburt
Copy link
Collaborator Author

erikburt commented Nov 8, 2024

Ok the build cache does help - down to 5m10s...

https://github.com/smartcontractkit/chainlink/actions/runs/11732567357?pr=15141

@erikburt erikburt requested a review from jmank88 November 8, 2024 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants