Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CI times #903

Open
Mirko-von-Leipzig opened this issue Oct 1, 2024 · 1 comment
Open

Improve CI times #903

Mirko-von-Leipzig opened this issue Oct 1, 2024 · 1 comment

Comments

@Mirko-von-Leipzig
Copy link
Contributor

Mirko-von-Leipzig commented Oct 1, 2024

CI test job times are becoming exceedingly poor, exceeding 10min and often 30min. Other jobs are all < 3 minutes.

Here are some recent workflow runs:

which are both running against the next branch with caching enabled and cache is hit. The bulk of the time is taken to compile internal crates again:

   Compiling miden-objects v0.6.0 (/home/runner/work/miden-base/miden-base/objects)
   Compiling miden-lib v0.6.0 (/home/runner/work/miden-base/miden-base/miden-lib)
   Compiling miden-tx v0.6.0 (/home/runner/work/miden-base/miden-base/miden-tx)
   Compiling miden-bench-tx v0.1.0 (/home/runner/work/miden-base/miden-base/bench-tx)
    Finished `test-release` profile [optimized + debuginfo] target(s) in 10m 05s

which occurs 3 times within the job. So the vast majority of the time is compiling despite the cache working.

What's interesting is that without the cache these multiple compilations don't occur. This CI run for #901 onto main has a full compile time of ~10minutes which it only pays once. The other two steps for some reason don't require a recompile. Compilation should be a once off; step 1 is build, and 2 & 3 are different test invocations. So it appears that this concept of build once, run twice is no longer working on next.

This probably indicates a change in workflow, features or dependencies between main and next which causes this triple compile.

Summary of initial findings:

  • next branch has some difference to main which causes it to recompile between steps (probably, this is an assumption based on the slim evidence above).
  • Cache is being hit on next but its not the dependencies which take long to compile. Its our own crates which take 10 minutes and are set to release which takes even longer. Essentially a full compile is 10 minutes which is roughly the same with or without dependencies precompiled.

Some initial thoughts:

  • Figure out why main and next recompile differently between steps,
  • Find different optimisation level which takes less time,
  • Investigate what is taking so long to compile in our crates - it might very well be link time which could a simple matter of using mold.
  • Consider moving proof related tests into a separate job which might run on a schedule instead of on each PR
@Mirko-von-Leipzig
Copy link
Contributor Author

The actual proof tests take 3 minutes to run. The non-proof tests are < 10s.

So despite the proof tests being slow, tests are only like 5% of the current time; the rest is all compilation.

On main this is more like 30% test, 70% build which is still not great but is better at least.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant