Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(e2e): parallel setup for test envs #11917

Merged
merged 33 commits into from
Oct 31, 2024
Merged

Conversation

jakubdyszkiewicz
Copy link
Contributor

@jakubdyszkiewicz jakubdyszkiewicz commented Oct 29, 2024

Motivation

Speed up CI and test execution.

Locally

multizone tests locally on M1 Max

Ginkgo ran 1 suite in 4m32.566718875s
Test Suite Passed
/Applications/Xcode.app/Contents/Developer/usr/bin/make test/e2e/k8s/stop
KIND_CLUSTER_NAME=kuma-1 /Applications/Xcode.app/Contents/Developer/usr/bin/make k3d/stop
KIND_CLUSTER_NAME=kuma-2 /Applications/Xcode.app/Contents/Developer/usr/bin/make k3d/stop
INFO[0000] Deleting cluster 'kuma-2'
INFO[0000] Deleting cluster 'kuma-1'
INFO[0001] Deleting 1 attached volumes...
INFO[0001] Removing cluster details from default kubeconfig...
INFO[0001] Removing standalone kubeconfig file (if there is one)...
INFO[0001] Successfully deleted cluster kuma-2!
INFO[0002] Deleting 1 attached volumes...
INFO[0002] Removing cluster details from default kubeconfig...
INFO[0002] Removing standalone kubeconfig file (if there is one)...
INFO[0002] Successfully deleted cluster kuma-1!

________________________________________________________
Executed in  383.97 secs    fish           external
   usr time  238.16 secs    0.10 millis  238.16 secs
   sys time   83.86 secs    1.19 millis   83.86 secs

6m23s

vs on master

Ginkgo ran 1 suite in 5m34.492306167s
Test Suite Passed
/Applications/Xcode.app/Contents/Developer/usr/bin/make test/e2e/k8s/stop
KIND_CLUSTER_NAME=kuma-1 /Applications/Xcode.app/Contents/Developer/usr/bin/make k3d/stop
KIND_CLUSTER_NAME=kuma-2 /Applications/Xcode.app/Contents/Developer/usr/bin/make k3d/stop
INFO[0000] Deleting cluster 'kuma-1'
INFO[0000] Deleting cluster 'kuma-2'
INFO[0001] Deleting 1 attached volumes...
INFO[0001] Removing cluster details from default kubeconfig...
INFO[0001] Removing standalone kubeconfig file (if there is one)...
INFO[0001] Successfully deleted cluster kuma-1!
INFO[0001] Deleting 1 attached volumes...
INFO[0001] Removing cluster details from default kubeconfig...
INFO[0001] Removing standalone kubeconfig file (if there is one)...
INFO[0001] Successfully deleted cluster kuma-2!

________________________________________________________
Executed in  439.51 secs    fish           external
   usr time  252.61 secs  106.00 micros  252.61 secs
   sys time   84.38 secs  837.00 micros   84.38 secs

7m19s

CI

this PR

image

master

image

Implementation information

We can improve setup of E2E tests by running it in parallel. For example, if we deploy multiple deployments on kube (demo-client, test-server etc.) we can do this in parallel.
On top of it, in multizone, we can setup all zones in parallel.
This help with

  1. Quicker execution of a single test, meaning that it's easier to iterate on a test
  2. Quicker execution of all tests

I did not implement Parallel in universal e2e_env because this PR is already quite big. Universal e2e_env is not a bottleneck, and deployments are usually quick there.

@jakubdyszkiewicz jakubdyszkiewicz removed the ci/run-full-matrix PR: Runs all possible e2e test combination (expensive use carefully) label Oct 31, 2024
Signed-off-by: Jakub Dyszkiewicz <[email protected]>
@jakubdyszkiewicz jakubdyszkiewicz marked this pull request as ready for review October 31, 2024 08:43
@jakubdyszkiewicz jakubdyszkiewicz requested a review from a team as a code owner October 31, 2024 08:43
@jakubdyszkiewicz jakubdyszkiewicz requested review from lobkovilya and lukidzi and removed request for a team October 31, 2024 08:43
@jakubdyszkiewicz
Copy link
Contributor Author

flakes summary

https://github.com/kumahq/kuma/actions/runs/11606560457/job/32320335697 k3d did not start
https://github.com/kumahq/kuma/actions/runs/11587490031/job/32260402389 The hosted runner: GitHub Actions 2 lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
https://github.com/kumahq/kuma/actions/runs/11585733498/job/32255361819 could not upload artifact
https://github.com/kumahq/kuma/actions/runs/11582487071/job/32246731463 fail on make build gzip: stdin: not in gzip format (network error?)

Copy link
Contributor

@lukidzi lukidzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvment 🎉

Signed-off-by: Jakub Dyszkiewicz <[email protected]>
@jakubdyszkiewicz jakubdyszkiewicz merged commit b98468f into master Oct 31, 2024
13 checks passed
@jakubdyszkiewicz jakubdyszkiewicz deleted the test/e2e-parallel-setup branch October 31, 2024 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants