Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Disable unreliable test #11105

Merged
merged 1 commit into from
May 21, 2023
Merged

Conversation

Joibel
Copy link
Member

@Joibel Joibel commented May 19, 2023

Motivation

The test TestSynchronizationForPendingShuttingdownWfs/PendingShuttingdownStoppingWf regularly fails in github actions unit test stage.

This should probably be looked at, but for now stop needing quite so many retries to push through simple changes.

Modifications

Commented out test with a note on reason why.

@Joibel Joibel force-pushed the unreliable-test branch 2 times, most recently from 5cf0d9d to ea06304 Compare May 19, 2023 13:32
@Joibel Joibel marked this pull request as ready for review May 19, 2023 14:10
Copy link
Member

@terrytangyuan terrytangyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are other ones too. We might want to fix each one of them. #10807

@terrytangyuan
Copy link
Member

Can you do a go test -v -run TestSynchronizationForPendingShuttingdownWfs ./workflow/controller -count=5 to see if you can reproduce it?

@Joibel
Copy link
Member Author

Joibel commented May 19, 2023

Can you do a go test -v -run TestSynchronizationForPendingShuttingdownWfs ./workflow/controller -count=5 to see if you can reproduce it?

I can't reproduce it. Did it 25 times, and works every time for me.
(Arch Linux/x86_64/inside the devcontainer from this repo).

This specific test failure gets me much more frequently than the other ones. Or at least it fails fast, so if another one would fail, this one happens first.

@Joibel
Copy link
Member Author

Joibel commented May 19, 2023

There are other ones too. We might want to fix each one of them. #10807

Are you only interested in fixed tests, or is disabling an option with an issue to investigate and re-enable?

@terrytangyuan
Copy link
Member

Are you only interested in fixed tests, or is disabling an option with an issue to investigate and re-enable?

A bit concerned if it's related to core functionalities. Perhaps we can skip it only in GitHub Actions but enable it locally.

})
// This test regularly fails in Github Actions CI
// t.Run("PendingShuttingdownStoppingWf", func(t *testing.T) {
// // Create and acquire the lock for the first workflow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of commenting out, it's probably better to do:

t.Run("PendingShuttingdownStoppingWf", func(t *testing.T) {
  t.Skip("This test regularly fails in Github Actions CI")
  ...
}

I don't know if Github Actions also has any environment variables that it sets, but if so, you could make it a conditional based on that environment variable being there

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just skipped in CI now based on GITHUB_ACTIONS=true.

@Joibel
Copy link
Member Author

Joibel commented May 21, 2023

Are you only interested in fixed tests, or is disabling an option with an issue to investigate and re-enable?

A bit concerned if it's related to core functionalities. Perhaps we can skip it only in GitHub Actions but enable it locally.

I've done that now.

The test PendingShuttingdownStoppingWf regularly fails in github
actions unit test stage.

This should probably be looked at, but for now stop needing quite
so many retries to push through simple changes.

Signed-off-by: Alan Clucas <[email protected]>
@terrytangyuan terrytangyuan merged commit 9a2bb5e into argoproj:master May 21, 2023
terrytangyuan pushed a commit that referenced this pull request May 25, 2023
@Joibel Joibel deleted the unreliable-test branch May 28, 2023 18:24
JPZ13 pushed a commit to pipekit/argo-workflows that referenced this pull request Jul 4, 2023
@agilgur5 agilgur5 added area/build Build or GithubAction/CI issues area/mutex-semaphore labels Sep 26, 2023
dpadhiar pushed a commit to dpadhiar/argo-workflows that referenced this pull request May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build Build or GithubAction/CI issues area/mutex-semaphore
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants