Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: fix unit test flakiness on MjpegScreenshotTest #678

Merged
merged 2 commits into from
Jan 7, 2025

Conversation

vbarbaresi
Copy link
Contributor

@vbarbaresi vbarbaresi commented Jan 2, 2025

I realized a bit too late that the tests I added in #676 were flaky when re-running the entire test suite:
./gradlew testServerDebugUnitTest --rerun-tasks

One of the tests would randomly (roughly 50% of the runs) fail with:

MjpegScreenshotTest > shouldNotBlockOnUnInitializedClient FAILED
    java.lang.AssertionError: expected:<200> but was:<-1>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at io.appium.uiautomator2.server.mjpeg.MjpegScreenshotTest.shouldNotBlockOnUnInitializedClient(MjpegScreenshotTest.java:101)

I realized I was not sending a proper HTTP response, just some raw data on the socket (the HTTP header is added in getScreenshot() method, which was mocked)
As we send this data as streaming: the HTTP client doesn't really know how to interpret this data.
It was surprising that it worked in most of the test runs.

There is no need to test that we can really read this streaming response: just tweaking the mock data to send the proper header was enough to make the test reliable and see the response as 200.

  • Skipped those tests in the CI using an environment variable to avoid flakiness issues
    For the record: adding a Thread.sleep(1000) after starting the serverThread made the test pass in the CI, but it's not going to be 100% reliable. Easier to skip if for now

I realized a bit too late that the tests I added were flaky when  re-running the entire test suite:
` ./gradlew testServerDebugUnitTest   --rerun-tasks`

One of the tests would randomly (roughly 50% of the runs) fail with:
```
MjpegScreenshotTest > shouldNotBlockOnUnInitializedClient FAILED
    java.lang.AssertionError: expected:<200> but was:<-1>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at io.appium.uiautomator2.server.mjpeg.MjpegScreenshotTest.shouldNotBlockOnUnInitializedClient(MjpegScreenshotTest.java:101)
```

After trying to understand what this meant: I realized I was not sending proper HTTP response, just some raw data on the socket.
It's even worse than that as we send it as streaming: the client doesn't really get an ending response.

There is no need to test that we can read a streaming response: just tweaking the mock data to send the proper header was enough to make the test reliable
@mykola-mokhnach
Copy link
Contributor

mykola-mokhnach commented Jan 2, 2025

the PR prefix should rather be test:. It should not bump the module version as it does not introduce any changes to the production code

@vbarbaresi vbarbaresi changed the title chore: fix unit test flakiness on MjpegScreenshotTest test: fix unit test flakiness on MjpegScreenshotTest Jan 2, 2025
@vbarbaresi
Copy link
Contributor Author

Changed the PR prefix
Though it seems CI has failed 🤔 It was 100% reliable on my machine this time

@mykola-mokhnach
Copy link
Contributor

Changed the PR prefix Though it seems CI has failed 🤔 It was 100% reliable on my machine this time

GH CI is slow, so it might (and usually does) affect the result.
You might skip the test conditionally if it is running in the CI env by setting the CI environment variable in the workflow and then checking for its presence in the test.

@vbarbaresi
Copy link
Contributor Author

You might skip the test conditionally if it is running in the CI env by setting the CI environment variable in the workflow and then checking for its presence in the test.

Done 👍
I also updated the PR description: test is now conditionally skipped in the CI.

@mykola-mokhnach mykola-mokhnach merged commit 70aba96 into appium:master Jan 7, 2025
11 checks passed
Copy link

github-actions bot commented Jan 7, 2025

🎉 This PR is included in version 7.1.10 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants