-
-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: fix unit test flakiness on MjpegScreenshotTest #678
Conversation
I realized a bit too late that the tests I added were flaky when re-running the entire test suite: ` ./gradlew testServerDebugUnitTest --rerun-tasks` One of the tests would randomly (roughly 50% of the runs) fail with: ``` MjpegScreenshotTest > shouldNotBlockOnUnInitializedClient FAILED java.lang.AssertionError: expected:<200> but was:<-1> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:633) at io.appium.uiautomator2.server.mjpeg.MjpegScreenshotTest.shouldNotBlockOnUnInitializedClient(MjpegScreenshotTest.java:101) ``` After trying to understand what this meant: I realized I was not sending proper HTTP response, just some raw data on the socket. It's even worse than that as we send it as streaming: the client doesn't really get an ending response. There is no need to test that we can read a streaming response: just tweaking the mock data to send the proper header was enough to make the test reliable
the PR prefix should rather be |
Changed the PR prefix |
GH CI is slow, so it might (and usually does) affect the result. |
Done 👍 |
🎉 This PR is included in version 7.1.10 🎉 The release is available on: Your semantic-release bot 📦🚀 |
I realized a bit too late that the tests I added in #676 were flaky when re-running the entire test suite:
./gradlew testServerDebugUnitTest --rerun-tasks
One of the tests would randomly (roughly 50% of the runs) fail with:
I realized I was not sending a proper HTTP response, just some raw data on the socket (the HTTP header is added in getScreenshot() method, which was mocked)
As we send this data as streaming: the HTTP client doesn't really know how to interpret this data.
It was surprising that it worked in most of the test runs.
There is no need to test that we can really read this streaming response: just tweaking the mock data to send the proper header was enough to make the test reliable and see the response as 200.
For the record: adding a
Thread.sleep(1000)
after starting the serverThread made the test pass in the CI, but it's not going to be 100% reliable. Easier to skip if for now