Fixed Flaky test of JsonSimpleStepTest #1886
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for the PR!
|
deekshacheruku
force-pushed
the
fix-flaky-json-simple-test
branch
from
54ec66b to
1c4329e
Compare
|
@nedtwigg I have removed the changelog entry. Can you please elaborate on |
deekshacheruku
force-pushed
the
fix-flaky-json-simple-test
branch
from
1c4329e to
05496ef
Compare
|
@nedtwigg I fixed the formatting issue! Let me know if there is anything else left. Thanks! |
|
@nedtwigg There are some failing tests, Not sure why. Can you please have a look? |
nedtwigg
reviewed
Nov 20, 2023
| @@ -58,14 +59,14 @@ public static StepHarness forFormatter(Formatter formatter) { | |||
| /** Asserts that the given element is transformed as expected, and that the result is idempotent. */ | |||
| public StepHarness test(String before, String after) { | |||
| String actual = formatter.compute(LineEnding.toUnix(before), new File("")); | |||
| assertEquals(after, actual, "Step application failed"); | |||
There was a problem hiding this comment.
This gets used for all kinds of tests, not only json tests. Also, it should be exactly the same as the snapshot, if it isn't, there's a bug in the formatter.
|
I haven't observed flakiness in the tests in question, and the fix is counter to the point of this library - exact formatting. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes the flaky tests handlesComplexNestedObject, handlesNestedObject, handlesObjectWithNull inside JsonSimpleStepTest class
Root Cause
The tests has been reported as flaky when ran against the NonDex tool. The tests failed because it was trying to compare the two Json strings, where the JSON library internally uses HashMap. The HashMap in Java does not store the inserted order of keys and their values. As a result, when the expected Json string (hard-coded) is compared with the actual one, it failed.
Fix
The
assertJsonEqualsmethod from the JsonUnit library is used to verify that two JSON entities are structurally and value-wise equivalent in unit tests, ignoring the order of elements in objects. If the actual JSON does not match the expected JSON, the method throws anAssertionError.Dependencies
The fix adds two Java dependencies: JsonUnit for flexible JSON assertions in tests, ensuring structural and value equivalence regardless of element order, and Jackson Databind for efficient parsing and generation of JSON data to and from Java objects. JsonUnit's assertJsonEquals method is specifically used to fix flaky tests by comparing JSON objects without considering the order of elements.
Java: openjdk version "11.0.20.1"
Module build - Successful
Command used -
./gradlew build -x testRegular test - Successful
Command used -
./gradlew --info testlib:test --tests=<test above>NonDex test - Failed
Command used -
./gradlew --info testlib:nondexTest --tests=<test above>NonDex test passed after the fix.
I am open to any other methods of fixing the issues.
=> I've experimented with ObjectMapper and JsonNode, which require additional dependencies and necessitate handling JsonProcessingException, significantly increasing the codebase due to the added exception management.