Rename 'Fail' to 'Expected Failures' in test output #206
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Background ========== The fuzzy tester has a bit of added functionality not present in many testing tools: a test can be expected to pass, or expected to fail. This is because a geocoder is a complex machine and we can't always make a certain test pass. We often use this when creating tests for new functionality we haven't yet implemented. Marking the tests as passing then becomes a bit of a celebration of a job completed :) On the other hand it might be because there's a known regression that we can't currently fix, and have deemed acceptable. Maybe it came along with an improvement in another area that is considered "worth it", or we just plain don't know how to fix it yet. This leads to 4 possible outcomes for any individual test. We currently refer to them as follows: **Passed:** the test was expected to pass and it did pass **Improvement:** the test was not expected to pass, but it passed (usually as a result of a bug fix or new feature) **Fail**: The test was expected to fail, and it failed **Regression**: The test was expected to pass, but it failed. This is usually a problem and we look at these closely The Problem =========== Even for those of us familiar with the fuzzy tester, remembering if it's 'regressions' or 'fails' that represent a _new_, unexpected failure, can sometimes be a challenge. When viewing test output directly, the difference in color can be helpful, but depending on how the tests are run, colors in the output might not be available. What's changed ============== This PR replaces 'Fails' with 'Expected Failures' so that it's very clear which of the two types of failures are which.
missinglink
approved these changes
Mar 15, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
The fuzzy tester has a bit of added functionality not present in many testing tools: a test can be expected to pass, or expected to fail.
This is because a geocoder is a complex machine and we can't always make a certain test pass.
We often use this when creating tests for new functionality we haven't yet implemented. Marking the tests as passing then becomes a bit of a celebration of a job completed :)
On the other hand it might be because there's a known regression that we can't currently fix, and have deemed acceptable. Maybe it came along with an improvement in another area that is considered "worth it", or we just plain don't know how to fix it yet.
This leads to 4 possible outcomes for any individual test. We currently refer to them as follows:
Passed: the test was expected to pass and it did pass
Improvement: the test was not expected to pass, but it passed (usually as a result of a bug fix or new feature)
Fail: The test was expected to fail, and it failed
Regression: The test was expected to pass, but it failed. This is usually a problem and we look at these closely
The Problem
Even for those of us familiar with the fuzzy tester, remembering if it's 'regressions' or 'fails' that represent a new, unexpected failure, can sometimes be a challenge.
When viewing test output directly, the difference in color can be helpful, but depending on how the tests are run, colors in the output might not be available.
What's changed
This PR replaces 'Fails' with 'Expected Failures' so that it's very clear which of the two types of failures are which.
Before:
After:
