Reliable Expectedness

[sopos]

As we all know the tests are sometime failing in an expected way in certain environments. It is usually not a good idea to hard-code the expected stuff in the test itself.
It is much better to have a post-processing of the test results at the results gather phase.

While the final decision about the result correctness is always on the QE, the tooling should offer some tools to make it easier.
For example, beakerlib currently generates the asserts and phases fingerprints. That is pretty simply solution for comparing either two test runs or one against an expected value. More sophisticated approach may be a pattern matching on top of the test's output.

In either case, the fmf format with its adjust feature offers a nice way to specify an expected fingeprint / pattern for different environments. It is also quite natural to expect such feature in the report step of the tmt.

The test result should be whoever flipped from pass to fail of vice versa. It should rather use a expected boolean just to indicate if the result id expected, based on the rule or not so the QE can see in while review the results.

[sopos]

@t184256 may have some relevant notes to this

[t184256]

Previous discussion on this topic: #506