Store the original test, check and subresult outcome in results

docs/releases.rst

@@ -40,6 +40,13 @@ reference, either branch, tag, git-describe, or if all fails the commit hash.
 You may encounter this in the verbose log of ``tmt tests show`` or plan/test
 imports.
 
+:ref:`Result specification</spec/plans/results>` now defines
+``original-result`` key holding the original outcome of a test, subtest
+or test checks. The effective outcome, stored in ``result`` key, is
+computed from the original outcome, and it is affected by inputs like
+:ref:`test result interpretation</spec/tests/result>` or
+:ref:`test checks</spec/tests/check>`.
+
 In verbose mode, the ``discover`` step now prints information
 about the beakerlib libraries which were fetched for the test
 execution. Use ``tmt run discover -vvv`` to see the details.

spec/plans/results.fmf

@@ -27,9 +27,12 @@ description: |
          name: /test/one
          path: /
 
-       # String, outcome of the test execution.
+       # String, the effective outcome of the test execution.
        result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
 
+       # String, the original outcome of the test execution.
+       original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
+
        # String, optional comment to report with the result.
        note: "Things were great."
 
@@ -79,9 +82,12 @@ description: |
       # key. Fields have the same meaning as fields of the "parent" test result, but
       # relate to each check alone.
       check:
-          # String, outcome of the check execution.
+          # String, the effective outcome of the check execution.
         - result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
 
+          # String, the original outcome of the check execution.
+          original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
+
           # String, optional comment to report with the result.
           note: "Things were great."
 
@@ -114,9 +120,12 @@ description: |
           # String, name of the test phase.
         - name: First test case
 
-          # String, outcome of the phase execution.
+          # String, the effective outcome of the phase execution.
           result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
 
+          # String, the original outcome of the phase execution.
+          original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"
+
           # String, optional comment to report with the result.
           note: "Things were great."
 
@@ -210,6 +219,26 @@ description: |
     original test result in the custom result set to the value known
     to tmt.
 
+    The ``original-result`` key holds the outcome of a test, check or
+    subresult as reported by the test, check or subresult itself, without
+    any additional influence.
+    A reported outcome may be a subject to interpretation, and tmt may
+    consider additional inputs and eventually report different effective
+    outcome in the ``result`` key.
+
+    The following rules apply when it comes to test, test check and subresult
+    outcomes and their interpretation and effects:
+
+    * Test outcome is interpreted according to
+      :py:ref:`/spec/tests/result`. It is not yet planned for test check
+      outcomes be interpreted. It is not yet planned for subresult
+      outcomes to be interpreted.
+    * Test checks and subresults do not influence the effective test
+      outcome. It is however planned for test check outcomes to affect
+      the test outcome, see https://github.com/teemtee/tmt/issues/3185
+      for more details. It is not planned for subresults to affect
+      the test outcome, and we do not expect them to gain this effect.
+
     See also the complete `JSON schema`__.
 
     For custom results files in JSON format, the same rules and schema
@@ -224,6 +253,9 @@ description: |
     .. versionchanged:: 1.36
        phase results are now renamed to the ``subresult`` key.
 
+    .. versionchanged:: 1.37
+       original result of test, subtest and check is stored in ``original-result`` key.
+
     __ https://github.com/teemtee/tmt/blob/main/tmt/schemas/results.yaml
 
 example:

tmt/result.py

@@ -123,6 +123,11 @@ class BaseResult(SerializableContainer):
         serialize=lambda result: result.value,
         unserialize=ResultOutcome.from_spec
         )
+    original_result: ResultOutcome = field(
+        default=ResultOutcome.PASS,
+        serialize=lambda result: result.value,
+        unserialize=ResultOutcome.from_spec
+        )
     note: Optional[str] = None
     log: list[Path] = field(
         default_factory=cast(Callable[[], list[Path]], list),
@@ -133,6 +138,9 @@ class BaseResult(SerializableContainer):
     end_time: Optional[str] = None
     duration: Optional[str] = None
 
+    def __post_init__(self) -> None:
+        self.original_result = self.result
+
     def show(self) -> str:
         """ Return a nicely colored result with test name (and note) """
 

tmt/schemas/results.yaml

@@ -29,6 +29,9 @@ definitions:
       result:
         $ref: "/schemas/common#/definitions/result_outcome"
 
+      original-result:
+        $ref: "/schemas/common#/definitions/result_outcome"
+
       note:
         $ref: "/schemas/common#/definitions/result_note"
 
@@ -63,6 +66,9 @@ definitions:
       result:
         $ref: "/schemas/common#/definitions/result_outcome"
 
+      original-result:
+        $ref: "/schemas/common#/definitions/result_outcome"
+
       note:
         $ref: "/schemas/common#/definitions/result_note"
 
@@ -120,6 +126,9 @@ items:
     result:
       $ref: "/schemas/common#/definitions/result_outcome"
 
+    original-result:
+      $ref: "/schemas/common#/definitions/result_outcome"
+
     context:
       $ref: "/schemas/common#/definitions/context"