Online tests results (e.g.: Report Portal)

[roniezr]

Can we upload the results to RP immediately upon completion of an individual test, instead of as it works today,
When all test results are uploaded only at the end of a plan?

Please see also:

1. https://github.com/teemtee/tmt/pull/2088/files#r1196173037

[happz]

From the discussion in the PR mentioned above:

I see you are uploading the results after they already finished.
Can we upload the results of each test separately immediately when it is finished, in this case, we will get online results,
E.g.:

Open RP branch on TMT start
loop over TMT tests:
2. Open the RP test before executing the test
3. Close the RP test after the test ended
Close RP launch
Note:
In any case, we must keep and provide the JUnit results, which include a full report

My response was:

Not at this moment, the execute and report steps are isolated, and there are no before testing/before test/after test hooks for report plugins to use. It is an interesting idea though, could you file an issue for it?

Adding support for such events & hooks should be made in a general fashion, for other plugins to use when needed, and therefore in a stand-alone PR (or several PRs, this might get too complicated to fit into a single patch).

Plugins are not often involved in the work of their predecessors, there's no established schema for plugins registering callbacks for others to call at important points of the workflow. I don't know how common the use case could be, some time ago I personally was in need of pytest HTML report being updated after each test (long-running tests), but if we decide to support this, we should do it properly, without ad-hoc constructs for a single plugin.

[kkaarreell]

I was experimenting with this approach and while it may seem handy, to me the benefit was not that big because RP I didn't want to access a launch multiple times (while it was still running) to do a review. Also, in RP I am missing the information how many tests will be executed because even though you can "initialize" all tests at the beginning the RP UI would display only finished tests (and you won't be able to see the number of tests in progress). With that, I am not saying that this "feature" won't be useful, I am just trying to point out the ups and downs should be properly evaluated as the implementation won't be definitely cheap.