Multihost Support

[psss]

This discussion is dedicated for announcing new multihost support topics, issues and pull requests. If you are interested in the area of detailed hardware specification you might want to subscribe to this discussion to get notified about the new stuff available for feedback.

[kkaarreell]

I would like to present my use case.
In keylime we are using tests also as setup tasks, e.g. to configure swtpm or kernel parameters (example).
We expect to use similar approach also for multihost tests, i.e. we would be mixing singlehost and multihost tests together within a plan.

To easy things on tmt side I think it would help to have separate configs in discover for singlehost and multihost tests.
Example:

discover:
  - name: Server setup
    where:
      - server
    test:
      - /setup/install_upstream_keylime
  - name: Client setup
    where:
      - client
    test:
      - /setup/configure_tpm_emulator
      - /setup/install_upstream_keylime
  - name: Multihost test
    where:
      - client
      - server
    test:
      - /Multihost/basic-attestation

Please note that for "Setup" parts we actually do not need any syncing, however for the "Multihost test" part we need to have server and client synced. Eventually, if there would be more multihost tests present in a plan we would like to have them synced for each test.
Finally, we assume one test could be listed multiple times in a plan and therefore we need sync mechanism to be able to handle that correctly.

One additional note on syncing. On the mtg there was an example:

        discover:
            how: shell
            tests:
              - name: run-the-client-code
                test: client.py
                where: client
              - name: run-the-server-code
                test: server.py
                where: server

In this example the need of syncing is not obvious. In fact, such a need should be stated explicitly. I think that the default "sync" behavior could assume that tests with identical names should be synced but that won't apply to shell example above. I think one could also "group" items that should be synced.
E.g.

        discover:
            how: shell
            tests:
              - name: run-the-client-code
                test: client.py
                where: client
                sync: group1
              - name: run-the-server-code
                test: server.py
                where: server
                sync: group1

This idea could be extended further where different "groups" can be defined first and then later referenced by specific tests or steps, overriding a global plan default.

[sopos]

Currently I use just the simplest scenario we use in beaker for ages, i.e. 1:1 test sets synchronized as each.

However I can easily imagine following semi-multihost setup for clevis testing:

• spin up a SUT for tang server instance
• spin up a SUT for clevis where singlehost tests run against the tang server

The thing is that the clevis tests may start after and only after the tang server is running and responding, i.e. the clevis SUT start needs to be synced to the tang readiness.

Such scenario would be probably done by running the tang setup in the prepare step on one machine and tests in the execute step on the other machine where the execute step of the tang machine would be empty. The syncing would be done on the steps only.

Together with what I proposed in #969 (comment) it could look like:

discover:
  - how: fmf
    url: ...
    tests:
      - /setup/some_service
    name: SETUP
  - how: fmf
    url: ...
    filter: tag:!setup
    name: TEST
prepare:
    how: tmt
    what: SETUP
    where: server
execute:
    how: tmt
    what: TEST
    where: client

or

discover:
  - how: fmf
    url: ...
    tests:
      - /setup/some_service
    name: SETUP
    where: server
  - how: fmf
    url: ...
    filter: tag:!setup
    name: TEST
    where: client
prepare:
    how: tmt
    what: SETUP
execute:
    how: tmt
    what: TEST

Depending on where we actually want to decide the tests assignment to the specific SUT.

[sopos]

Another multihost approach would be the controller-base or orchestrated. The idea is simple, one SUT would run special tests which would control the others which would just wait for instructions. The setup would be similar to the previous use-case syncing on the steps:

discover:
  - how: fmf
    url: ...
    filter: tag:!setup
provision:
  - name: CONTROLLER
    ...
  - name: SUT1
    ...
  - name: SUT2
    ...
execute:
    how: tmt
    where: CONTROLLER

Assuming there would be metadata available on the SUTs so the running tests on the CONTROLLER would know hostnames / IPs of the other SUTs.

[adiosnb]

The sync specification and implementation discussion can be also found here: #1705

[The-Mule]

I am wondering - is there some implementation (of the original version of the specification) available for testing?

[psss]

Only the very simple scenario is working. But please note that support for the where key under the execute step is planned to be dropped in favor of #1606.

[lukaszachy]

Controller approach - What is the idea?
I understand it in the high level idea: tmt plan specifies it needs multiple machines, tmt run has machine provisioned and prepared (prepare config assigned by 'where' key) and in execute single machine is controller and decides what should run where.

But I fail to understand how one write it.

Questions:

• Will this support multiple tests within single plan?
• How tests on other machines will be executed?
• How and who will process the results / logs? Beakerlib needs special processing, in the future there should be additional checks (avc detection).
• Will there be libraries (and which languages) to help writing the controller script?

Possible use case:
Plan for chat application, 2 clients and 1 server. Two tests: 1) client A sends msg to client B, both online. 2) client B sends message client A while A is offline, later A connects and receives the message.

Should the above be a plan similar to bellow and tmt will capture just the output of test (/a-to-b) ran on the controller? So the responsibility to log / capture everything will be on this test? In other words: tmt in the execute step will only connect to the controller.

How about test dependencies? Controller will have different set than server/client. Should the burden of installing requires be moved to the prepare script? E.g. /a-to-b on controller needs beakerlib, on server foo-server and on client foo-client.
Where will that information (and how) be stored?

discover:
   how: fmf
   tests:
   - /a-to-b
   - /b-to-offline-a
   where: controller
prepare:
  - name: setup on server
    how: ansible
    where: server
  - name: setup on client
    how: ansible
    where: client
provision:
  - role: server
  - role: client
  - role: client
  - role: controller
 execute:
    how: tmt
    sync: start

[sopos]

Very good questions!

Questions:

* Will this support multiple tests within single plan?

Originally I was thinking of that as a test per plan, but it actually makes sense to make it possible to have more of them in one plan as you sketched up above.

* How tests on other machines will be executed?

There would be no tests executed on the other machines. There would be just the controler-based test executed. The rest would be on the test itself. It would be just server with enough data to be able to access and control the other SUTs.

* How and who will process the results / logs? 

The logs would be produced by the controller only. If there are some log on the SUTs, the test should gather them on its own.

  Beakerlib needs special processing, in the future there should be additional checks (avc detection).

This might a real issue. There would not be any place to hook an AVC test if there were not tests on the SUTs. We cannot expected the test owner will be able to cover all the necessary check which normally would run automatically by the framework.

* Will there be libraries (and which languages) to help writing the controller script?

Definitely it is a place to use libraries like https://github.com/beakerlib/ControlFlow/tree/master/SessionControl in case of beakerlib tests. Beakerlib might not be the best thing to use for such type of tests, thought.

How about test dependencies? Controller will have different set than server/client.
This is also an issue as the tests would be planned to controlled only. There might be some additional L1 metadata for this purpose

test_type: controller
require:
    controller:
      - beakerlib
    server:
      - apache
    client:
      - wget

That would, of course, mean that the L1 and L2 metadata must meach.

[kkaarreell]

I would like to follow up a bit on a topic discussed on yesterday's mtg, I mean the question of parallelism of discover configs.
Starting with Lukas's example above, how should we understand it?

discover:
   how: fmf
   tests:
   - /a-to-b
   - /b-to-offline-a
   where: controller
...
 execute:
    how: tmt
    sync: start

Discover says that those two tests should be run on controller. What does that mean for server and clients? Is there an assumption that these systems are blocked until tests running on controller are finished? Clearly we do not want to dispose those systems while tests are running but from where in multihost design is this behavior comes from?

It becomes even more unclear when multiple configs are present in a discover step (to simplify it I am using the other syntax with sync in the dicover step)

discover
  - name 1on1 peer tests
    how: fmf
    tests:
      - /test1
      - /test2
    where:
      - server
      - client
    sync: name
 - name controller tests
   how: fmf
   tests:
    - /test-A
    - /test-B
    where: controller
  sync: start

What a user would like to get is to have server and client systems synced at the beginning of each test from "peer tests" set and then sync ALL systems synced at the beginning of "controller tests" and last until the very end.

However, one could also understand this syntax differently. It could mean that "peer tests" are run just on client and server and since controller is not required here it can proceed with the other config and run "controller tests". This would clearly lead to undesired behavior. So the crucial question here is: Does where apply both to where tests are run and what systems are synced or are all provisioned systems always synced? In the example above we want/assume the 2nd to be true.

Just to present a use case for the 1st behavior (without controller role):

discover
  - name: server setup
    how: fmf
    tests:
      - /server/setup
    where:
      - server
 - name: client setup
   how: fmf
   tests:
    - /client/setup
    where: client
  - name: the actual test
    how: fmf
    tests:
      - /test1
    where:
      - server
      - client
    sync: start

Here we benefit from server and client configs being run in parallel.
And just to present it a bit differently:

discover
  - name: common setup and sync
    how: fmf
    tests:
      - /common/setup
    where:
      - server
      - client
    sync: finish   (not sure this is defined somewhere but I want here them to be synced when the test/setup ends)
  - name: server part of tested scenario
    how: fmf
    tests:
      - /server/part
    where:
      - server
 - name: client part of tested scenario
   how: fmf
   tests:
    - /client/part1
    - /client/part2
    - /client/part3
    where: client
  - name: log collection
    how: fmf
    tests:
      - /log/collection
    where:
      - server
      - client
    sync: start

which utilize the parallelism and also the fact that server and client tests may have different names. This could in fact become handy, e.g. when server part is doing some generic setup while clients when be performing different tasks against this setup. I could also add more discover configs changing the setup on the server and then running again various client tasks. This would allow me to test multiple scenarios in a single test plan.

I think that this behavior is even more important when how: shell is used instead.

Above I tried to present on examples why I believe it is crucial to define the expected behavior for the where step and discover with multiple configs, each possibly requesting different sync. I am not saying we should support every use case presented above, I am just trying to present that the actual behavior would influence the actual test case and test plan design.

And just to make it even more complicated. tmt could export the actual system role in a context and therefore discover could return different test set for client or server role even in the same discover config.. In fact, this could happen even without that assuming client and server have different system configuration and test relevancy takes system configuration into account. The point is that users knowing that could utilize it for generating different server and client test sets on purpose, within a single discover.

[sopos]

At the end the the mtg we actually hit the crucial question - are we gonna solve the in step syncing (task/test syncing) or just the step syncing? There are two separate things where the former one is much more complicated and may bring more confusion.

It was considered as a food for thoughts for the Xmas period.

May be we should focus on the simple scenarios, i.e. sync: start for the step sync and possibly sync: index for the in step sync which would cover the backwards wow-compatible workflow.

Those more complicated setups would be probably preferred to treat as the controller based approach.

[adiosnb]

+1 for the finish sync value

[adiosnb]

What if you add sync: start to the second discover config? Can this solve the problem int the last example?

[adiosnb]

I added extend example of multihost and sync usage into the PR#1706. Please have a look and any tips, how it can be improved.

[sopos]

however, considering it as s dicover "config" stuff is rather the in step kind of sync

[adiosnb]

The question from the last meeting discussion. In the example pasted below, which guests should be synchronized in the beggining? All guests or guests specified by where?

discover:
 - name: client part of tested scenario
   how: fmf
   tests:
    - /client/part1
    - /client/part2
    - /client/part3
    where: client
    sync: index
    # Question: Which host to sync?

  - name: server part of tested scenario
    how: fmf
    tests:
      - /server/part
    where:
      - server

[adiosnb]

I have also another question or proposal for the start sync method. What if it synchronizes all guests at the beginning of the step? In the case of multiple configs with start, all guests are synced together regardless where at the begging of different config parts. What do you think?

[adiosnb]

And the last topic for the discussion. What do you think about the implicit synchronization barrier between all step phases? I mean all provision steps for all guests start at once, then all prepare steps start at once, execute, report, etc.

Do you have any preferences about this?

[sopos]

This was the step sync we thought about initially, I guess. In particular the sync: start. While we kind-of agreed the default would be sync: none I would refuse this implicit syncing.

[kkaarreell]

I've spent some time thinking about synchronization yesterday and I think we should consider making things as simple as possible. We didn't really gather much real-life scenarios (it would be nice to have them gathered somewhere - I mean not links to existing tests but rather an outline how the tmt plan would look like in some proposed/abstract syntax) and it therefore seems to me that we may be over-engineering the syntax too much without having a real need for high flexibility.

I will be talking about this example

discover
  - name: server setup
    how: fmf
    tests:
      - /server/setup
    where:
      - server
 - name: client setup
   how: fmf
   tests:
    - /client/setup
    where: client
  - name: the actual test
    how: fmf
    tests:
      - /test1
    where:
      - server
      - client
    sync: start

My question is: Is there a real need to run subsequent phases (or steps) in parallel? For the example above it means, we we need to be able to run server setup and client setup together? I think it will make the implementation more complicated and I am not sure this use case is really necessary/justified. We should make things simple in tmt, delegating the complexity to tests and their custom sync frameworks.

So my proposal goes a bit against what is mentioned by @sopos above. I would propose that each phase (one named block in discover above) would be synced at the beginning for all provisioned systems independently on whether these systems are mentioned in where clause - this would be used only to define on which systems the actual command/test should be executed. But all systems will be synced and with the beginning of the next phase all systems with be synced again. I believe it will make implementation way easier. Also, it would reduce a risk that some systems would be asked to run two distinct tests/commands at the same time.
Clearly, we will loose something (running tests/commands on some systems in parallel to running other tests/commands on other systems) but I think it is worth it, at least for the initial implementation.

The above applies to phases, what remains is sync between individual tests. For that I think we could utilize options that have been already proposed (by name, index etc.).

[sopos]

According to the second picture above there are, at least the server setup and client setups.
Otherwise the first picture, where no parallelism on the config / phase level is allowed, seems fine to me as well.

[kkaarreell]

I wouldn't touch test synchronization yet.

Exactly. Our primary motivation for test synchronization was compatibility with beaker. But what beaker/restraint does is synchronization of tests with multihost type and such synchronization is be pretty simple to implement. E.g. using semaphores and following implementation in sync library (https://github.com/RedHat-SP-Security/keylime-tests/tree/main/Library/sync)
for server, client, storage being provisioned and having

- name: tests
  where: server, client

one (tmt) needs to schedule this:

1. for each multihost test build unique semaphore name (involving iteration counter for repetitive test executions), e.g. MULTI_SYNC_1_TEST_FOO_BAR
2. for server and client do syncSet MULTI_SYNC_1_TEST_FOO_BAR
3. for server and client do syncBlock MULTI_SYNC_1_TEST_FOO_BAR server client - since we are syncing servers where tmt would run a multi-host test

If we wouldn't be willing to build-in such execution into tmt directly we could insert explicite sync task prior each multi-host test or at the worst (as a temporary solution) update (migrate) every multihost test to do this kind of sync internally where tmt would only help to build that unique semaphore name (similarly to what restraint does with XTRA variable).

[kkaarreell]

According to the second picture above there are, at least the server setup and client setups. Otherwise the first picture, where no parallelism on the config / phase level is allowed, seems fine to me as well.

I see. TBH I do not get the second example, I do not understand why in-sync-with is there and necessary.

Also, the parallel client and server setup is incorrect, my proposed implementation would separate these two phases.

[happz]

Ok, I think we're still seeing something different, and this time it's definitely my fault, my diagrams do not describe what I wanted to convey. So much for clearing a table with a simple-to-understand picture :/ Let me start again.

[happz]

According to the second picture above there are, at least the server setup and client setups. Otherwise the first picture, where no parallelism on the config / phase level is allowed, seems fine to me as well.

I see. TBH I do not get the second example, I do not understand why in-sync-with is there and necessary.

Also, the parallel client and server setup is incorrect, my proposed implementation would separate these two phases.

Sorry, that was my mistake. I misunderstood & draw something that did not match my understanding anyway. I'm trying to start from scratch, I must admit I'm struggling to keep up with all the ideas around, and I'm trying to look for some concrete, materialized "common ground" I could grasp, understand and build upon. Again, sorry for the confusion. #1702 (comment)

[happz]

My take. I'd like to focus on some basics, multiple guests, and multiple phases to be run on each guest, with some parallelization.

Let's start with the following tmt configs, nothing fancy:

discover
  - name: server setup
    how: fmf
    tests:
      - /server/setup
    where:
      - server

 - name: client setup
   how: fmf
   tests:
      - /client/setup
    where:
      - client1
      - client2

  - name: the actual test
    how: fmf
    tests:
      - /test1
      - /test2
      - /test3
    where:
      - server
      - client1
      - client2

  - name: another test
    how: fmf
    tests:
      - /test4
      - /test5
    where:
      - client1
      - client2

  - name: yet another test
    how: fmf
    tests:
      - /test6
    where:
      - client1

I see the parallel aspect as being crucial and unavoidable. tmt can perform each guest queue after another, in a true sequential manner, but then I see much harder support for the server/client test scenarios: the actual test @ client2 might run hours after the actual test @ server (if phases run one by one, let's say from the left to right, from top to bottom). But:

• for runs where synchronization is not needed - e.g. testing a Koji build on multiple architectures -, tmt would finish sooner, no harm done.
• for runs with server/client tests, it would be much more easier for whatever user synchronization library the actual test might be using to synchronize its tests if the actual test phase would start at the same time, more or less, on server and both clients.

This seems to me like the very basic prototype we could land.

• No complicated implementation. If there are more phases to be run on the same guest, they would be run in sequence, just like today, just waiting for their counterpart in the same position in other queues. Provision guests, establish a queue for each of them, process discover phases, and distribute them into correct queues (a phase might be copied into several queues if needed).
• If the synchronization between phases is not needed, we're good.
• If the synchronization between phases is needed, split queues into parallel slots by barriers where different tmt threads of testing would wait for queues to reach before starting another phase.
• The order of phases and their distribution into queues is given by order and the order of phases in source fmf nodes.
• Both types of barriers are optional: both steps and phases lockstep can be enabled or disabled, whatever is the default. For the first example, none of them is needed, for the server/client, both might be necessary to guarantee a stable environment for the test-level synchronization.

My point was, AFAICT, this would be the very basic prototype supporting both trivial scenarios, "run plan on multiple hosts" and the very basic "run tests on server and client". User libraries can take care of synchronizing individual tests as needed, tmt would guarantee phases would begin at the same time & can export necessary variables, to describe the topology.

On top of this, we should be able to build more features. The bracketing @sopos suggested I accidentally illustrated in my previous diagrams. Or any other custom synchronization or parallelization modes, the core implementation would exist and would be already testable for the basic use cases.

Now, I'm adding yet another idea on top of the pile of ideas here, but I'd like to ask you to bear with me. I tried to add a diagram to illustrate the workflow, and I'd like to discuss various use cases or ideas & try to amend them with this proposal. If you something that's an obvious blocker, please, point it out.

[sopos]

All but the first set makes sense to me. Where tmt gets the information to run the server and client setups in parallel?

[happz]

@sopos by the mere fact both server and client setup phases are the first two phases discovered, and the do not overlap when it comes to guests. The server setup is placed in server queue first, then client setup is placed in both cluent queues, and becase there were no other phases put into client queues before, it makes them sitting in the first positions of all three queues.

No magic, just take phases one by one and stack them into correct queues. Minimal example & basic functionality.

I believe the explicit synchronization - run this phase with that phase - can be added, sure, absolutely. It would be just a matter of shaping those queues, the machine crunching phases wouldn’t care. But until we agree on how, why, when, we can maybe agree on & build an MVP which doesn’t close any door to advanced features.

[kkaarreell]

The problem I see with such parallelism is that it is in fact more tricky than it seems. At first we need to implement some logic that would be deciding which phases on which systems can be run or which systems can wait. E.g.

discover:
 - name: /phase/1
    where: server, client1
 - name: /phase/2
   where: server, client2
 - name: /phase/3
   where: client1, client2

What should be the rigth sequence?

1. /phase/1 on server and client1
2. /phase/2 on server and client2
3. /phase/3 on client1 and client2

or

1. /phase/1 on server and client1 + /phase/2 on client2
2. /phase/2 on server + /phase/3 on client1 and client2

I know you will say the first one but the problem is that it is not obvious at the first sight, especially if there would be more systems involved.

The other problem is that the question which systems should run a test script (where) is not the same as the question which systems should be synced and involved in test scenario. E.g.

discover:
 - name: ssh to client2 and create something
   where: server
 - name: ssh to client2 and destroy something
   where: client1

presents this discrepancy quite well I guess. Do we want to run both actions in parallel (race condition present) or one phase after the other?
From these reasons I believe that avoiding parallelism is justified, at least for the initial implementation. It should be easier to improve it later with more features than to implement those features properly from scratch.

[happz]

The problem I see with such parallelism is that it is in fact more tricky than it seems. At first we need to implement some logic that would be deciding which phases on which systems can be run or which systems can wait. E.g.

discover:
 - name: /phase/1
    where: server, client1
 - name: /phase/2
   where: server, client2
 - name: /phase/3
   where: client1, client2

What should be the rigth sequence?

1. /phase/1 on server and client1
2. /phase/2 on server and client2
3. /phase/3 on client1 and client2

or

1. /phase/1 on server and client1 + /phase/2 on client2
2. /phase/2 on server + /phase/3 on client1 and client2

The logic already exists: obey the user. The user expressed themselves already, and it's in the configuration: take one phase after another and perform them. /phase/1 on server and client1, plus /phase/2 on client2, because server is occupied. When done, move to the next.

If this is not what the user wanted, then we need to give him the tools to better express their needs, explicitly. For example:

discover:
  - name: /phase/1
    where: server, client1
  - name: /phase/2
    where: server, client2
  - name: /phase/3
    where: client1, client2
    in-sync-with: client1, client2

I took the liberty to add a new field, in-sync-with, to allow user to let tmt know it's their wish to run /phase/3 at the same time on client1 and client2. Given the order of phases, this will wait after /phase/2 finishes on client2, simulating your first suggested workflow. While server and client2 crunch /phase/2, client1 just sits and waits.

The other problem is that the question which systems should run a test script (where) is not the same as the question which systems should be synced and involved in test scenario. E.g.

discover:
 - name: ssh client2 'touch something'
   where: server
 - name: ssh client2 'destroy something'
   where: client1

presents this discrepancy quite well I guess. Do we want to run both actions in parallel (race condition present) or one phase after the other? From these reasons I believe that avoiding parallelism is justified, at least for the initial implementation.

Again, the default is to obey the user: one phase for server, one for client1. If they are not supposed to run at the same slot of guests' queues, the user must say so:

discover:
 - name: ssh client2 'touch something'
   where: server
 - name: ssh client2 'destroy something'
   where: client1
   after: ssh client2 'touch something' @ server

Again, new directive. Wrap it with sync, or not, not important, again just expresses what I wanted to achieve: the user tells this phase runs after that phase.

I'm afraid we're trying to teach tmt to guess a lot, and we're trying to make it all happen at once, all features supported and specified, and it makes things harder and harder and messier and messier. At the core, it's a simple matter of queues, items in those queues, and points where tmt should wait for something. I just wanted to propose a basic MVP which would add those queues, very simple schema to fill them with phases, and very basic set of places where tmt would wait and for what. That we could prototype, and, when the dust settles, the state machine and queues will be there no matter what we agree on for sync to be like. tmt already has logic and rules for performing phases, runs them in sequence as they come from fmf files - I'd say that's good enough. At least it's good enough for current single-host scenarios. For cases where it's not good enough, yeah, for those we definitely need sync or something like that, but until we agree on what it should behave like, I wanted to propose #1702 (comment) as something we might start with, as long as it wouldn't make anyone angry, break their existing workflow or close door to future enhancements.

The two scenarios you described here are definitely compatible with what I proposed as a starting point, and the functionality can be added once fully specified. I'm sure of that.

[kkaarreell]

I'm afraid we're trying to teach tmt to guess a lot

Exactly the opposite. No parallelism means safe default with no guessing involved. Richer syntax for optimized execution can be added later.

[kkaarreell]

I am not saying there is not a solution for presented questions/problems. I am saying that what you are proposing is in fact not a MVP since you are introducing additional complexity that is not necessary for multihost test execution but rather optimize resource utilization. Our current multihost tests scheduled by wow don't produce such complicated recipes. I can't see a reason why things you are proposing can't be added later.

[happz]

+1 for implementing simple config / phase processing, in sequence with parallelism just inside it, first, as described by @kkaarreell in #1702 (comment) and visualized by @happz in #1702 (reply in thread). That should cover our basic usecase of standard multihost tests which runs 1:1 on two machines. Later we can add some explicit tweaking of it.

Just be aware my diagram is actually incorrect, the actual test phases should not run at the same time, the client's one should run after the server's one completes.

[sopos]

Just be aware my diagram is actually incorrect, the actual test phases should not run at the same time, the client's one should run after the server's one completes.

Actually, it seems correct to me as it is defined within one config / phase so it should run in parallel and it should cover the MVP (restraint-like multihost execution) - the barrier is at the config / phase start.

[happz]

So there will be phases running in parallel on different guests...

[kkaarreell]

So there will be phases running in parallel on different guests...

Only identical phases, not distinct ones. By "no parallelism" I actually mean "distinct phases are not run in parallel". 😁

[happz]

First draft in #1790 - should be following the #1702 (reply in thread) schema, and I was able to run /tests/multihost/prepare/name with it:

$ tmt run --scratch provision prepare -vvv execute -vvv finish plan -n name
/var/tmp/tmt/run-076

/plan/prepare/name
    provision
        how: container
        name: server-one
        image: fedora
        distro: Fedora Linux 35 (Container Image)
    
        how: container
        name: server-two
        image: fedora
        distro: Fedora Linux 35 (Container Image)
    
        how: container
        name: client-one
        image: fedora
        distro: Fedora Linux 35 (Container Image)
    
        how: container
        name: client-two
        image: fedora
        distro: Fedora Linux 35 (Container Image)
    
        summary: 4 guests provisioned
    prepare
        queued: multihost on server-one (server), server-two (server), client-one (client) and client-two (client)
        queued: default-0 on server-one (server)
    
        queue tick #0: multihost on server-one (server), server-two (server), client-one (client) and client-two (client)
        queued: multihost on server-one (server)
        how: multihost
        summary: Setup guest for multihost testing
        queued: multihost on server-two (server)
        name: multihost
        how: multihost
        order: 10
        queued: multihost on client-one (client)
        summary: Setup guest for multihost testing
        guest: server-one
        name: multihost
        how: multihost
        queued: multihost on client-two (client)
        order: 10
        summary: Setup guest for multihost testing
        how: multihost
        guest: server-two
        name: multihost
        summary: Setup guest for multihost testing
        order: 10
        name: multihost
        guest: client-one
        order: 10
        guest: client-two
        finished: multihost on client-two (client)
        finished: multihost on server-two (server)
        finished: multihost on server-one (server)
        finished: multihost on client-one (client)

        queue tick #1: default-0 on server-one (server)
        queued: default-0 on server-one (server)
        how: shell
        order: 50
        guest: server-one
        where: server-one
        overview: 1 script found
        script: echo 'Server one'
            cmd: set -eo pipefail; echo 'Server one'
            out: Server one
        finished: default-0 on server-one (server)
    
        summary: 5 preparations applied
    execute
        queued: default-0 on server-one (server), server-two (server), client-one (client) and client-two (client)
    
        queue tick #0: default-0 on server-one (server), server-two (server), client-one (client) and client-two (client)
        queued: default-0 on server-one (server)
        how: tmt
        queued: default-0 on server-two (server)
        order: 50
        exit-first: False
        how: tmt
        order: 50
        queued: default-0 on client-one (client)
        exit-first: False
        how: tmt
        queued: default-0 on client-two (client)
        order: 50
        how: tmt
        exit-first: False
        order: 50
        exit-first: False
        finished: default-0 on client-two (client)
        finished: default-0 on server-two (server)
        finished: default-0 on server-one (server)
        finished: default-0 on client-one (client)
    
        summary: 0 tests executed
    finish
        container: stopped
        container: removed
        container: stopped
        container: removed
        container: stopped
        container: removed
        container: stopped
        container: removed
        summary: 0 tasks completed

total: no results found
$ 

• logging is a mess and needs more work & preparational patches building on recent logging changes,
• more complicated examples and tests will reveal more issues.

[kkaarreell]

Hello. With bits from #1790 and minor adjustment I was able to successfully run our keylime multihost test on 3 (and also 4) guests using 1minutetip. Details are in the PR itself. There are still some things to improve (like logging in the execution step) but in general it seems to work.
Great work @happz !!!

[happz]

A quick update on #1790:

• Keeping it rebased & up-to-date with the main development.
• Fixing tests, although not much left to fix.
• Blocked by phases fighting over shared directories, @psss promised to volunteer & take a look at a solution. This is related to one-test-multiple-times support discussed in Support executing one test multiple times #1614 (comment), although the conflict on the phase level is a bigger roadblock right now.
• Looking for more test cases and reviews.

[happz]

A quick update on #1790:

• Rebased after tests.yaml and results.yaml listification.
• Proposed Assign a data path and serial number to each test in discover #1876 and Include guest name in execute phase data paths #1877 to take care of blockers mentioned in Multihost Support #1702 (comment)

[happz]

A quick update on #1790:

State of things

• I'm done with basics, no obvious conflict in sight :)
  • Assign a data path and serial number to each test in discover #1876 has been merged into main, therefore each test should have its own dedicated data directory, i.e. no conflict on test level should a test be executed multiple times.
  • Include guest name in execute phase data paths #1877 has been merged into main, therefore each phase should have its own dedicated directiry, i.e. no conflict on phase level should a phase be executed on multiple guests at the same time.

Loose ends

• some changes will be extracted and submitted in their own PRs (exception printing done by tmt CLI, changed position of log prefixes), but that should be transparent for folks interested in this PR.
• better docs for tests.yaml and results.yaml - the current is usable, but recent changes (listification, serialnumber addition) showed there are some grey areas that deserve clarification.
• And, as always, looking for more test cases and reviews.

[kkaarreell]

@happz Hi, I tested the latest tmt COPR build on keylime multi-host scenario but it failed to run properly because the SERVERS variable is not exported anymore (at least it seems so). Is there a different way how a running test can find out about an assigned role?

Also, it seems to me that tests have not been truly executed in parallel on assigned hosts. At least I am not able to tell that from the log, the output looks weird and it is not really obvious which system was running which command. TBH I liked more parallel logging with role/system prefix.

[happz]

Hmm, it sounds like the Copr build is not related to #1790. I suppose this would be a misunderstanding - please, be aware that this change is not yet merged. Its components and various preparation patches are, as I’ve been curting it into smaller pieces, easier to digest, but the heart of the multi-host processing is not in main branch yet. The PR lacks proper testing, reviews and feedback in general to be merged. I’m not sure there’s a Copr build from the #1790 branch, I’m afraid you might need clone & checkout.

[kkaarreell]

I see.. Will give it another try tomorrow.

[kkaarreell]

I see.. Will give it another try tomorrow.

I have managed to run it now, following the plan from #1790 (comment) The multi-host test has passed but the setup task /setup/configure_tpm_emulator which was supposed to be executed only on agent was executed on all three systems.

[happz]

Another attempt pushed in. There are some good news:

• the missing feature, when in discover not being implemented - the code I pushed adds this functionality, so that's one.
• it all actually works, I added a trivial test for execute - it doesn't check the output yet, but log clearly shows the process, and it's doing fine.

But also some not that good:

• using provision on the command line will drop everything specified in the provision steps, including roles and names, therefore it can't be used when each provision config needs to have a different name and/or role.
• there's a bug in minute plugin, the plugin ignores roles.
• there's a bug in how where is handled internally, it may be multiple strings, yet internally it's a single string only.
• there's a bug in how guests are saved in results.yaml.

[happz]

/var/tmp/tmt/run-163
Found 1 plan.

/plans
    discover
        how: fmf
        name: server-setup
        directory: /home/happz/git/tmt/tests/multihost/complete/data
        tests: /tests/A
        how: fmf
        name: tests
        directory: /home/happz/git/tmt/tests/multihost/complete/data
        tests: /tests/B
        how: fmf
        name: teardown
        directory: /home/happz/git/tmt/tests/multihost/complete/data
        tests: /tests/C
        summary: 3 tests selected
            /server-setup/tests/A
            /tests/tests/B
            /teardown/tests/C
    provision
        how: container
        name: client-1
        image: fedora:36
        Check for container image 'fedora:36'.
        name: tmt-163-eJOUxSDI
        full name: client-1 (client)
        distro: Fedora Linux 36 (Container Image)
        kernel: 6.1.8-200.fc37.x86_64
    
        how: container
        name: client-2
        image: fedora:36
        Check for container image 'fedora:36'.
        name: tmt-163-xmQrDRCv
        full name: client-2 (client)
        distro: Fedora Linux 36 (Container Image)
        kernel: 6.1.8-200.fc37.x86_64
    
        how: container
        name: server
        image: fedora:36
        Check for container image 'fedora:36'.
        name: tmt-163-TIiWTKXu
        full name: server (server)
        distro: Fedora Linux 36 (Container Image)
        kernel: 6.1.8-200.fc37.x86_64
    
        summary: 3 guests provisioned
            client-1
            client-2
            server
    prepare
        queued: multihost on client-1 (client), client-2 (client) and server (server)
        queued: default-0 on client-1 (client) and client-2 (client)
        queued: default-1 on server (server)
        
        queue tick #0: multihost on client-1 (client), client-2 (client) and server (server)
[client-1 (client)]         queued
[client-1 (client)]         how: multihost
[client-2 (client)]         queued
[client-1 (client)]         summary: Setup guest for multihost testing
[client-2 (client)]         how: multihost
[client-1 (client)]         name: multihost
[client-2 (client)]         summary: Setup guest for multihost testing
[server (server)]         queued
[client-2 (client)]         name: multihost
[server (server)]         how: multihost
[client-1 (client)]         guest: client-1
[server (server)]         summary: Setup guest for multihost testing
[client-2 (client)]         guest: client-2
[server (server)]         name: multihost
[server (server)]         guest: server
[client-1 (client)]         finished
[client-2 (client)]         finished
[server (server)]         finished
        
        queue tick #1: default-0 on client-1 (client) and client-2 (client)
[client-1 (client)]         queued
[client-1 (client)]         how: shell
[client-2 (client)]         queued
[client-2 (client)]         how: shell
[client-1 (client)]         guest: client-1
[client-1 (client)]         where: client
[client-2 (client)]         guest: client-2
[client-1 (client)]         overview: 1 script found
[client-2 (client)]         where: client
[client-1 (client)]         script: /bin/true
[client-2 (client)]         overview: 1 script found
[client-2 (client)]         script: /bin/true
[client-1 (client)]             cmd: set -eo pipefail; /bin/true
[client-2 (client)]             cmd: set -eo pipefail; /bin/true
[client-1 (client)]         finished
[client-2 (client)]         finished
        
        queue tick #2: default-1 on server (server)
        how: shell
        guest: server
        where: server
        overview: 1 script found
        script: /bin/true
            cmd: set -eo pipefail; /bin/true
    
        summary: 6 preparations applied
    execute
        queued: server-setup on server (server)
        queued: tests on client-1 (client), client-2 (client) and server (server)
        queued: teardown on client-1 (client), client-2 (client) and server (server)
        
        queue tick #0: server-setup on server (server)
        how: tmt
        exit-first: False
            test: Dummy test A
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
                00:00:00 pass /server-setup/tests/A [1/1]
        
        queue tick #1: tests on client-1 (client), client-2 (client) and server (server)
[client-1 (client)]         queued
[client-1 (client)]         how: tmt
[client-2 (client)]         queued
[client-2 (client)]         how: tmt
[server (server)]         queued
[client-1 (client)]         exit-first: False
[server (server)]         how: tmt
[client-2 (client)]         exit-first: False
[server (server)]         exit-first: False
            test: Dummy test B
            test: Dummy test B
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
                00:00:00 pass /tests/tests/B [1/1]
            test: Dummy test B
                00:00:00 pass /tests/tests/B [1/1]
[client-2 (client)]         finished
[client-1 (client)]         finished
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
                00:00:00 pass /tests/tests/B [1/1]
[server (server)]         finished
        
        queue tick #2: teardown on client-1 (client), client-2 (client) and server (server)
[client-1 (client)]         queued
[client-1 (client)]         how: tmt
[client-2 (client)]         queued
[client-2 (client)]         how: tmt
[server (server)]         queued
[client-1 (client)]         exit-first: False
[server (server)]         how: tmt
[client-2 (client)]         exit-first: False
[server (server)]         exit-first: False
            test: Dummy test C
            test: Dummy test C
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
            test: Dummy test C
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
                00:00:00 pass /teardown/tests/C [1/1]
                00:00:00 pass /teardown/tests/C [1/1]
[client-1 (client)]         finished
                cmd: echo "In a test"
                                    echo "With data path $TMT_TEST_DATA"
                                    exit 0
[client-2 (client)]         finished
                00:00:00 pass /teardown/tests/C [1/1]
[server (server)]         finished
    
        summary: 7 tests executed
    report
        how: display
            pass /server-setup/tests/A
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/server/server-setup/tests/A-1/output.txt
            pass /tests/tests/B
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/client-1/tests/tests/B-2/output.txt
            pass /tests/tests/B
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/client-2/tests/tests/B-2/output.txt
            pass /tests/tests/B
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/server/tests/tests/B-2/output.txt
            pass /teardown/tests/C
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/client-1/teardown/tests/C-3/output.txt
            pass /teardown/tests/C
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/client-2/teardown/tests/C-3/output.txt
            pass /teardown/tests/C
                output.txt: /var/tmp/tmt/run-163/plans/execute/data/guest/server/teardown/tests/C-3/output.txt
        summary: 7 tests passed
    finish
        container: stopped
        container: removed
        container: stopped
        container: removed
        container: stopped
        container: removed
        summary: 0 tasks completed

total: 7 tests passed

[The-Mule]

I tested the previous packit build from #1790 (tmt-1.22.dev-1.20230321144934500513.pr1790.48.gc8f7556.fc37.noarch) and I confirm that it works very well for multihost tests using /distribution/sync library (alternative approach to using rhts-test-sync and rhts-test-block commands). There were a couple of issues mentioned already in the comments above:

• Roles are not exported, there is just SERVERS with all the node (easy fix I assume, easy workaround available).
• HTML report does not contain roles so it is hard to tell what run where (without looking into logs).

There is one more additional issue affecting /distribution/sync library. TMT does not use/export TEST variable as Beaker does and the library is using this to create synchronization files in shared storage space. This can be fixed easily in the library.

I used the following plan executed from the libreswan test dist-git:

$ cat multihost.fmf
summary: multihost testing experiment

discover:
  - name: server-test
    how: fmf
    where:
      - server
      - client
    test:
      - /Sanity/Multihost-options-replay-window
      - /Sanity/Multihost-ikev2-and-esp-over-TCP

provision:
  - name: server
    role: server
  - name: client
    role: client

prepare:
    - name: Add beaker-tasks repository to get beakerlib libraries
      where:
        - server
        - client
      how: shell
      script:
        - dnf config-manager --enable beaker-tasks

execute:
  how: tmt

And executed it by:

tmt run -vvv plans --name multihost discover prepare provision --how minute --image rhel-9.2 execute report finish

Another issue related to /distribution/sync beakerlib library is that beakerlib in tmt looks to github.com/beakerlib for libraries but it won't find sync even though it is there because it looks for it in distribution directory but that can be fixed easily by moving the library for now (until tmt will be able to find the libraries better).

One more thing, interactive execution should be blocked then using parallel execution.

[kkaarreell]

Another issue related to /distribution/sync beakerlib library is that beakerlib in tmt looks to github.com/beakerlib for libraries but it won't find sync even though it is there because it looks for it in distribution directory but that can be fixed easily by moving the library for now (until tmt will be able to find the libraries better).

This should work, assuming you have the keylime repo in tests require:

 - url: https://github.com/RedHat-SP-Security/keylime-tests.git
   name: /Library/sync

and importing it using rlImport keylime-tests/sync

EDIT: I just realized you are speaking about distribution/sync library, that I am not aware of.

[happz]

• Roles are not exported, there is just SERVERS with all the node (easy fix I assume, easy workaround available).

If you point me to documentation for what variables should be set, how, and when, I can implement it on top of the patch.

• HTML report does not contain roles so it is hard to tell what run where (without looking into logs).

This should be taken care of in #1920

[kkaarreell]

• Roles are not exported, there is just SERVERS with all the node (easy fix I assume, easy workaround available).
  If you point me to documentation for what variables should be set, how, and when, I can implement it on top of the patch.

Basically what restraint does is that systems having role=SERVERS in the XML recipe would be stored inside the variable SERVERS while systems having role=CLIENTS will be present in the variable CLIENTS. One could define also other roles but the Restraint internal sync mechanism works only with CLIENTS and SERVERS anyway so such a test would most likely fail. I.e. the implementation in tmt should be simple, you just need to store hostnames in a variable with a name matching a particular role (maybe in uppercase).

[happz]

Ok, so IIUIC, we want sort of a dynamic envvars for this, created from guest roles, uppercased, and containing all guest addresses with the same role? I.e. all guests with role: server would end up in SERVER=..., all guests with role: servers would end up in SERVERS=..., all guests with role: foo would end up in FOO=.... That sounds trivial enough :)

[kkaarreell]

Yes. I think that in fact those vars are matching the role exactly (no extra uppercase conversion) . See
https://restraint.readthedocs.io/en/latest/jobs.html?highlight=CLIENTS#task-roles

[psss]

Please, vote for the preferred mapping of the guest role name to environment variables:

• Names of environment variables used for multihost testing #2020