Integration/end-to-end testing

[mattgeddes]

This task covers the development of a test suite within a test harness that can be executed regularly to ensure that the protocol and other services all work as expected when it comes to high-level, user-visible functionality (as opposed to more localised unit testing).

I've added everyone to this one so that we can gather ideas or thoughts on what we might want to do for this one, and get volunteers to help out with parts of it.

[eureka-cpu]

Excited for this. Happy to chip in if we can break this into some smaller tasks.

[mattgeddes]

100%. It's going to take a little from all of us. Before we can split it up, we first need to come to some consensus for how this will work. Which framework we want to use, how we want to deploy/execute the test harness/suite in the short term, etc. If you want to take a bit of a look at potential test harnesses for this kind of thing and how you think it could be implemented/deployed, please feel free to drop ideas and thoughts into this issue and we can go from there.

[mattgeddes]

The initial plan was for us to each drop ideas for a test suite into this issue in the betanet timeframe and implement something later. I don't see a lot of ideas here, so we'll likely spend an Engineering call post-betanet to come up with a plan and raise child tasks to get it done by us as a team.

This isn't holding up betanet, so pushing to the next milestone.

[mattgeddes]

With betanet almost out, we'll likely start to take a look at this in earnest and hopefully include some specific tasks for the Crestone release that will get us some of what we're looking for. I'll add some thoughts here on requirements/criteria for what we're looking for:

1. A test harness/suite that will allow us to perform more end-to-end testing to cover cases that aren't suitable for unit testing. For example, where we have multiple separate services interacting with each other.
2. Chief among the test cases we add should be exercising the workflows we expect developers to follow
3. Ideally we'd like to be possible to run the test suite locally, as well as an automated pipeline running on some shared resource somewhere.
4. Need the ability to group/execute test cases in various ways to allow for local execution as well as larger-scale testing.
5. As few large dependencies as possible to execute.
6. Should also support the setup/teardown of clusters/networks/testbeds of machines/instances/services as a way to also help get test coverage over some of the operator tooling and workflow.

Feel free to add comments below with additional requirements or with suggestions for tools that fit the above requirements.

[vsawant10949]

It is imperative that our test cases comprehensively validate the workflows expected from developers. The test suite should encompass scenarios such as:

• Successful mining of the Proposal Block under the conditions of honest nodes
• Successful mining of the Convergence Block under the conditions of honest nodes
• Successful rejection of the Proposal Block with the presence of 'k' malicious nodes
• Successful rejection of the Convergence Block with the presence of 'k' malicious nodes
• Thorough testing of honest reputation
• Validation of quorum formation
• Testing the inclusion of a new node in the network when it submits a claim
• Testing the rejection of a node's participation in the election when its stake is very low
• Scenarios to throttle transactions