Support test suites

[razvan]

First, thank you for chainsaw. We have been using kuttl extensively for the past two years but are in dear need of something more structured and flexible. We are currently using a test generator to go around some kuttle limitations, but this cannot address everything.

We would really love to be able to define test suites.

Test suites would provide:

• A "global" set up and a tear down step for all tests. The main usecase would be operators installations and/or third-party resources that are time consuming to install in every test.
• A selector for the tests to run. The use case here are operators that support multiple operand versions. Not all operand versions need to be tested all the time. Often it's enough to test only one version (usually the latest).
• Templates (think Jinja2 templates). Here multiple use cases would be covered:
  • Same Test with multiple Resource versions. When an operator can manage multiple versions of a product/operand, the tests for all versions are identical and the only difference is the operand version. This should be a template variable that is expanded at test execution time. This means that a Test is possibly expanded into several Tests before execution. Add more than one variable to a resource and you have a test matrix out of a single Test manifest.
  • Conditional test steps. When running operands with TLS enabled, additional resources need to be created (secrets, pvcs, etc.) but the structure and subject of the test remains identical to the case where no TLS is used. In such cases it would be useful to plug those additional resources conditionally depending on whether TLS is enabled or not.

I realize this is quite a big request but I'd be more then happy to help clarify individual aspects of it. I'm convinced that especially the first point would be extremly beneficial for us and certainly others too.

[shubham-cmyk]

@razvan
Thanks for the feedback.

Currently, we have been using the nested dir to write down the test it worked as a complete test suite and you can setup and teardown from the top level of the dir.

Your ideas about putting some selectors and conditions seem to be promising.
We can surely do something about this would you like to join our next community meeting at Thursday 2PM UTC

[shubham-cmyk]

Also Join our #kyverno-chainsaw channel under the Kubernetes Slack

[eddycharly]

Hey @razvan

Quick feedback, happy to discuss it more.

A "global" set up and a tear down step for all tests. The main usecase would be operators installations and/or third-party resources that are time consuming to install in every test.

I understand the idea behind this but I kinda like the fact that Chainsaw is not responsible for setting up the operator.
What is the advantage of coupling Chainsaw with the operator install ? Versus installing the operator first with helm, kustomize, kubectl, whatever... then invoking Chainsaw ?

A selector for the tests to run. The use case here are operators that support multiple operand versions. Not all operand versions need to be tested all the time. Often it's enough to test only one version (usually the latest).

Right we don't support test selector yet (are we talking about labels selector here ?). Still, you can pass a regex to include/exclude tests to run.

Templates (think Jinja2 templates).

I have been thinking about this for a while. It's getting more mature in my brain, chances are this will land soon.
I hope to be able to discuss it more in the coming days :)

[eddycharly]

chainsaw looks like a very good step forward but it currently doesn't provide me enough incentives for a switch.

That's because you didn't discover the Chainsaw secret sauce yet 😂

I admit that our docs don't highlight enough why Chainsaw is so unique. A blog post is coming, it should clarify what the real value of Chainsaw is and you will never compare Chainsaw to KUTTL anymore.

[eddycharly]

For us it's awkward to not include the operator as a resource in the test suite. It is, after all, the thing we are testing.

I understand the reasoning. On the other hand i like the idea that Chainsaw doesn't care about how the operator it is testing was installed.
Let's say I want to run tests against a cluster with everything already installed, now i need to edit the test suite just because it should not install the operator ?
TBH, it doesn't sound right to me.

Right now Chainsaw only knows about configuration and tests, maybe we can introduce a new concept. As long as it does not live in the the configuration i'm fine with it I guess.

I don't really care about the the implementation here. What we need is a way to tell chainsaw to only run tests matching a certain label (or tag, or marker). This has the purpose of speeding up some runs and only execute the full suite when necessary.

Ok, you want something like test metadata, not related to the actual folder hierarchy right ?
Looks like labels are a good choice.
Another approach would be to let the test decide if it needs to run or not.

We use them extensively now to generate the test matrix prior to actually running the tests and have lots of examples. The Trino operator test suite for exmaple, uses the values defined in test-definition.yaml to replace variables defined in templates like this: 10-install-trino.yaml.j2

But we are not hooked on Jinja2 and would be willing to switch to something on par if need be.

Didn't look at those links yet. Thanks for sharing examples, I'll look at them later and comment.

Thanks, that's great feedback !
Chainsaw is a young project, feedback like this helps a lot and that's great to see folks considering the switch to Chainsaw :)

[razvan]

Ok, you want something like test metadata, not related to the actual folder hierarchy right ?
Looks like labels are a good choice.
Another approach would be to let the test decide if it needs to run or not.

Yes, whatever the implementation, it should be orthogonal to any folder hierarchy.

Chainsaw is a young project, feedback like this helps a lot and that's great to see folks considering the switch to Chainsaw :)

Thank you for starting it and looking forward to the upcoming announcements.

[eddycharly]

Yes, whatever the implementation, it should be orthogonal to any folder hierarchy.

I created #670, targeting next release.

[eddycharly]

Implemented in #671 :)