Automated testing for plans

[predominant]

Refs: habitat-sh/core-plans#1118

This adds a plan for testing .. plans.

The intention is that this would be integrated with studio in the same way hab-plan-build is, with a wrapper script placed in /bin/build-test that would call $(hab pkg path core/hab-plan-build-test)/bin/hab-plan-build-test.

It is designed and inspired by hab-plan-build, in that each step of the process can be overwritten, but assumes defaults and standards and will operate fine without customization.

The tests perform the following steps:

• Build
• Install package
• Check if plan has a service, if so, load
• Wait for service to load, if plan has a service
• Run tests

To test this now, I have a simple binary plan (terraform) plan on my fork of core-plans that can be used for testing:

Pre-integration Testing

# 1. Grab testing core-plans branch
git checkout -b predominant-predominant/terraform-buildtest master
git pull https://github.com/predominant/core-plans.git predominant/terraform-buildtest

# 2. Enter studio
hab studio enter

# 3. Temporary steps (to simulate this being shipped with studio)
hab pkg install --binlink predominant/hab-plan-build-test/0.63.0-dev/20180911153418
ln -s /bin/hab-plan-build-test /bin/build-test

# 4. Testing suite is ready.. Test:
build-test terraform

Steps 1 and 3 are temporary.

Proposed testing story after integration / merge

hab studio enter
build-test terraform

Default testing scenario

• Plan will have BATS tests at plan-name/tests/test.bats
• Inside a studio, run: build-test plan-name

Custom testing scenario

You have special-plan that needs some dependency, or other tools installed for testing.

• Write tests at special-plan/tests/test.bats
• Write custom test.sh at special-plan/tests/test.sh, contents below:

do_test_bats() {
  # Install netstat for checking port availability in BATS
  hab pkg install --binlink core/netstat
  do_default_test_bats
}

[thesentinels]

Thanks for the pull request! Here is what will happen next:

1. Your PR will be reviewed by the maintainers
2. If everything looks good, one of them will approve it, and your PR will be merged.

Thank you for contributing!

[nellshamrell]

I'm about to be on PTO for 5 days...but I really like what I'm seeing here. Will leave this in @smacfarlane's hands until I get back.

[smacfarlane]

😻 I really like what I'm seeing here. One point that I think worth discussing is the scope of this software.

My opinion is that this should focus on the testing of packages/artifacts and have no knowledge of the plan file or build process, we can push the "glue" bits into the studio. What I mean by that is that the studio already has a helper function "build" that wraps around the call to hab-plan-build. I think we should do the same for "test".

This is going to change the invocation as you would need to give it a path to tests rather than a plan, and an artifact to test. It might require us to make a few assertions/assumptions about the state of the system going into the test (what env vars are set, etc.) but I'm ok with that as long as the shape of things is well defined and it does the Right Thing ™️ by default.

[predominant]

@smacfarlane What you're describing is just using BATS. You would call bats /path/to/tests.

This was designed as a wrapper to build, with the ability to skip the build steps if required. This wraps the build call as well to serve as a mechanism to completely test from source to final artifact, including services. At the core its doing what you want, its calling bats to run the tests themselves.

[bdangit]

This is great!

Is there anyway that we don't have to use BATS? Take do_check() callback for example, it would allow us to define any steps we would like to program before the artifact is packaged.

[predominant]

@bdangit Can rename do_test_bats to something more generic if that suits. As it is, you can override, and not use bats.

[dmccown]

@predominant Thanks for submitting this pull request! This is something we want to add to Chef Habitat in the future. However, due to our limited bandwidth we are going to close this PR at this moment. I'm mostly concerned around how we'll work this into our release cycle and ensuring we can support this well. We are working to improve our release process, which will help reduce the burden on this team. At the point we have better automation and some more bandwidth we can re-open the conversation on how best to get a feature like this added.