Windows testing using kitchen-vagrant #112

dafyddj · 2019-08-22T17:07:27Z

I'm putting this out there to gather some opinions.
Would this be useful to add to the formula, despite the fact that the tests can't be run in any CI/CD service that I know of?

Testing requires Virtualbox and Vagrant on the local machine.
The OpenVPN server is first run on a linux box, and the tests are subsequently run on the Windows box.

KITCHEN_YAML=kitchen.windows.yml kitchen converge default-ubuntu
KITCHEN_YAML=kitchen.windows.yml kitchen verify default-windows

myii

Just some initial feedback.

.travis.yml

test/salt/pillar/windows.sls

.travis.yml

myii · 2019-08-23T01:35:49Z

@dafyddj Apologies for the poor-quality communication earlier, both here and in Slack -- I was out and about so it got a bit messy. I'm glad you figured out what I was trying to say!

I'm putting this out there to gather some opinions.
Would this be useful to add to the formula, despite the fact that the tests can't be run in any CI/CD service that I know of?

I'd very much welcome Windows testing, even if it was only local. It would help with PRs that have got stuck, such as the following:

refactor(salt-minion): remove old code, rearrange & reformat salt-formula#416 (comment)

In terms of CI/CD, there's a potential solution. You may have noticed that Cirrus CI has been enabled for this org's repos and is being used for a few of them (e.g. rkhunter and syslog-ng). They also have Windows Containers available, which could be useful. We've had contact with Cirrus CI and they were responsive and helpful, both on GitHub and via. e-mail, such as the discussion going forward from here:

Resolve Travis concurrent jobs or consider Azure pipelines template-formula#118 (comment)

dafyddj · 2019-08-23T11:30:43Z

I have had a little play with kitchen-docker using Windows containers (support was only added a matter of weeks ago), and there's a slight gotcha in that files cannot be uploaded to the container while it's running - so, at least at the moment, it's not really a viable testing method.

dafyddj · 2019-10-22T17:48:11Z

@myii care to take another look at this?

I've changed the approach a little: pre-Salted Windows image; re-use of Inspec tests and kitchen.yml.
Comments welcome.

myii · 2019-10-22T17:50:31Z

@dafyddj Sounds good! I'll try to have a look at this soon.

* saltstack-formulas/openvpn-formula#112

myii

@dafyddj This is an excellent PR! I can see this becoming the basis for all sorts of Vagrant-based testing across our formulas. Thank you very much for addressing this problem.

I've got a few requests that I've mentioned inline. I haven't clearly defined the one about a separate test pillar, so please shout if anything's unclear. I also want to draw the attention of the other contributors to this, so we can confirm a standard structure that can be used in other formulas as well.

Checked with: myii/ssf-formula#81

.yamllint

Gemfile

kitchen.yml

myii · 2019-10-23T05:19:16Z

@dafyddj Whatever decision is reached, it would be good to update the testing section in the README to explain how to run the local Vagrant-based testing.

dafyddj · 2019-10-23T10:59:51Z

While I look over this, what do you think of this approach (1037162) whereby .sls files are rendered to .yaml before being linted by yamllint? (kitchen.yml could be rendered in a similar way.)

dafyddj · 2019-10-23T11:02:50Z

Resulting Travis output here.

dafyddj · 2019-10-23T11:03:33Z

@dafyddj Whatever decision is reached, it would be good to update the testing section in the README to explain how to run the local Vagrant-based testing.

Yes, this is on my TODO list.

myii · 2019-10-23T13:36:52Z

While I look over this, what do you think of this approach (1037162) whereby .sls files are rendered to .yaml before being linted by yamllint? (kitchen.yml could be rendered in a similar way.)

@dafyddj This has crossed my mind plenty of times, since getting my hands dirty with all of the linters. It's nice to see a working method. It definitely appeals to the perfectionist in me but that's not necessarily a good thing! Having cast my eye over a lot of debug output, it's obvious that we'd have to use a fairly relaxed set of yamllint rules, which would be easy to do via. a secondary configuration file (that can be customised per formula).

So what are my concerns? They're a bit elusive right now but I suppose one is that we could end up distracting ourselves unnecessarily, or giving ourselves too many hoops to jump through. Another is that the lint won't cover the whole of each source file; the rendered files will miss sections due to conditionals, etc. The issue surrounding jinja_env:trim_blocks also springs to mind -- in that, we could end up with excellent rendered output, which actually breaks when used by consumers.

I'd still like to see this in action, though. How about we do this in a separate job with jobs:allow_failures set, like we had with rubocop? That will give us a chance to see how it works out. We can also share our findings with the others, to attempt to achieve a consensus.

Please don't mistake this as a negative response in any way. I would definitely use this in a personal capacity, within a well-defined environment. My feedback here is more measured, trying to consider the wider implications.

myii · 2019-10-23T16:13:59Z

@dafyddj Another reason for having a general purpose kitchen.vagrant.yml: supporting FreeBSD images as well. There's a fairly long thread on Slack but today's update is here:

https://freenode.logbot.info/saltstack-formulas/20191023#c2787123

- - -

16:06 myii @kp We're getting closer to establishing Vagrant-based testing again: https://saltstack-formulas/openvpn-formula #112. We could easily add the FreeBSD images to the kitchen.vagrant.yml file. How are you getting along with the pre-salted images? CC: @noel.mcloughlin.

16:10 kp @myii thanks for the pointer, was pretty busy with $real_job issues, I hope I can prepare something meaningful this weekend

16:10 myii Excellent, look forward to it.

myii · 2019-10-25T16:31:38Z

@dafyddj Is this still a WIP?

dafyddj · 2019-10-25T16:32:44Z

Yes, still have a couple of things to add to it.

myii · 2019-10-25T16:33:28Z

@dafyddj OK, please ping me when you're ready for the next stage.

dafyddj · 2019-10-28T18:57:18Z

@myii I've added to the documentation and updated my Vagrant box (to version 2019.10.141), so this PR is ready for further review.

docs/README.rst

* and with paths that contain spaces

* so that adapters are created before service is started

* use `rspec-retry` to make retrying the logfile test platform-independent * do some DRYing of the inspec control files

saltstack-formulas-travis · 2019-11-06T00:02:14Z

🎉 This PR is included in version 0.15.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

* saltstack-formulas/openvpn-formula#112

myii requested changes Aug 22, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

test/salt/pillar/windows.sls Outdated Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

dafyddj force-pushed the kitchen-vagrant branch from 718e172 to c89b98c Compare August 22, 2019 18:53

myii mentioned this pull request Sep 25, 2019

Add chocolatey management support saltstack-formulas/packages-formula#60

Merged

dafyddj force-pushed the kitchen-vagrant branch from c89b98c to c24e1cc Compare October 19, 2019 21:57

myii added a commit to myii/ssf-formula that referenced this pull request Oct 23, 2019

feat(openvpn): implement Windows testing using kitchen-vagrant

270ea1b

* saltstack-formulas/openvpn-formula#112

myii mentioned this pull request Oct 23, 2019

feat(openvpn): implement Windows testing using kitchen-vagrant myii/ssf-formula#81

Merged

myii requested changes Oct 23, 2019

View reviewed changes

.yamllint Outdated Show resolved Hide resolved

.yamllint Show resolved Hide resolved

Gemfile Outdated Show resolved Hide resolved

kitchen.yml Outdated Show resolved Hide resolved

kitchen.yml Outdated Show resolved Hide resolved

dafyddj force-pushed the kitchen-vagrant branch 2 times, most recently from bc7b160 to d2575e2 Compare October 23, 2019 18:53

dafyddj changed the title ~~WIP: Windows testing using kitchen-vagrant~~ Windows testing using kitchen-vagrant Oct 28, 2019

myii reviewed Oct 29, 2019

View reviewed changes

docs/README.rst Show resolved Hide resolved

dafyddj force-pushed the kitchen-vagrant branch from 80cce57 to c09c15a Compare November 5, 2019 19:20

dafyddj added 4 commits November 5, 2019 23:21

fix(dhparams): allow generating Diffie Hellman parameters on Windows

3802024

* and with paths that contain spaces

fix(adapters): call adapters.sls in init.sls

3ebcbe9

* so that adapters are created before service is started

test(windows): add local testing of Windows using Vagrant/Virtualbox

96c3001

* use `rspec-retry` to make retrying the logfile test platform-independent * do some DRYing of the inspec control files

ci(travis): merge rubocop linter into main lint job

9f82955

docs: add steps required for testing using Vagrant

0229d14

dafyddj force-pushed the kitchen-vagrant branch from c09c15a to 0229d14 Compare November 5, 2019 23:22

myii approved these changes Nov 5, 2019

View reviewed changes

myii merged commit 5a263e0 into saltstack-formulas:master Nov 5, 2019

saltstack-formulas-travis added the released label Nov 6, 2019

dafyddj deleted the kitchen-vagrant branch November 6, 2019 00:06

myii added a commit to myii/ssf-formula that referenced this pull request Nov 8, 2019

feat(openvpn): implement Windows testing using kitchen-vagrant

d083ff2

* saltstack-formulas/openvpn-formula#112

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows testing using kitchen-vagrant #112

Windows testing using kitchen-vagrant #112

dafyddj commented Aug 22, 2019

myii left a comment

myii commented Aug 23, 2019

dafyddj commented Aug 23, 2019

dafyddj commented Oct 22, 2019

myii commented Oct 22, 2019

myii left a comment •

edited

Loading

myii commented Oct 23, 2019

dafyddj commented Oct 23, 2019

dafyddj commented Oct 23, 2019

dafyddj commented Oct 23, 2019

myii commented Oct 23, 2019

myii commented Oct 23, 2019

myii commented Oct 25, 2019

dafyddj commented Oct 25, 2019

myii commented Oct 25, 2019

dafyddj commented Oct 28, 2019

saltstack-formulas-travis commented Nov 6, 2019

Windows testing using kitchen-vagrant #112

Windows testing using kitchen-vagrant #112

Conversation

dafyddj commented Aug 22, 2019

myii left a comment

Choose a reason for hiding this comment

myii commented Aug 23, 2019

dafyddj commented Aug 23, 2019

dafyddj commented Oct 22, 2019

myii commented Oct 22, 2019

myii left a comment • edited Loading

Choose a reason for hiding this comment

myii commented Oct 23, 2019

dafyddj commented Oct 23, 2019

dafyddj commented Oct 23, 2019

dafyddj commented Oct 23, 2019

myii commented Oct 23, 2019

myii commented Oct 23, 2019

myii commented Oct 25, 2019

dafyddj commented Oct 25, 2019

myii commented Oct 25, 2019

dafyddj commented Oct 28, 2019

saltstack-formulas-travis commented Nov 6, 2019

myii left a comment •

edited

Loading