Merge branch 'master' into mikkergp/allow_custom_templates_in_control…

…_repo

.travis.yml

@@ -17,3 +17,4 @@ rvm:
   - 2.3.4
   - 2.4.0
   - 2.4.1
+  - 2.4.2

README.md

@@ -110,10 +110,20 @@ Why an array of hashes? Well, that is so that we can refer to the same node or n
 
 In the example below we have referred to `centos6a` and `centos7b` in all of our tests as they are in `all_nodes`, `non_windows_servers` and `centos_severs`. However we have *left the more specific references to last*. This is because entries in the test_matrix will override entries above them if applicable. Meaning that we are still only testing each class on the two Centos servers once (Because the gem does de-duplication before running the tests), but also making sure we run `roles::frontend_webserver` twice before checking for idempotency.
 
-**functions** In this section we can add functions that we want to mock when running spec tests. Each function takes the following agruments:
+**functions** In this section we can add functions that we want to mock when running spec tests. Each function takes the following arguments:
   - **type** *statement or rvalue*
   - **returns** *Optional: A value to return*
 
+**before and after conditions** We can set `before` and `after` blocks before each spec test. These are usually used when the functions to stub are conditional: stub functionx if the OS is windows, stub functiony if the fact java_installed is true. The facts are available through the `node_facts` hash.
+
+```yaml
+before:
+  - "Puppet::Util::Platform.stubs(:'windows?').returns(node_facts['kernel'] == 'windows')"
+
+after:
+  - "puts 'Test finished running'"
+``` 
+
 **opts** The `opts` section overrides defaults for the `Onceover::Controlrepo` class' `opts` hash.
 
 ```yaml
@@ -304,11 +314,23 @@ HOSTS:
 
 ### Hiera Data
 
-If you have hiera data inside your controlrepo (or somewhere else) the Controlrepo gem can be configured to use it. Just dump your `hiera.yaml` file from the puppet master into the `spec/` directory or the root of your controlrepo and you are good to go.
+If you have hiera data inside your controlrepo (or somewhere else) Onceover can be configured to use it. It is however worth noting the the `hiera.yaml` file that you currently use may not be applicable for testing right away. For example; if you are using `hiera-eyaml` I recommend creating a `hiera.yaml` purely for testing that simply uses the `yaml` backend, meaning that you don't need to provide the private keys to the testing machines.
+
+It is also worth noting that any hiera hierarchies that are based on custom facts will not work unless those facts are part of your factsets. Trusted facts will also not work at all as the catalogs are being compiled without the node's certificate. In these instances it may be worth creating a hierarchy level that simply includes dummy data for testing purposes in order to avoid hiera lookup errors.
+
+#### Creating the config file
+
+If your `hiera.yaml` is version 4 or 5 and lives in the root of the controlrepo (as it should), Onceover will pick this up automatically. If you would like to make changes to this file for testing purposes, create a copy under `spec/hiera.yaml`. Onceover will use this version of the hiera config file first if it exists.
 
-**WARNING:** This assumes that the path to your hiera data (datadir) is relative to the root of the controlrepo, if not it will fall over.
+#### Setting the `datadir`
 
-**Alternatively:**, if you are using cool new per-environment hiera config made available in puppet 4.x (Now called Hiera 5), the tool will automatically detect this and everything should work. If you want to use a different v5 `hiera.yaml` for testing, pleace it under the spec directory. Note that the datadir must be relative to the location of the hiera.yaml file in this instance. i.e. `../data`
+| Hiera Version | Config File Location | Required datadir |
+|---------------|----------------------|------------------|
+| 3 | `spec` folder | relative to the root of the repo e.g. `data` |
+| 4 *deprecated* | Root of repo | relative to the root of the repo e.g. `data` |
+| 4 *deprecated* | `spec` folder | relative to the spec folder e.g. `../data` |
+| 5 | Root of repo | relative to the root of the repo e.g. `data` |
+| 5 | `spec` folder | relative to the spec folder e.g. `../data` |
 
 ## Spec testing
 
@@ -658,3 +680,4 @@ Cheers to all of those who helped out:
   - aardvark
   - Mandos
   - Nekototori
+  - LMacchi

features/cache.feature

@@ -0,0 +1,33 @@
+@cache
+Feature: Create and maintain a .onceover cache
+  Onceover should be able to cache things in the .onceover directory for speed
+  increases and debugging of external modules. This cache should remain
+  up-to-date and should exactly mirror what would be created on the Puppet
+  master.
+
+  Background:
+    Given onceover executable
+
+  Scenario: Creating a cache
+    Given control repo "caching"
+    When I run onceover command "run spec"
+    Then the cache should exist
+    And the cache should contain all controlrepo files
+
+  Scenario: Creating a new file
+    Given existing control repo "caching"
+    When I create a file "example.txt"
+    And I run onceover command "run spec"
+    Then "example.txt" should be cached correctly
+
+  Scenario: Deleting a file
+    Given existing control repo "caching"
+    When I delete a file "deleteme.txt"
+    And I run onceover command "run spec"
+    Then "deleteme.txt" should be deleted from the cache
+
+  Scenario: Caching hidden files
+    Given existing control repo "caching"
+    When I create a file ".hidden/.hiddenfile"
+    And I run onceover command "run spec"
+    Then ".hidden/.hiddenfile" should be cached correctly

features/step_definitions/cache.rb

@@ -0,0 +1,38 @@
+Then(/^the cache should exist$/) do
+  cache_dir = File.join(@repo.root_folder,'.onceover')
+  expect(File.directory?(cache_dir)).to be true
+end
+
+Then(/^the cache should contain all controlrepo files/) do
+  # Get all root files
+  puts "Calculating MD5 hashes in repo"
+  repo_digest = Cache_Helper.digest(@repo.root_folder)
+  puts "#{repo_digest.count} MD5 hashes calculated"
+  puts "Calculating MD5 hashes in cache"
+  cache_digest = Cache_Helper.digest(File.join(@repo.root_folder,'.onceover/etc/puppetlabs/code/environments/production/'))
+  puts "#{cache_digest.count} MD5 hashes calculated"
+  expect(cache_digest).to include(repo_digest)
+end
+
+When(/^I (\w+) a file "(.*)"$/) do |action,file|
+  require 'securerandom'
+  actual_file = Pathname.new(File.join(@repo.root_folder,file))
+  case action
+  when "create"
+    FileUtils.mkdir_p(actual_file.dirname)
+    File.write(actual_file,SecureRandom.hex)
+  when "delete"
+    FileUtils.rm(actual_file)
+  end
+end
+
+Then(/^"(.*)" should be cached correctly$/) do |file|
+  original_digest = Cache_Helper.digest(File.join(@repo.root_folder,file))
+  cache_digest    = Cache_Helper.digest(File.join(@repo.root_folder,'.onceover/etc/puppetlabs/code/environments/production/',file))
+  expect(original_digest).to include(cache_digest)
+end
+
+Then(/^"([^"]*)" should be deleted from the cache$/) do |file|
+  deleted_file = Pathname.new(File.join(@repo.root_folder,'.onceover/etc/puppetlabs/code/environments/production/',file))
+  expect(deleted_file.exist?).to be false
+end

features/step_definitions/common.rb

@@ -1,4 +1,4 @@
-Given /^onceover executable$/ do
+Given(/^onceover executable$/) do
   @cmd = Command_Helper.new
 end
 
@@ -10,6 +10,11 @@
   FileUtils.cp_r "spec/fixtures/controlrepos/#{controlrepo_name}", @repo.tmp_folder
 end
 
+Given(/^existing control repo "([^"]*)"$/) do |controlrepo_name|
+  @repo = ControlRepo_Helper.new( controlrepo_name )
+  @cmd.controlrepo = @repo
+end
+
 Given(/^initialized control repo "([^"]*)"$/) do |controlrepo_name|
   step %Q(control repo "#{controlrepo_name}")
   step %Q(I run onceover command "init")
@@ -20,13 +25,13 @@
   FileUtils.rm_rf "#{@repo.root_folder}/#{filename}"
 end
 
-When /^I run onceover command "([^"]*)"$/  do |command|
+When(/^I run onceover command "([^"]*)"$/)  do |command|
   @cmd.command = command
   puts @cmd
   @cmd.run
 end
 
-Then /^I see help for commands: "([^"]*)"$/ do |commands|
+Then(/^I see help for commands: "([^"]*)"$/) do |commands|
   # Get chunk of output between COMMANDS and OPTION, there should be help section
   commands_help = @cmd.output[/COMMANDS(.*)OPTIONS/m, 1]
   commands.split(',').each do |command|

features/support/cache_helper.rb

@@ -0,0 +1,55 @@
+require 'pathname'
+
+class Cache_Helper
+
+  def cache_exists?
+    File.directory?(dir + '/.onceover')
+  end
+
+  def self.digest(path, opts = {
+    exceptions: ['.','..','.onceover']
+    })
+    if File.directory?(path)
+      # Get the list of files
+      children = Cache_Helper.get_children(path, opts[:exceptions])
+    else
+      children = [File.expand_path(path)]
+    end
+
+    # Calculate hashes
+    hashes = children.map do |child_path|
+      if File.directory? child_path
+        :directory
+      else
+        Digest::MD5.file(child_path)
+      end
+    end
+
+    root = Pathname.new(File.expand_path(path))
+    # Move pathnames back to relative
+    children.map! do |child_path|
+      Pathname.new(child_path).relative_path_from root
+    end
+    Hash[children.zip(hashes)]
+  end
+
+  def self.get_children(dir, exclusions)
+    root_files = []
+    files      = []
+    Dir.chdir(dir) do
+      # Get all root files
+      root_files = Dir.glob('*',File::FNM_DOTMATCH)
+      root_files = root_files - exclusions
+      root_files.each do |file|
+        files << file
+        files << Dir.glob("#{file}/**/*",File::FNM_DOTMATCH)
+      end
+      files.flatten!
+      # Calculate absolue paths
+      files.map! do |file|
+        File.expand_path(file)
+      end
+    end
+    files
+  end
+end

lib/onceover/testconfig.rb

@@ -23,6 +23,8 @@ class TestConfig
     attr_accessor :filter_classes
     attr_accessor :filter_nodes
     attr_accessor :mock_functions
+    attr_accessor :before_conditions
+    attr_accessor :after_conditions
     attr_accessor :skip_r10k
     attr_accessor :strict_variables
 
@@ -35,15 +37,17 @@ def initialize(file, opts = {})
         raise "Could not parse #{file}, check that it is valid YAML and that the encoding is correct"
       end
 
-      @classes          = []
-      @nodes            = []
-      @node_groups      = []
-      @class_groups     = []
-      @spec_tests       = []
-      @acceptance_tests = []
-      @opts             = opts
-      @mock_functions   = config['functions']
-      @strict_variables = opts[:strict_variables] ? 'yes' : 'no'
+      @classes           = []
+      @nodes             = []
+      @node_groups       = []
+      @class_groups      = []
+      @spec_tests        = []
+      @acceptance_tests  = []
+      @opts              = opts
+      @mock_functions    = config['functions']
+      @before_conditions = config['before']
+      @after_conditions  = config['after']
+      @strict_variables  = opts[:strict_variables] ? 'yes' : 'no'
 
       # Initialise all of the classes and nodes
       config['classes'].each { |clarse| Onceover::Class.new(clarse) } unless config['classes'] == nil
@@ -181,29 +185,45 @@ def deploy_local(repo = Onceover::Controlrepo.new, opts = {})
       #
       # If there are more situations like this we can add them to this array as
       # full paths
-      excluded_files = []
-      if ENV['GEM_HOME']
-        logger.debug "Excluding #{ENV['GEM_HOME']} from controlrepo copy"
-        excluded_files << Dir.glob("#{ENV['GEM_HOME']}/**/*")
-        excluded_files.flatten!
+      excluded_dirs = []
+      excluded_dirs << Pathname.new("#{repo.root}/.onceover")
+      excluded_dirs << Pathname.new(ENV['GEM_HOME']) if ENV['GEM_HOME']
+
+      controlrepo_files = get_children_recursive(Pathname.new(repo.root))
+
+      # Exclude the files that should be skipped
+      controlrepo_files.delete_if do |path|
+        parents = [path]
+        path.ascend do |parent|
+          parents << parent
+        end
+        parents.any? { |x| excluded_dirs.include?(x) }
       end
 
-      # Exclude the files we need to
-      controlrepo_files = Dir.glob("#{repo.root}/**/*")
-      files_to_copy     = (controlrepo_files - excluded_files).delete_if { |path| Pathname(path).directory? }
-      folders_to_copy   = (controlrepo_files - excluded_files).keep_if { |path| Pathname(path).directory? }
+      folders_to_copy = controlrepo_files.select { |x| x.directory? }
+      files_to_copy   = controlrepo_files.select { |x| x.file? }
 
       logger.debug "Creating temp dir as a staging directory for copying the controlrepo to #{repo.tempdir}"
       temp_controlrepo = Dir.mktmpdir('controlrepo')
 
       logger.debug "Creating directories under #{temp_controlrepo}"
-      FileUtils.mkdir_p(folders_to_copy.map { |folder| "#{temp_controlrepo}/#{(Pathname(folder).relative_path_from(Pathname(repo.root))).to_s}"})
+      FileUtils.mkdir_p(folders_to_copy.map { |folder| "#{temp_controlrepo}/#{(folder.relative_path_from(Pathname(repo.root))).to_s}"})
 
       logger.debug "Copying files to #{temp_controlrepo}"
       files_to_copy.each do |file|
-        FileUtils.cp(file,"#{temp_controlrepo}/#{(Pathname(file).relative_path_from(Pathname(repo.root))).to_s}")
+        FileUtils.cp(file,"#{temp_controlrepo}/#{(file.relative_path_from(Pathname(repo.root))).to_s}")
       end
 
+      logger.debug "Writing manifest of copied controlrepo files"
+      require 'json'
+      # Create a manifest of all files that were in the original repo
+      manifest = controlrepo_files.map do |file|
+        # Make sure the paths are relative so they remain relevant when used later
+        file.relative_path_from(Pathname(repo.root)).to_s
+      end
+      # Write all but the first as this is the root and we don't care about that
+      File.write("#{temp_controlrepo}/.onceover_manifest.json",manifest[1..-1].to_json)
+
       # When using puppetfile vs deploy with r10k, we want to respect the :control_branch
       # located in the Puppetfile. To accomplish that, we use git and find the current
       # branch name, then replace strings within the staged puppetfile, prior to copying.
@@ -218,11 +238,22 @@ def deploy_local(repo = Onceover::Controlrepo.new, opts = {})
       new_puppetfile_contents = puppetfile_contents.gsub(/:control_branch/, "'#{git_branch}'")
       File.write("#{temp_controlrepo}/Puppetfile", new_puppetfile_contents)
 
-
-      FileUtils.mkdir_p("#{repo.tempdir}/#{repo.environmentpath}/production")
+      # Remove all files written by the laste onceover run, but not the ones
+      # added by r10k, because that's what we are trying to cache but we don't
+      # know what they are
+      old_manifest_path = "#{repo.tempdir}/#{repo.environmentpath}/production/.onceover_manifest.json"
+      if File.exist? old_manifest_path
+        logger.debug "Found manifest from previous run, parsing..."
+        old_manifest = JSON.parse(File.read(old_manifest_path))
+        logger.debug "Removing #{old_manifest.count} files"
+        old_manifest.reverse.each do |file|
+          FileUtils.rm_f(File.join("#{repo.tempdir}/#{repo.environmentpath}/production/",file))
+        end
+      end
+      FileUtils.mkdir_p("#{repo.tempdir}/#{repo.environmentpath}")
 
       logger.debug "Copying #{temp_controlrepo} to #{repo.tempdir}/#{repo.environmentpath}/production"
-      FileUtils.cp_r(Dir["#{temp_controlrepo}/*"], "#{repo.tempdir}/#{repo.environmentpath}/production")
+      FileUtils.cp_r("#{temp_controlrepo}/.", "#{repo.tempdir}/#{repo.environmentpath}/production")
       FileUtils.rm_rf(temp_controlrepo)
 
       # Pull the trigger! If it's not already been pulled
@@ -331,5 +362,18 @@ def run_filters(tests)
       tests
     end
 
+    private
+
+    def get_children_recursive(pathname)
+      results = []
+      results << pathname
+      pathname.each_child do |child|
+        results << child
+        if child.directory?
+          results << get_children_recursive(child)
+        end
+      end
+      results.flatten
+    end
   end
 end

onceover.gemspec

@@ -4,7 +4,7 @@ $LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
 
 Gem::Specification.new do |s|
   s.name        = "onceover"
-  s.version     = "3.3.1"
+  s.version     = "3.4.0"
   s.authors     = ["Dylan Ratcliffe"]
   s.email       = ["[email protected]"]
   s.homepage    = "https://github.com/dylanratcliffe/onceover"

spec/fixtures/controlrepos/caching/.gitignore

@@ -0,0 +1 @@
+.onceover

spec/fixtures/controlrepos/caching/.travis.yml

@@ -0,0 +1,2 @@
+---
+some: stuff

spec/fixtures/controlrepos/caching/Gemfile

@@ -0,0 +1,5 @@
+source 'https://rubygems.org'
+
+gem 'onceover'
+gem 'beaker', '~> 2.0'
+

spec/fixtures/controlrepos/caching/Puppetfile

@@ -0,0 +1,17 @@
+forge "http://forge.puppetlabs.com"
+#
+# I want to download some modules to check if r10k feature in Onceover works correctly.
+#
+
+# Versions should be updated to be the latest at the time you start
+mod "puppetlabs/stdlib",      '4.11.0'
+
+# Modules from Git
+# Examples: https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd#examples
+mod 'apache',
+ :git    => 'https://github.com/puppetlabs/puppetlabs-apache',
+ :commit => '83401079053dca11d61945bd9beef9ecf7576cbf'
+
+#mod 'apache',
+#  :git    => 'https://github.com/puppetlabs/puppetlabs-apache',
+#  :branch => 'docs_experiment'

spec/fixtures/controlrepos/caching/Rakefile

@@ -0,0 +1 @@
+require 'onceover/rake_tasks'