[YT-CC-1204] Adds orphan monitoring to the sidekiq recipe

[radamanthus]

Github note: This closes issue #279

Description of your patch

Adds orphan monitoring to the sidekiq recipe

Recommended Release Notes

Adds orphan monitoring to the sidekiq recipe

Estimated risk

Low. Only the sidekiq recipe is changed. The new feature is disabled by default.

Components involved

sidekiq custom chef recipe

Description of testing done

Tested the unit tests on a solo environment with redis and sidekiq utility instances

QA Instructions

Enable both the redis and the sidekiq recipes.
Modify custom-sidekiq/attributes/default.rb and set sidekiq['orphan_monitor_enabled'] to true
Upload the chef recipe before booting the environment.

Boot a new cluster environment for the rails_activejob_example application

• Unicorn
• Ruby 2.3
• Rubygems 2.6.3
• Postgres 9.4
• any region
• QA stack

The cluster should have the following instances:

• 1 app_master
• 1 redis
• 1 sidekiq
• 1 db_master

On the sidekiq instance:

• verify that the file /engineyard/bin/sidekiq_orphan_monitor exists
• run crontab -l for the deploy user. Verify that a cron job was created to periodically run /engineyard/bin/sidekiq_orphan_monitor

On all the other instances:

• verify that the file /engineyard/bin/sidekiq_orphan_monitor does not exist
• run crontab -l for the deploy user. There should be no cron job for /engineyard/bin/sidekiq_orphan_monitor

[jegasega]

@radamanthus.
Could you calrify for me one moment?

In the main recipe attributes we have this flag

   # unless a utility name is set, in which case, Sidekiq will
   # only be installed on to a utility instance that matches
   # the name
   sidekiq['is_sidekiq_instance'] = true 

By default it will install sidekiq on all the instances

And in custom recipe attributes we have

# this is the default
# default['sidekiq']['is_sidekiq_instance'] = true

# run the recipe on a utility instance named background_workers
# default['sidekiq']['is_sidekiq_instance'] = (node['dna']['instance_role'] == 'util' && node['dna']['name'] == 'background_workers')

# run the recipe on a solo instance
# default['sidekiq']['is_sidekiq_instance'] = (node['dna']['instance_role'] == 'solo') 

What do you think maybe we need to change the default behaviour for sidekiq to be installed only on solo and util instances?

[radamanthus]

@jegasega The instructions we provide to customers tell them to use custom-sidekiq, not sidekiq.
https://github.com/engineyard/ey-cookbooks-stable-v5/blob/next-release/custom-cookbooks/sidekiq/cookbooks/custom-sidekiq/README.md

custom-sidekiq overrides whatever setting is there in the sidekiq recipe.

We have not had many tickets about customers about sidekiq getting installed on all instances when they only wanted it to install on a particular instance.

Having said that, I think it's a good idea to modify the sidekiq recipe to install only on a named utility instance by default. But that should be done in another PR, not on this one.