Compute Integration

Introduction

The jclouds !ComputeService api is a provisioning system. Its goal is to start your configuration and integration process. One configuration and systems integration tool is Chef. Chef has a Ruby DSL that allows you to modularize your configuration logic in items called cookbooks. Even though Chef and its cookbooks are written in ruby, they can still be managed in jclouds via the Chef REST api and bootstrapping. The following wiki will show you a couple ways to do that.

Prerequisites

Opscode Platform Access

The examples below assume that you have an opscode platform account, and you have assigned the group clients the permission to create new clients. Please log in here if you need to do that.

Java Example

dependencies

You must run mvn install locally, as jclouds-chef is not published to a repository. However, you also need to pull in dependencies for the jclouds compute components.

  <repository>
    <id>jclouds</id>
    <url>http://jclouds.googlecode.com/svn/repo</url>
  </repository>

  <dependencies>
    <dependency>
      <groupId>org.jclouds</groupId>
      <artifactId>jclouds-chef-compute</artifactId>
      <version>1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>org.jclouds</groupId>
      <artifactId>jclouds-allcompute</artifactId>
      <version>1.0-beta-7</version>
    </dependency>
    <!-- for runScript support. note incompatible with google app engine -->
    <dependency>
      <groupId>org.jclouds</groupId>
      <artifactId>jclouds-jsch</artifactId>
      <version>1.0-beta-7</version>
    </dependency>
  </dependencies>

the code

In this example, I'll use BlueLock's vCloud 1.0. However, this works on all compute clouds jclouds supports.

import static com.google.common.collect.Iterables.*;
import static org.jclouds.compute.options.TemplateOptions.Builder.runScript;
import static org.jclouds.chef.predicates.CookbookVersionPredicates.*;
...

computeProvider = "bluelock-vclouddirector";

// setup connection properties
Properties props = new Properties();
props.setProperty("bluelock-vclouddirector.identity",  "YOUR_USER@YOUR_ORG");
props.setProperty("bluelock-vclouddirector.credential", "YOUR_PASSWORD");
props.setProperty("chef.endpoint", "https://api.opscode.com/organizations/YOUR_ORG");
props.setProperty("chef.identity", "YOUR_USER");
  // note if this is in the classpath, use .resource, not .file
props.setProperty("chef.credential.file", System.getProperty("user.home") + "/.chef/YOUR_USER.pem");

// make a connection to compute provider.  note, we use ssh to bootstrap chef
computeContext = new ComputeServiceContextFactory().createContext(
   "bluelock-vclouddirector", ImmutableSet.of(new JschSshClientModule()), props);

// make a connection to the opscode platform
chefContext = new ChefContextFactory().createContext(props);

// group all nodes in both chef and the compute provider by this tag
tag = "test";
recipe = "apache2";

// check to see if the recipe you want exists
cookbookVersions = chefContext.getChefService().listCookbookVersions();
if (any(cookbookVersions, containsRecipe(recipe)))
   runList = new RunListBuilder().addRecipe(recipe).build();

// update the chef service with the run list you wish to apply to all nodes with the tag
chefContext.getChefService().updateRunListForTag(runList, tag);

//  build the script that will bootstrap the chef client
bootstrap = chefContext.getChefService().createClientAndBootstrapScriptForTag(tag);

// run a node on the compute provider that bootstraps chef
nodes = computeContext.getComputeService().runNodesWithTag(tag, 1, runScript(bootstrap);

Clojure Example

dependencies

You will likely be best off using leiningen. Here are the dependencies you should add, which allow all compute providers supported in jclouds.

[org.clojure/clojure "1.2.0"]
[org.clojure/clojure-contrib "1.2.0"]
[org.danlarkin/clojure-json "1.1"]
[org.jclouds/jclouds-jsch "1.0-beta-7"]
[org.jclouds/jclouds-allcompute "1.0-beta-7"]
[org.jclouds/jclouds-chef-compute "1.0-SNAPSHOT"]

Note that you need to build jclouds-chef locally using mvn install. To get the other jclouds components, configure the repository.

  :repositories {"jclouds" "http://jclouds.googlecode.com/svn/repo"}

the code

In this example, I'll use BlueLock's vCloud 1.0. However, this works on all compute clouds jclouds supports.

(ns
  chef-compute-demo
  (:require
    (org.jclouds [compute :as compute] [chef :as chef])))

;; make a connection to the bluelock vcloud 1.0 service.  note, we use ssh to bootstrap chef
(def compute
  (compute/compute-service "bluelock-vclouddirector" "YOUR_USER@YOUR_ORG" "YOUR_PASSWORD" :ssh))

;; make a connection to the opscode platform
(def chef
  (chef/chef-service "chef"
    "YOUR_USER" (chef/load-pem "YOUR_USER")
    :chef.endpoint "https://api.opscode.com/organizations/YOUR_ORG"))

;; group all nodes in both chef and the compute provider by this tag
(def tag "test")
;; assign these nodes to run apache
(def run-list #{"recipe[apache2]"})

;; as a shortcut, lets use the same chef context for all below commands
(chef/with-chef-service [chef]

  ;; update the chef service with the run list you wish to apply to all nodes with the tag
  (chef/update-run-list run-list tag)

  ;; build a compute template that will bootstrap the chef client
  (def template
    (compute/build-template compute
      {:run-script (chef/create-bootstrap tag)}))
 
  ;; run a node on the compute provider that bootstraps chef
  (compute/run-node tag template compute))

Details

relationship between tags and run lists

jclouds compute-chef integration is facilitated by the Chef concept of databags. We use a databag to store the relationships between Chef nodes and jclouds compute tags. By default these relationships are stored in a databag named "bootstrap". However, you can change this by adjusting the property chef.bootstrap-databag. We provide a couple of utilities to help manage the data in this special bag. The two methods are named updateRunListForTag and getRunListForTag in java, and update-run-list and run-list in clojure at the moment.

what does create-bootstrap do?

The method named createClientAndBootstrapScriptForTag in java and create-bootstrap in clojure do all the heavy lifting required to create a valid bootstrap script for chef. Here is the overall process.

1. create a new client for the tag if one isn't in memory
   • note that the naming convention for this client is tag-validator-%02d sequentially. Ex. hadoop-validator-21
2. grab the run-list associated with the tag from the bootstrap databag
3. write a single shell script that does the following
   1. installs Ruby and Chef Gems using the same process as [http://wiki.opscode.com/display/chef/Knife+Bootstrap Knife bootstrap]
   2. mkdir /etc/chef
   3. write /etc/chef/client.rb, which sets the nodename as tag-ip_address Ex. hadoop-175.2.2.3
      • note that the ip address comes from ohai
   4. write /etc/chef/validation.pem associated with the client from step 1 above
   5. write /etc/chef/first-boot.json with the run-list from step 2 above
   6. execute chef-client -j /etc/chef/first-boot.json