ElastiCluster

ElastiCluster aims to provide a user-friendly command line tool to create, manage and setup computing clusters hosted on cloud infrastructures (like Amazon's Elastic Compute Cloud EC2 or Google Compute Engine) or a private OpenStack cloud). Its main goal is to get your own private cluster up and running with just a few commands; a YouTube video demoes the basic features of elasticluster.

Documentation for ElastiCluster is available on the Read The Docs website. General discussion over ElastiCluster's usage, features, and bugs takes also place on the [email protected] mailing-list (only subscribers can post).

This project is an effort of the Services and Support for Science IT (S3IT) unit at the University of Zurich, licensed under the GNU General Public License version 3.

Features

ElastiCluster is in active development, but offers the following features at the current state:

• Simple configuration file to define cluster templates
• Can start and manage multiple independent clusters at the same time

• Automated cluster setup:

  • use Debian GNU/Linux, Ubuntu, or CentOS as a base operating system
  • choose SLURM, Grid Engine or TORQUE+MAUI as a batch-queueing system
  • setup your Hadoop cluster to start your favorite map/reduce job
  • or create your IPython cluster to run your python code in parallel over multiple virtual machines
  • configure a distributed storage like GlusterFS or Ceph, or a parallel filesystem like OrangeFS (formerly known as PVFS2)
  • add useful tools like Ganglia for monitoring...
  • ...or anything that you can install with an Ansible playbook!

• Grow and shrink a running cluster

ElastiCluster is currently in active development: please use the GitHub issue tracker to file enhancement requests and ideas

We appreciate pull requests for new features and enhancements. Please use the master branch as starting point.

Quickstart

Installing from PyPI

ElastiCluster is a Python program; Python version 2.6 or 2.7 is required to run it.

It's quite easy to install elasticluster using pip; the command below is all you need to install elasticluster on your system:

pip install elasticluster

If you want to run elasticluster from source you have to install Ansible first:

pip install ansible
python setup.py install

Installing the development version from github

The source code of elasticluster is on GitHub, if you want to test the latest development version you should clone the github elasticluster repository.

You need the git command in order to be able to clone it, and we suggest you to use python virtualenv in order to create a controlled environment in which you can install elasticluster as normal user.

Assuming you already have virtualenv installed on your machine, you first need to create a virtualenv and install ansible, which is needed by elasticluster:

virtualenv elasticluster
. elasticluster/bin/activate
pip install ansible

Then you have to download the software. We suggest you to download it within the created virtualenv:

cd elasticluster
git clone git://github.com/gc3-uzh-ch/elasticluster.git src
cd src
python setup.py install

Now the elasticluster command should be available in your current environment.