HA Cluster bootstrap Salt formula

Salt formula to bootstrap and manage a ClusterLabs high availability cluster.

Mainly adapted to Linux distributions for SUSE (it is based in crmsh), but it should be usable on other distributions with some modifications.

Features

The formula provides the capability to create and configure a multi node HA cluster. Here are some of the features:

• Initialize a cluster
• Join a node to an existing cluster
• Remove a node from an existing cluster
• Configure the pre-requirements (install required packages, configure ntp/chrony, create ssh-keys, etc)
• Auto detect if the cluster is running in a cloud provider (Azure, AWS, or GCP)
• Configure SBD
• Configure Corosync
• Configure the resource agents
• Install and configure the ha_cluster_exporter

Installation

The project can be installed in many ways, including but not limited to:

1. RPM
2. Manual clone

RPM

On openSUSE or SUSE Linux Enterprise use zypper package manager:

zypper install habootstrap-formula

Important! This will install the formula in /usr/share/salt-formulas/states/cluster. Make sure that /usr/share/salt-formulas/states entry is correctly configured in your Salt minion configuration file_roots entry if the formula is used in a masterless mode.

You can find the latest development repositories at SUSE's Open Build Service network:ha-clustering:sap-deployments:devel/habootstrap-formula.

Manual clone

git clone https://github.com/SUSE/habootstrap-formula
cp -R cluster /srv/salt

Important! The formulas depends on salt-shaptools package. Make sure it is installed properly if you follow the manual installation.

Usage

Follow the next steps to configure the formula execution. After this, the formula can be executed using master/minion or masterless options:

1. Modify the top.sls file (by default stored in /srv/salt) including the cluster entry.

   Here an example to execute the cluster formula in all of the nodes:

   # This file is /srv/salt/top.sls
   base:
     '*':
       - cluster
   

2. Customize the execution pillar file. Here an example of a pillar file for this formula with all of the options: pillar.example

3. Set the execution pillar file. For that, modify the top.sls of the pillars (by default stored in /srv/pillar) including the cluster entry and copy your specific cluster.sls pillar file in the same folder.

   Here an example to apply the recently created cluster.sls pillar file to all of the nodes:

   # This file is /srv/pillar/top.sls
   base:
     '*':
       - cluster
   

4. Execute the formula.

   1. Master/Minion execution.

      salt '*' state.highstate

   2. Masterless execution.

      salt-call --local state.highstate

Important! The hostnames and minion names of the cluster nodes need to be the same for the cluster join procedure to work correctly, and the nodes need to be able to reach each other by hostname/minion name.

Salt pillar encryption

Pillars are expected to contain private data such as user passwords required for the automated installation or other operations. Therefore, such pillar data need to be stored in an encrypted state, which can be decrypted during pillar compilation.

SaltStack GPG renderer provides a secure encryption/decryption of pillar data. The configuration of GPG keys and procedure for pillar encryption are desribed in the Saltstack documentation guide:

• SaltStack pillar encryption

• SALT GPG RENDERERS

Note:

• Only passwordless gpg keys are supported, and the already existing keys cannot be used.

• If a masterless approach is used (as in the current automated deployment) the gpg private key must be imported in all the nodes. This might require the copy/paste of the keys.

OBS Packaging

The CI automatically publishes new releases to SUSE's Open Build Service every time a pull request is merged into master branch. For that, update the new package version in _service and add the new changes in habootstrap-formula.changes.

The new version is published at:

• https://build.opensuse.org/package/show/network:ha-clustering:sap-deployments:devel/habootstrap-formula
• https://build.opensuse.org/package/show/openSUSE:Factory/habootstrap-formula (only if the spec file version is increased)

Test

The test folder contains a set of tests to check the integrity of the formula. The tests check if the provided pillar data is correctly rendered to find inconsistencies on the usage of the user input. The tests don't really check if the salt code works properly, they rather test if the formula uses and renders the states with the correct values.

In order to run the tests execute:

cd habootstrap-formula
bash ./test/validate-formula.sh

In order to improve or add new tests the pillar example from test/test_pillars can be changed (or add new pillar files).

Troubleshooting

Note: This advice is specific to openSUSE / SUSE distributions. For other distributions, the specific commands needed may be different.

To run the tests, libvirt must be installed and the daemon running:

zypper in libvirt
systemctl start libvirtd