Skip to content

Latest commit

 

History

History
175 lines (132 loc) · 10 KB

README.mediawiki

File metadata and controls

175 lines (132 loc) · 10 KB
Raw

This ZenPack allows for monitoring of RabbitMQ. See the Usage section for details on what is monitored. You can watch the Monitoring RabbitMQ video for a quick introduction that covers most of the details below.

Table of Contents

Gallery

<gallery widths="250px" heights="127px"> rabbitmq_components.png rabbitmq_exchanges.png rabbitmq_nodes_channels.png rabbitmq_nodes_throughput.png rabbitmq_nodes.png rabbitmq_queues_metrics.png rabbitmq_queues.png rabbitmq_vhosts.png </gallery>

Related ZenPacks

There already exist at least two community Zenpacks that provide monitoring for RabbitMQ.

  • ZenPacks.dnalley.AMQPEventMonitor by David Nalley: Very different functionality than what's provided by this ZenPack. It allows you to pull messages from a defined queue and automatically turn them into Zenoss events.
  • ZenPacks.community.RabbitMQ by Greg Guthe: More similar to this ZenPack in its functionality. Global metrics for queued messages and rates. It appears to require that the HTTP management API plugin be installed into your RabbitMQ instances, and that a Net-SNMP extension also written by Greg Guthe be installed.
The major differences between the ZenPacks.community.RabbitMQ and this pack are that this pack simply runs various rabbitmqctl commands over SSH both to model the node, vhosts, exchanges and queues; as well as to monitor connection, channel and per-queue metrics. So you don't need to install anything extra on your RabbitMQ server, and you get more granularity on the monitoring.

In the future this pack might be extended to also support RabbitMQ's HTTP management API plugin in addition to the SSH method.

Usage

Installing the ZenPack will add the following objects to your Zenoss system.

Modeling Plugins
  • zenoss.ssh.RabbitMQ
Monitoring Templates
  • RabbitMQNode in /Devices
  • RabbitMQQueue in /Devices
Event Classes
  • /Status/RabbitMQ
  • /Perf/RabbitMQ
Configuration Properties
  • zRabbitMQAdminUser
  • zRabbitMQAdminPassword
Command Parsers
  • ZenPacks.zenoss.RabbitMQ.parsers.RabbitMQCTL
  • ZenPacks.zenoss.RabbitMQ.parsers.RabbitMQAdmin
These monitoring templates should not be bound directly to any devices in the system.

To start monitoring your RabbitMQ server you will need to setup SSH access so that your Zenoss collector server will be able to SSH into your RabbitMQ server(s) as a user who has permission to run the rabbitmqctl command. This almost always means the root user. See the Using a Non-Root User section below for instructions on allowing non-root users to run rabbitmqctl. To do this you need to set the following zProperties for the RabbitMQ devices or their device class in Zenoss.

  • zCommandUsername
  • zCommandPassword
  • zKeyPath
The zCommandUsername property must be set. To use public key authentication you must verify that the public portion of the key referenced in zKeyPath is installed in the `~/.ssh/authorized_keys` file for the appropriate user on the RabbitMQ server. If this key has a passphrase you should set it in the zCommandPassword property. If you'd rather use password authentication than setup keys, simply put the user's password in the zCommandPassword property.

You should then add the zenoss.ssh.RabbitMQ modeler plugin to the device, or device class containing your RabbitMQ servers and remodel the device(s). This will automatically find the node, vhosts, exchanges and queues and begin monitoring them immediately for the following metrics.

Node Values
  • Status - Running or not? Generates event on failure.
  • Open Connections & Channels
  • Sent & Received Bytes Rate
  • Sent & Received Packets Rate
  • Depth of Send Queue
  • Consumers
  • Unacknowledged & Uncommitted Messages
Queue Values
  • Ready, Unacknowledged & Total Messages
  • Memory Usage
  • Consumers
  • Incoming & Outgoing Message Rates
There is a default threshold of 1,000,000 messages per queue. This is almost certainly an absurdly high threshold that shouldn't trip in normal systems. However, by clicking into the details of any individual queue you can set the per-queue threshold to a more reasonable value that makes sense for a given queue.

RabbitMQAdmin Update
Version 1.0.7+ of this ZenPack adds two additional zProperties as well as an additional parser and provides the ability to monitor incoming and outgoing message rates per queue using rabbitmqadmin.

For detailed instructions on configuring and using rabbitmqadmin, please refer to the Using RabbitMQAdmin section, below.

Using a Non-Root User

This ZenPack requires the ability to run the rabbitmqctl command remotely on your RabbitMQ server(s) using SSH. By default, the rabbitmqctl command is only allowed to be run by the root and rabbitmq users. Furthermore, this ZenPack expects the rabbitmqctl command be in the user's path. Normally this is only true for the root user.

There's a very good reason for this restriction. Once a user is allowed to execute the ``rabbitmqctl`` command, they are able to perform the following actions.

  • Stop, Start or Reset RabbitMQ
  • Control a RabbitMQ Cluster
  • Close Open Connections
  • Manage Users and Security
  • Manage VHosts
In a nutshell, this means that any user with permission to run rabbitmqctl can wreak total havoc on your RabbitMQ server if they had the intent to do so.

Assuming that you've created a user named zenmonitor on your RabbitMQ servers for monitoring purposes, you can follow these steps to allow the zenmonitor user to run rabbitmqctl.

  1. Install the sudo package on your server.
  2. Make sudo not require a TTY. This allows sudo to be run via ssh.
    1. Run visudo as root.
    2. Find a line containing Defaults requiretty and comment it out by prefixing the line with a #.
    3. Type ESC then :wq to save the sudo configuration.
  3. Allow the *zenmonitor* user to run rabbitmqctl.
    1. Run visudo as root.
    2. Add the following line to the bottom of the file.
      zenmonitor ALL=(ALL) NOPASSWD: /usr/sbin/rabbitmqctl
    3. Type ESC then :wq to save the sudo configuration.
  4. Alias rabbitmqctl for the zenmonitor user.
    1. Add the following lines to /home/zenmonitor/.bashrc.
      shopt -s expand_aliases
      alias rabbitmqctl="sudo /usr/sbin/rabbitmqctl"

Using RabbitMQAdmin

Rabbitmqadmin is a script provided by RabbitMQ included with the rabbitmq_management plugin. Rabbitmqadmin can perform a variety of tasks and provide performance statistics for RabbitMQ. To install rabbitmqadmin, the following conditions must be met on each RabbitMQ server you want to monitor:

  1. Erlang R14B or greater must installed
  2. A RabbitMQ user with administrator tags must be used (will be stored in zProperties)
  3. The rabbitmq_management plugin must be enabled
  4. The rabbitmqadmin script must be saved to the RabbitMQ server filesystem
1. Erlang
If the installed version of Erlang on each RabbitMQ server is < R14B, Erlang must first be updated. Please follow whatever requirements are needed to update Erlang for each RabbitMQ server(s). On RHEL/CentOS, for instance, you can check the installed version of Erlang with the command rpm -qa | grep -i erlang.

If you are updating Erlang on a RHEL/CentOS server (with or without Zenoss): 

 * Download an updated Erlang version for your platform - we recommend version R15B03 (wget http://packages.erlang-solutions.com/site/esl/esl-erlang/FLAVOUR_1_general/esl-erlang_15.b.3-1~centos~6_amd64.rpm)
 * Download an Erlang compat RPM (wget https://github.com/jasonmcintosh/esl-erlang-compat/blob/master/rpmbuild/RPMS/noarch/esl-erlang-compat-R14B-1.el6.noarch.rpm?raw=true --no-check-certificate)
 * If zenoss is present:  service zenoss stop
 * service rabbitmq-server stop
 * Remove your previous version of Erlang - if your version of Erlang is R12B, you can use the command: rpm -e erlang-R12B --nodeps
 * yum localinstall esl-erlang_15.b.3-1~centos~6_amd64.rpm
 * yum localinstall esl-erlang-compat-R14B-1.el6.noarch.rpm
 * service rabbitmq-server start
 * If zenoss is present: service zenoss start

2. Give a RabbitMQ user administrator tags (and configure needed zProperties)
Rabbitmqadmin uses a user to authenticate with Rabbit for performing operations. If you are using a non-root user or have additional RabbitMQ users that you will use for rabbitmqadmin monitoring, you must give them rabbitmq adminstrator permissions. In the example below, we will be using the "zenoss" RabbitMQ user to run rabbitmqadmin commands (replace as appropriate): 

 * rabbitmqctl set_user_tags zenoss administrator
 * In the Zenoss UI, for your RabbitMQ device, change the Configuration Property '''zSshConcurrentSessions''' to 1
 * In the Zenoss UI, for your RabbitMQ device, set your '''zRabbitMQAdminUser''' and '''zRabbitMQAdminPassword''' appropriately

3. Enable the RabbitMQ management plugin
 * rabbitmq-plugins enable rabbitmq_management
 * service rabbitmq-server restart

4. Download the rabbitmqadmin script
1. Download the rabbitmqadmin script to /usr/local/bin/: 

 * cd /usr/local/bin
 * RabbitMQ 2.x: wget http://{your-rabbitmq-server}:55672/cli/rabbitmqadmin --user {your-rabbitmq-user} --password {your-rabbitmq-user-password}
 * RabbitMQ 3.x: wget http://{your-rabbitmq-server}:15672/cli/rabbitmqadmin --user {your-rabbitmq-user} --password {your-rabbitmq-user-password}

2. Make the rabbitmqadmin file executable and update ownership: 

 * chmod +x /usr/local/bin/rabbitmqadmin
 * If you are running the SSH commands as a non-root user, chown (your user):(your group) /usr/local/bin/rabbitmqadmin

Change Log

1.0.10
  • (ZPS-6912) Fix Traceback generated when no queries are being returned for the vhost name
  • (ZPS-6937) Fix RabbitMQ 3.8.x targets break parsing
  • Tested with Zenoss Cloud, Zenoss 6.6.0, Zenoss 6.5.0 and Service Impact 5.5.3
1.0.9
  • (ZPS-2772) Fix modeler omission of RabbitMQ nodes from rabbitmqctl status
1.0.8
  • (ZEN-20898) Support for Federation Plugin
  • Parser improvements and bug fixes
1.0.7
  • (ZEN-11451) Zenoss should track and present event processing rates
1.0.6
  • (ZEN-5533) /Status/RabbitMQ for queues don't auto-clear when resolved
1.0.5
  • (ZEN-3526) RabbitMQ: No data returned for command