When you are going to monitor your engine, you probably try first to find some tool that can do it. Very likely, of course, that tool should be able to handle monitoring online and to notify you when the health of your engine is starting to go bad. There are already several well-known and widespread online monitoring tools available for many systems. Super! But what if I try to find a suitable tool for my specific application and… ohh, failure – I cannot find what I need. Unfortunately, it’s also very likely that no one can even suggest a suitable tool. What can I do in such a situation? Probably try to do it myself. Who can help me? You’ve probably already guessed that I am talking about Monitis.  You may ask “why Monitis?”  Because Monitis suggests Monitis Open API, and that gives you a chance to build any monitoring tool yourself. In line with Monitis slang, such a monitor has been named the custom monitor.

But let’s cease the generic talk and get back to my story.

Recently, I had to use RabbitMQ server in one of my projects, and naturally I had a wish to monitor it; just to measure data that I want to monitor. RabbitMQ contains a nice plugin named RabbitMQ Management that in fact provides monitoring of RabbitMQ via any browser (thanks to the embedded WEBUI). Well, it is undoubtedly good, but I don’t want to sit whole days next to the monitor waiting for a problem to come in. I want to monitor online, to have the possibility to go back in time by viewing the monitoring history and, why not, to get a notification on my mobile while having a beer in the bar. I think every admin dreams of such a life.

Well, since I use an Ubuntu server I have decided to implement my custom monitor as Bash script to avoid any unnecessary dependencies and to take into account that the Bash script wrapper for Monitis API  is already implemented.

First of all I have to install RabbitMQ server. The easiest way to do this is to download the “deb” package from the original RabbitMQ site and install it by using the following command:

sudo dpkg -i rabbitmq-server_2.8.x_all.deb

So far so good. Next step – enabling the RabbitMQ Management HTTP API  that allows, in addition, getting necessary information by using REST technology. The management plugin has been included in the RabbitMQ distribution since version 2.8.1. To enable it, use the following command:

sudo rabbitmq-plugins enable rabbitmq_management

Please note that for older versions you have to install this plugin separately.

That’s all. Now I can use the RabbitMQ server by writing the following command to control RabbitMQ:

sudo /etc/init.d/rabbitmq-server {start|stop|status|rotate-logs|restart}

For instance, RabbitMQ server status will be shown as depicted below:

All is okay up to now. Well, after some investigation I have decided to measure the following available metrics:

•  osd_pr – The percentage of open socket descriptors RabbitMQ server to the allowed maximum number of open sockets by process.
•  ofd_pr – The percentage of open file descriptors RabbitMQ server to the allowed maximum number of open files by process.
•  cpu_usage – the percentage of cpu usage by RabbitMQ server.
•  mem_usage – the percentage of memory usage by RabbitMQ server.
•  msg_in_queue – the number of messages that are still in the queue.
•  timeout – queues timeout in seconds.
•  pub_rate – Average value of total published messages into queues per second.
•  from_client_rate – Total inbound throughput value estimated in Kbytes per second.
•  to_client_rate – Total outbound throughput value estimated in Kbytes per second.
•  get_rate – Average value of total got messages from queues per second.
•  status – the evaluation of health status of RabbitMQ server (OK, IDLE, NOK, FAIL)

The health status of RabbitMQ should be evaluated as ‘NOK’ when at least one of the following events is detected:

• The percentage of open file descriptors (ofd_pr) exceeds 90%
• The percentage of open socket descriptors (osd_pr) exceeds 90%
• The percentage of used Erlang processes to available Erlang processes exceeds 90%
• The percentage of memory usage (mem_usage) exceeds 95%
• The percentage of cpu usage (cpu_usage) exceeds 95%
• There are messages in the queue (msg_in_queue > 0)

The health status ‘FAIL’ should be generated when RabbitMQ server is unavailable for some reason and the health status ‘IDLE’ should be generated when RabbitMQ server isn’t receiving any messages from clients.

It seems that is all I want for now. Okay, I developed quite quickly such a Bash Script for monitoring and got the following set of files:

monitis_api.sh         Monitis API wrapper implementation

monitis_util.sh        Utilities function set

monitis_global.sh      Monitis API wrapper global variables

monitis_constant.sh    Monitis API constants

rabbitmq_monitor.sh    RabbitMQ custom monitor implementation

monitor_constant.sh    RabbitMQ monitor constants

rmqmon_start.sh        Main executable script

ticktick.sh            Bash JSON parser

Note that I have really developed only the “rabbitmq_monitor.sh” script. The other scripts were simply adapted. Please also notice that I have to use the third party open source JSON parser (to allow processing JSON in bash script) because RabbitMQ Management HTTP API responses are in the JSON form.

Well, now it’s time for testing.

I have prepared two clients on JavaScript (Node.js) and Java for connecting to RabbitMQ. In addition, the load simulator was prepared in a way that is intended to generate quite a big load for processing.

So, I have run the monitor and simulator and after some time opened my dashboard in Monitis. The tests showed nice results which I saw in the Monitis dashboard:

Beside this, double-clicking on any line leads to an alternate view which shows additional data about the RabbitMQ state at that moment.

Eventually, you can also see a graphical representation of your data:

I built a rule for notification by using Monitis dashboard features and, satisfied, went to rest.

Perfect, let me summarize. I have obtained in a very short time the desired tool and moreover it can send me a notification any time of day about any troubling situation in my RabbitMQ engine.

Finally, I have uploaded the monitor I created into GitHub where you can find more details about it.