Welcome!

Eclipse Authors: David H Deans, Mano Marks, Liz McMillan, JP Morgenthal, Yeshim Deniz

Blog Feed Post

How I built a monitor for RabbitMQ

When you are going to monitor your engine, you probably try first to find some tool that can do it. Very likely, of course, that tool should be able to handle monitoring online and to notify you when the health of your engine is starting to go bad. There are already several well-known and widespread online monitoring tools available for many systems. Super! But what if I try to find a suitable tool for my specific application and… ohh, failure – I cannot find what I need. Unfortunately, it’s also very likely that no one can even suggest a suitable tool. What can I do in such a situation? Probably try to do it myself. Who can help me? You’ve probably already guessed that I am talking about Monitis.  You may ask “why Monitis?”  Because Monitis suggests Monitis Open API, and that gives you a chance to build any monitoring tool yourself. In line with Monitis slang, such a monitor has been named the custom monitor.

But let’s cease the generic talk and get back to my story.

Recently, I had to use RabbitMQ server in one of my projects, and naturally I had a wish to monitor it; just to measure data that I want to monitor. RabbitMQ contains a nice plugin named RabbitMQ Management that in fact provides monitoring of RabbitMQ via any browser (thanks to the embedded WEBUI). Well, it is undoubtedly good, but I don’t want to sit whole days next to the monitor waiting for a problem to come in. I want to monitor online, to have the possibility to go back in time by viewing the monitoring history and, why not, to get a notification on my mobile while having a beer in the bar. I think every admin dreams of such a life.

Well, since I use an Ubuntu server I have decided to implement my custom monitor as Bash script to avoid any unnecessary dependencies and to take into account that the Bash script wrapper for Monitis API  is already implemented.

First of all I have to install RabbitMQ server. The easiest way to do this is to download the “deb” package from the original RabbitMQ site and install it by using the following command:

 

sudo dpkg -i rabbitmq-server_2.8.x_all.deb

 

So far so good. Next step – enabling the RabbitMQ Management HTTP API  that allows, in addition, getting necessary information by using REST technology. The management plugin has been included in the RabbitMQ distribution since version 2.8.1. To enable it, use the following command:

 

sudo rabbitmq-plugins enable rabbitmq_management

 

Please note that for older versions you have to install this plugin separately.

That’s all. Now I can use the RabbitMQ server by writing the following command to control RabbitMQ:

 

sudo /etc/init.d/rabbitmq-server {start|stop|status|rotate-logs|restart}

 

For instance, RabbitMQ server status will be shown as depicted below:

 

 

All is okay up to now. Well, after some investigation I have decided to measure the following available metrics:

  •  osd_pr – The percentage of open socket descriptors RabbitMQ server to the allowed maximum number of open sockets by process.
  •  ofd_pr – The percentage of open file descriptors RabbitMQ server to the allowed maximum number of open files by process.
  •  cpu_usage – the percentage of cpu usage by RabbitMQ server.
  •  mem_usage – the percentage of memory usage by RabbitMQ server.
  •  msg_in_queue – the number of messages that are still in the queue.
  •  timeout – queues timeout in seconds.
  •  pub_rate – Average value of total published messages into queues per second.
  •  from_client_rate – Total inbound throughput value estimated in Kbytes per second.
  •  to_client_rate – Total outbound throughput value estimated in Kbytes per second.
  •  get_rate – Average value of total got messages from queues per second.
  •  status – the evaluation of health status of RabbitMQ server (OK, IDLE, NOK, FAIL)

 

The health status of RabbitMQ should be evaluated as ‘NOK’ when at least one of the following events is detected:

  • The percentage of open file descriptors (ofd_pr) exceeds 90%
  • The percentage of open socket descriptors (osd_pr) exceeds 90%
  • The percentage of used Erlang processes to available Erlang processes exceeds 90%
  • The percentage of memory usage (mem_usage) exceeds 95%
  • The percentage of cpu usage (cpu_usage) exceeds 95%
  • There are messages in the queue (msg_in_queue > 0)

The health status ‘FAIL’ should be generated when RabbitMQ server is unavailable for some reason and the health status ‘IDLE’ should be generated when RabbitMQ server isn’t receiving any messages from clients.

It seems that is all I want for now. Okay, I developed quite quickly such a Bash Script for monitoring and got the following set of files:

monitis_api.sh         Monitis API wrapper implementation

monitis_util.sh        Utilities function set

monitis_global.sh      Monitis API wrapper global variables

monitis_constant.sh    Monitis API constants

rabbitmq_monitor.sh    RabbitMQ custom monitor implementation

monitor_constant.sh    RabbitMQ monitor constants

rmqmon_start.sh        Main executable script

ticktick.sh            Bash JSON parser

Note that I have really developed only the “rabbitmq_monitor.sh” script. The other scripts were simply adapted. Please also notice that I have to use the third party open source JSON parser (to allow processing JSON in bash script) because RabbitMQ Management HTTP API responses are in the JSON form.

 

Well, now it’s time for testing.

I have prepared two clients on JavaScript (Node.js) and Java for connecting to RabbitMQ. In addition, the load simulator was prepared in a way that is intended to generate quite a big load for processing.

So, I have run the monitor and simulator and after some time opened my dashboard in Monitis. The tests showed nice results which I saw in the Monitis dashboard:

 

Beside this, double-clicking on any line leads to an alternate view which shows additional data about the RabbitMQ state at that moment.

 

Eventually, you can also see a graphical representation of your data:

 

I built a rule for notification by using Monitis dashboard features and, satisfied, went to rest.

Perfect, let me summarize. I have obtained in a very short time the desired tool and moreover it can send me a notification any time of day about any troubling situation in my RabbitMQ engine.

Finally, I have uploaded the monitor I created into GitHub where you can find more details about it.

 

 

 

Share Now:del.icio.usDiggFacebookLinkedInBlinkListDZoneGoogle BookmarksRedditStumbleUponTwitterRSS

Read the original blog entry...

More Stories By Hovhannes Avoyan

Hovhannes Avoyan is the CEO of PicsArt, Inc.,

@ThingsExpo Stories
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
In his General Session at 17th Cloud Expo, Bruce Swann, Senior Product Marketing Manager for Adobe Campaign, explored the key ingredients of cross-channel marketing in a digital world. Learn how the Adobe Marketing Cloud can help marketers embrace opportunities for personalized, relevant and real-time customer engagement across offline (direct mail, point of sale, call center) and digital (email, website, SMS, mobile apps, social networks, connected objects).
SYS-CON Events announced today that Tintri, Inc, a leading provider of enterprise cloud infrastructure, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Tintri offers an enterprise cloud platform built with public cloud-like web services and RESTful APIs. Organizations use Tintri all-flash storage with scale-out and automation as a foundation for their own clouds – to build agile development environments...
Have you ever noticed how some IT people seem to lead successful, rewarding, and satisfying lives and careers, while others struggle? IT author and speaker Don Crawley uncovered the five principles that successful IT people use to build satisfying lives and careers and he shares them in this fast-paced, thought-provoking webinar. You'll learn the importance of striking a balance with technical skills and people skills, challenge your pre-existing ideas about IT customer service, and gain new in...
SYS-CON Events announced today that Outlyer, a monitoring service for DevOps and operations teams, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outlyer is a monitoring service for DevOps and Operations teams running Cloud, SaaS, Microservices and IoT deployments. Designed for today's dynamic environments that need beyond cloud-scale monitoring, we make monitoring effortless so you...
SYS-CON Events announced today that Outlyer, a monitoring service for DevOps and operations teams, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outlyer is a monitoring service for DevOps and Operations teams running Cloud, SaaS, Microservices and IoT deployments. Designed for today's dynamic environments that need beyond cloud-scale monitoring, we make monitoring effortless so you ...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
Why do your mobile transformations need to happen today? Mobile is the strategy that enterprise transformation centers on to drive customer engagement. In his general session at @ThingsExpo, Roger Woods, Director, Mobile Product & Strategy – Adobe Marketing Cloud, covered key IoT and mobile trends that are forcing mobile transformation, key components of a solid mobile strategy and explored how brands are effectively driving mobile change throughout the enterprise.
Artificial Intelligence has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming Cloud Expo | @ThingsExpo at the Javits Center in New York, June 6-8, will find fresh new content in a new track called Cognitive Computing | Artificial Intelligence, Machine Learning, Deep Learning. Cloud Expo is still accepting submissions for this new track, so please visit www.cloudcomputingexpo.com for the latest information. 2...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists peeled away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud enviro...
SYS-CON Events announced today that Hitrons Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Hitrons Solutions Inc. is distributor in the North American market for unique products and services of small and medium-size businesses, including cloud services and solutions, SEO marketing platforms, and mobile applications.
SYS-CON Events announced today that TMC has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo and Big Data at Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY, , and the 21st International Cloud Expo, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Global buyers rely on TMC’s content-driven marketplaces to make purchase decisions and navigate markets. Learn how we can he...
"I think that everyone recognizes that for IoT to really realize its full potential and value that it is about creating ecosystems and marketplaces and that no single vendor is able to support what is required," explained Esmeralda Swartz, VP, Marketing Enterprise and Cloud at Ericsson, in this SYS-CON.tv interview at @ThingsExpo, held June 7-9, 2016, at the Javits Center in New York City, NY.
In his session at @ThingsExpo, Steve Wilkes, CTO and founder of Striim, will delve into four enterprise-scale, business-critical case studies where streaming analytics serves as the key to enabling real-time data integration and right-time insights in hybrid cloud, IoT, and fog computing environments. As part of this discussion, he will also present a demo based on its partnership with Fujitsu, highlighting their technologies in a healthcare IoT use-case. The demo showcases the tracking of patie...