Monit with Slack notifications

Monit is a small Open Source utility for managing and monitoring Unix systems, Monit sends out notifications when an alert is triggered usually through emails . Since email is getting a bit old approach nowadays, it’s way more intuitive to receive alerts via push notifications on your smartphone or chat client. Slack provides supports to the many service integrations and also an Incoming Web-hooks integration. This way you can receive those Monit alerts in a specific channel. Enable the mobile Slack application on your phone, enable push notifications on that specific channel and you’re all set. In this blog, we demonstrate how this can be done by sending the notification to Slack.

Setup Prerequisites

• Monit service installed on the server
• A valid Slack account with the channel where you can enable the integration to receive the push notifications

Create an incoming webhooks

• Go to https://<yourteam>.slack.com/apps/manage/custom-integrations
• Click Incoming WebHooks
• Click Add Configuration
• Select an existing channel or create a new one (e.g. #monit) – you can change as per your preference
• Click Add Incoming WebHooks integration
• Copy the Webhook URL

Let’s create a ruby script that uses curl to POST a message to a channel on Slack. This shell script does a simple http POST onto the webhook with our alert:

[js]
#!/usr/local/bin/ruby

require ‘net/https’
require ‘json’

uri = URI.parse("https://hooks.slack.com/services/T02GN5HNP/xxxxxxxx/xxxxxxxxxxxxxxxxxxx")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Post.new(uri.request_uri, {‘Content-Type’ =&gt; ‘application/json’})
request.body = {
"channel" =&gt; "@channel_name",
"username" =&gt; "@username",
"icon_emoji" => ":.trollface:",
"text" =&gt; "[#{ENV[‘MONIT_HOST’]}] #{ENV[‘MONIT_SERVICE’]} – #{ENV[‘MONIT_DESCRIPTION’]}"
}.to_json
response = http.request(request)
puts response.body

[/js]

You can pick whatever Slack icon you want to use OR you can upload your own for an emoji and use that(supported by the Slack). Obviously, you’ll have to change the URI, channel, username and the like to what applies to you.

Save the script file on the instances with the name called slack.rb (remember to replace channel, username with your slack.com integration token). We can now use this script in our Monit control file to get Slack alerts if a check fails like below screenshot:

You will need to invoke this file whenever something happens, so to test it out, we’re going to create a Jenkins-slave file to monitor Jenkins-slave agent. Let create the sample file to monitor the Jenkins-slave process and PID file if any conditions fail then this will trigger slack.rb which we called through bash script /home/devops/notify.sh in /etc/monit.d/jenkins-slave.

[js]
check process jenkins-slave with pidfile /var/run/jenkins-slave.pid
start program = "/etc/init.d/jenkins-slave start"
stop program = "/etc/init.d/jenkins-slave stop"
if does not exist for 2 cycle
then exec "/bin/bash -c /home/devops/notify.sh"
else if succeeded for 2 cycle then exec "usr/local/bin/ruby /home/devops/slack.rb"
group jenkins-slave

check file jenkins_slave_pid with path /var/run/jenkins-slave.pid
if does not exist for 2 cycle
then exec "/bin/bash -c /home/devops/notify.sh"
else if succeeded for 2 cycle then exec "usr/local/bin/ruby /home/devops/slack.rb"
group jenkins-slave
[/js]

In Monit configuration file jenkins-slave calling /home/devops/notify.sh when any of the defined events is missing like jenkins-slave process or PID file it will trigger the notification on the Slack channel you mentioned in the slack.rb script and then start the jenkins-slave.

[js]
#!/bin/bash
/usr/local/bin/ruby /home/devops/slack.rb
/etc/init.d/jenkins-slave start
[/js]

After consecutive two successful cycles by Monit then it will trigger the status of the jenkins-slave on Slack channel.There’s a few second delay but check out Slack! You should see the message with your info all in there.