Next generation, Opensource Performance Management


Tradional opensource performance management tools like Cacti, MRTG all have the same problem. Due to the way they store their metrics in multiple files they are unable to store all the data in due time. New techniques like timeTime-series databases are able to process metric data in a much faster way.Combined with CollectD, RabbitMQ and Grafana it is possible to create a enterpise level performance management solution.

This blog describes how these tools can be put together.

InfluxDB - CollectD

Requirements:

RedHat 7

 

Install InfluxDB

Add InfluxDB Yum repository:

 # cat <<EOF>/etc/yum.repos.d/influxdb.repo
 [influxdb]
 name = InfluxDB Repository - RHEL \$releasever
 baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
 enabled = 1
 gpgcheck = 1
 gpgkey = https://repos.influxdata.com/influxdb.key
 EOF

Install InfluxDB server

 # yum install influxdb

Configure the server

 # vi /etc/influxdb/influxdb.conf
 
 [collectd]
  enabled = true
  bind-address = "127.0.0.1:25826"
  database = "collectd"
  typesdb = "/usr/share/collectd/types.db"

Start the server

 # systemctl start influxdb

 

Install RabbitMQ

 # wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
 # yum --nogpgcheck localinstall epel-release-latest-7.noarch.rpm
 # yum install rabbitmq-server librabbitmq-tools librabbitmq
 # systemctl start rabbitmq-server

 

Install CollectD

 

 # yum install collectd collectd-amqp collectd-snmp
 # vi /etc/collectd.conf
 
 LoadPlugin amqp
 LoadPlugin snmp
 LoadPlugin network

 <Plugin amqp>
   <Subscribe "rabbitmq_collectd">
     Host "localhost"
     Port 5672
     VHost "/"
     User "guest"
     Password "guest"
     Exchange "amq.fanout"
     Queue "collectd"
     QueueDurable true
     QueueAutoDelete false
   </Subscribe>
 </Plugin>

 <Plugin network>
  Server "localhost" "25826"
  ReportStats true
 </Plugin>
 Include "/etc/collectd.d/

#vi /etc/collectd.d/snmp.conf
<Plugin snmp>
 <Data "hs_traffic">
   Type "if_octets"
   Table true
   Instance "IFMIB::ifName"
   Values "IFMIB::ifHCInOctets"  "IFMIB::ifHCOutOctets"
 </Data> 
</Plugin>
<Host "router">
  Address "192.168.1.1"
  Version 3
  SecurityLevel "authPriv"
  Username "snmpUser"
  AuthProtocol "MD5"
  AuthPassphrase "secret"
  PrivacyProtocol"DES"
  PrivacyPassphrase "topsecret"
  Collect "hs_traffic"
</Host>

Metrics can also be sent to the central CollectD server via the RabbitMQ server by connecting either one or more CollectD server that publish the metrics or the amqp-publish from the command line.

Configure CollectD for metrics-host

# vi /etc/collectd.conf
 
 LoadPlugin amqp
 LoadPlugin snmp

 <Plugin amqp>
   <Publish"rabbitmq_collectd">
     Host "rabbitmq-server"
     Port 5672
     VHost "/"
     User "guest"
     Password "guest"
     Exchange "amq.fanout"
     Queue "collectd"
     QueueDurable true
     QueueAutoDelete false
   </Publish>
 </Plugin>

Send metrict via command line

 $ /usr/bin/amqp-publish --url=amqp://guest:guest@rabbitmq-server:5672 --content-type="text/collectd" \
 > --routing-key "collectd" --body "PUTVAL metric-host/tablename/measurement 1457522933:23"

 

Install Grafana

# cat <<EOF>/etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecould.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

# yum install grafana
# chkconfig --add grafana-server
# service grafana-server start

 

Grafana defaults to connect to port 3000. So point your browser to http://<grafana-server>:3000. Click Datasources, then Add new.

grafana-ds

Now you can start adding new dashboards and panel.Create one you like. Then make a template from the JSON code and use this to generate scripted dashboards. The JSON code is to large to show here, but I’ll show a short perl code sample to read and update a dashboard.

 #!/usr/bin/env perl
 
 use REST::Client;
 use JSON::XS;
 use strict;

 my $apiKey = "eyJrIjoiNWUzZXh4bnBxbHZzWHVhaU8zak12MWJ0dmdJdHVTTDUiLCJuIjoiYXBpa2V5IiwiaWQiOjF9";
 my $coder = JSON::XS->new->pretty->allow_nonref;
 my $client = REST::Client->new();
 $client->addHeader("Aurhorization","Bearer ");
 $client->addHeader("Content-Type", "application/json");
 $client->addHeader("Accept", "application/json");
 $client->GET('http://<grafana-host>:3000/api/dashboards/db/mydash');

 my $json = $client->responseContent();
 my $dash = $coder->decode($json);
 $dash->{dashboard}->{version}++;
 $dash->{overwrite} = JSON::XS::true;

 $json = $coder->encode($dash);
 $client->POST('http://<grafana-host>:3000/api/dashboards/db', $json);

You can use this example to update your dashboard. Keep in mind that each panel has it’s own unique id within the dashboard.

References

Grafana API documentation

CollectD Homepage

InfluxDB Homepage

RabbitMQ Homepage

Leave a comment

Your email address will not be published. Required fields are marked *