Getting Started with Grafana and Collectd on Ubuntu 16.04 Server

Theres many companies out in the wild who run rather complex websites, yet have no idea what the health of their server is. This may be because monitoring tools have historically been straight out of the 90s (I’m looking at you RRDTool and Cacti) or just downright painfull to setup (Nagios anyone?). The good news is that there are now much better tools which are super easy to setup on a fresh machine.

End Goal

This is best summed up in a picture Grafana screenshot

Components

Grafana - This is the web UI tool which we will ultimately use to create our dashboards.

InfluxDb - This is the timeseries database which I’ll use for storing metrics.

Collectd - This is the deamon which will run on our host to collect various metrics.

Installation

I’m going to assume that you are running all of these commands as the root user. To become the root user on Ubuntu just type in sudo su at the console.

Grafana Installation

First we are going to install Grafana and make sure that component works. Installing Grafana is pretty easy once we get the repositories loaded into Ubuntu’s package manager. To load Grafana’s repositories run the following:

cat 'deb https://packagecloud.io/grafana/stable/debian/ jessie main' >> /etc/apt/sources.list
apt-get update
apt-get install grafana
service grafana-server start

At this point if you visit http://your-host:3000 a bare-bones install of Grafana should show up on your web browser. If for some reason it doesn’t you may have IP Tables set to block that port. If you have IP Tables setup to deny all incoming connections which aren’t explicitly allowed through (what I personally recommend) then you’ll also need to run the below to gain access to your page. If your local network isn’t in the 10.0.0.0/8 range then substitue the correct network in for the -d flag. I would not advise making this server accessible via the internet, however there will be a follow up post going over that process.

iptables -A INPUT -p tcp -m tcp -d 10.0.0.0/8 --destport 3000 -j ACCEPT

InfluxDB Installation

Grafana needs something to pull from, and one of the best ways of storing data which is sampled at set intervals is with a time series databse which is exactly what InfluxDB is. First off we need to install InfluxDB on our server.

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
apt-get update && apt-get install influxdb
systemctl start influxdb

All thats left to do is connect to Influx and create the collectd table to store our metrics.

influx
CREATE DATABASE collectd

Collectd Installation

Next up we need to setup the collectd deamon to actually gather some metrics. Collectd is included in the default Ubuntu repositories so installing this is as simple as a apt-get command.

apt-get install collectd

Configuration

Grafana is more or less configured out of the box. We’ll add a pre-built dashboard later but first lets get all of our datasources setup.

collectd

All we need to do is get collectd publishing over the network and enable a few plugins. To do this we simply need to enable the network plugin. Open up /etc/collectd.conf in your favorite editor and make the file look as shown below:

##############################################################################
# Plugin configuration                                                       #
#----------------------------------------------------------------------------#
# In this section configuration stubs for each plugin are provided. A desc-  #
# ription of those options is available in the collectd.conf(5) manual page. #
##############################################################################

 <Plugin network>
  Server "127.0.0.1" "8096"
 </Plugin>

Addiontally we need to enable a few plugins for our pre-built dashboard to work correctly. The plugins we need are: apache plugin, disk plugin, interface plugin, load plugin, memory plugin, uptime plugin, and the users plugin. Find the section of the config file that looks like the following and uncomment all of the mentioned plugins. Additionally a few of these require some addional configuration.

##############################################################################
# LoadPlugin section                                                         #
#----------------------------------------------------------------------------#
# Specify what features to activate.                                         #
##############################################################################

#LoadPlugin aggregation
#LoadPlugin amqp
LoadPlugin apache
#LoadPlugin apcups
#LoadPlugin ascent
#LoadPlugin barometer
LoadPlugin battery
#LoadPlugin bind
#LoadPlugin ceph
#LoadPlugin cgroups
#LoadPlugin conntrack
#LoadPlugin contextswitch
LoadPlugin cpu

To configure the apache plugin perform a search for <Plugin apache> and insert the following:

<Instance "apache">
        URL "http://localhost/server-status?auto"
        Host "localhost"
        Server "apache"
        User "www-user"
</Instance>

To configure the disk plugin perform a serach for <Plugin disk> and insert the values for your system. The only other plugin which needs configuration is the interface plugin. As you probably guessed perform a search for <Plugin interface> and make it match your system configuration. Addiontally you’ll need to uncomment the configurations for all of the above plugins as well.

Once you have that all set just run service collectd restart and then service collectd status to make sure that the service started up without issues.

InfluxDB

Configuring InfluxDB is a significantly easier than setting up collectd as theres not much to configure. All you need to do here in configure influx to pick up from the collectd feed. Perform a search for [[collectd]] and uncomment its configuration and make it match the following:

enabled = true
bind-address = "127.0.0.1:25826"
database = "collectd"
retention-policy = ""

# These next lines control how batching works. You should have this enabled
# otherwise you could get dropped metrics or poor performance. Batching
# will buffer points in memory if you have many coming in.

# Flush if this many points get buffered
batch-size = 5000

# Number of batches that may be pending in memory
batch-pending = 10

# Flush at least this often even if we haven't hit buffer limit
batch-timeout = "10s"

typesdb = "/opt/collectd/share/collectd/types.db"

After that run service influxdb restart and you should be good to go.

Grafana

Now for the fun part! If you naviagate back to your grafana install in your web browser and login as the admin user. The default username is ‘admin’ and the default password is also ‘admin’. The first thing we need to do is add Influx as a datasource, click the ‘add data source’ button and select ‘InfluxDB’ as the source. Assign a name to Influx though this doesn’t matter, under http settings just use the default of http://localhost:8086. Finall under InfluxDB details enter collectd as the database and click save.

Now to add the pre-built dashboard click on the Grafana logo in the upper left corner and hover over dashboards. On the pop-out menu select ‘Import’. The dashboard id we’ll be using is 558 so enter this in the Grafana.com dashboard field and click Load. Give your dashboard a name, select your influx source you just setup and click Import. Thats it you’re done!

You should now have a dashboard setup for Apache, and if you have other services feel free to checkout the Grafana dashboard repository at https://grafana.com/dashboards.