And finally… visualisation

Or, of course, visualization if you’re from that side of the pond 🙂

We’ve now got millions of lines of data pumping from a farm of servers across our network into a central logging server; we’re reading that data with logstash’s “file” input, we’re grokking it, applying logic, filtering, pulling data back out of Elasticsearch (if it exists) and finally dropping our grokked/filtered/analysed datastream into Elasticsearch.


Or… well, now we need the final piece of the puzzle. We could use any number of analytics tools, but those lovely Elasticsearch people have provided one already: Kibana.

After using the “default” dashboard, it was a reasonably simple step to producing a custom one. Essentially the steps are:

  • define some searches
  • use those searches in the available “panels”
  • draw pretty graphs and tables
  • profit! (Well, not strictly true, this is all in house)

We built up a simple dashboard, then added elements to it, then exported the dashboard to a local file which meant we could have a look inside it. As you may be aware, an email message can go to one or more people, so there are two distinct ways to view the data; to avoid visual overload, we decided to build two separate primary dashboards:

  • Dashboard aggregating messages
  • Dashboard aggregating recipients

We took the exported version, edited it, and saved it to two separate files in the Kibana dashboards directory (so we’re not relying on Elasticsearch to store them for us – pragmatism begins with backups).

Here’s the finished product – I’m writing this on a Saturday morning so things are pretty quiet, but it gives an idea of what’s being collected. Also worth mentioning that I’ve blurred the addresses here, because, well, you know. Privacy. You don’t need to know them!


And there we are!

[The above dashboard is available here, on Github]

Data visualisation, summarised, from the generation of the data to the web dashboards. I hope that helps someone else in the same state!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s