Here’s the science bit, #1 – first steps

Now things get a little more complicated.

We have a farm of machines, split across two different roles – primarily MX (inbound email from the outside world), and MTA (passing email back to the outside world, but also between various systems within the organisation).

We’ve been using the venerable syslog protocol for years to aggregate Exim‘s logs from across the farm onto a central syslog platform. This is a stable system, and I didn’t want to change the configuration of rsyslog on the collector to add in logstash.

Thankfully, logstash has a number of input modules so I could simply hook it into the existing files by defining an input as follows:

input {
  file {
    path => "/path/to/exim/mainlog"
    start_position => 'end'
    sincedb_path => "/dev/null"
  }
}

Pretty simple.

At first I ran it with a simple “take everything and pretty print it to stdout” output block:

output {
  stdout {
    codec => rubydebug
  }
}

Also pretty simple. Data started flying past – bear in mind we’re talking between 5 and 10 million log lines a day on average here, so parking this on the raw input data returned a stream of beautifully formatted JSON objects that were far, far too fast to read. But that wasn’t the point – I had it reading data.

Now all I needed to do was pre-analyse the data to generate relevant fields/tags for Elasticsearch/Kibana to work with. Well, I say “all”…

 

Advertisements

7 comments

  1. Dear, help me how to configure exim logs in NLS. ?
    I’ve added in NLS:
    File#1 000_inputs.conf
    input {
    file {
    path => “/var/log/exim_mainlog”
    start_position => ‘end’
    sincedb_path => “/dev/null”
    }
    }

    File#2 999_outputs.conf
    output {
    stdout {
    codec => rubydebug
    }
    }

    But getting error while restart

    • I’m afraid I have no idea what “NLS” refers to, but from a general perspective if you have an error as you say then… perhaps providing the error in your question would be a good idea?

    • I’m afraid you’ll need to be looking into this in more detail; Logstash itself has its own logs – you’ll likely find the error in there.

    • Hi – no, there’s no point doing that. If you’re having specific problems with NLS, you need to start back at basics – follow the install and configure instructions that come with it, get it working with something basic, and build on it from there.

      If I did it for you I’d be doing exactly the same thing, because I’ve never used it. I would learn, you would not, and that will not help you in the long run.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s