Setting Up the ELK Stack on AWS

At some point "grep through SSH" stops being a log analysis strategy. For me that point was around the third time I was trying to correlate Apache errors across six instances by hand at 11pm. The ELK stack — Elasticsearch, Logstash, Kibana — is the standard answer. AWS has a managed Elasticsearch Service that removes most of the operational pain, so that's what we'll use, with Logstash running on an EC2 instance to handle log shipping and parsing.

What Each Piece Does

• Elasticsearch: Stores and indexes logs. Full-text search, aggregations, near-real-time querying.
• Logstash: Ingests logs from various sources, parses and transforms them (grok patterns, date parsing, field extraction), ships to Elasticsearch.
• Kibana: The UI. Search logs, build dashboards, visualize aggregations.

The flow is: your servers → Logstash → Elasticsearch → Kibana.

AWS Elasticsearch Service

Go to the AWS console, find Elasticsearch Service, and create a domain. For a small setup:

• Elasticsearch version: 5.5
• Instance type: m3.medium.elasticsearch (t2 instances don't support dedicated master)
• Instance count: 2 (minimum for any kind of availability)
• Storage: 20GB EBS per node to start

Set the access policy. The easiest option for internal use is IP-based access control so your Logstash instance and your VPN IP can reach it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "*"},
      "Action": "es:*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "10.0.0.0/8",
            "203.0.113.42"
          ]
        }
      },
      "Resource": "arn:aws:es:us-east-1:123456789012:domain/my-logs/*"
    }
  ]
}

After a few minutes you'll have an endpoint like https://search-my-logs-abc123.us-east-1.es.amazonaws.com. That's your Elasticsearch URL.

Setting Up Logstash on EC2

Launch a t2.medium running Amazon Linux. Install Java first (Logstash requires it), then Logstash 5.x:

$ sudo yum install -y java-1.8.0-openjdk
$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ sudo vim /etc/yum.repos.d/logstash.repo

[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

$ sudo yum install logstash

The Logstash Config

Create /etc/logstash/conf.d/apache.conf. This reads Apache access logs, parses them with grok, and ships to Elasticsearch:

input {
  file {
    path => "/var/log/httpd/access_log"
    start_position => "beginning"
    type => "apache_access"
  }
  file {
    path => "/var/log/httpd/error_log"
    start_position => "beginning"
    type => "apache_error"
  }
}

filter {
  if [type] == "apache_access" {
    grok {
      match => {
        "message" => "%{COMBINEDAPACHELOG}"
      }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    mutate {
      convert => { "bytes" => "integer" }
      convert => { "response" => "integer" }
    }
    geoip {
      source => "clientip"
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://search-my-logs-abc123.us-east-1.es.amazonaws.com:443"]
    index => "apache-access-%{+YYYY.MM.dd}"
    ssl => true
    ssl_certificate_verification => true
  }
}

The %{COMBINEDAPACHELOG} grok pattern handles standard Apache log format automatically. The date filter parses Apache's timestamp format so Elasticsearch gets a proper @timestamp field. Daily indices (apache-access-2017.07.27) keep things manageable and let you delete old data easily.

Test the config before starting the service:

$ sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/apache.conf
Configuration OK
$ sudo systemctl start logstash
$ sudo systemctl enable logstash

Connecting Kibana

With AWS Elasticsearch Service, Kibana is included at /_plugin/kibana on your domain endpoint. Navigate to:

https://search-my-logs-abc123.us-east-1.es.amazonaws.com/_plugin/kibana

On first load, Kibana asks you to configure an index pattern. Type apache-access-* and select @timestamp as the time field. Kibana will detect your fields automatically.

A Useful Query

Once data is flowing, try this in Kibana's Discover view to find all 5xx errors in the last hour:

response:[500 TO 599]

Or in the API directly:

GET apache-access-*/_search
{
  "query": {
    "bool": {
      "filter": [
        { "range": { "response": { "gte": 500 } } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "aggs": {
    "errors_by_path": {
      "terms": { "field": "request.keyword", "size": 10 }
    }
  }
}

That shows you the top 10 endpoints throwing server errors in the past hour. This kind of query takes seconds in Elasticsearch and about 45 minutes of grep piping to approximate manually.

Building a Basic Dashboard

In Kibana, go to Visualize and create a few basics:

1. Line chart — count of requests over time (split by response field)
2. Pie chart — breakdown by HTTP status code
3. Data table — top IP addresses by request count

Pin them to a Dashboard. Save it. Now you have something useful to look at during an incident instead of a terminal full of grep output.

Wrapping Up

The AWS managed Elasticsearch Service is genuinely nice — no cluster management, automatic snapshots, and Kibana included. The painful part is always getting Logstash configured correctly; grok patterns in particular have a learning curve. The grok debugger in Kibana's Dev Tools is your friend there. Once you have a working config for one log type, adapting it to others is straightforward.