I am setting up a DynamoDB Stream to Elasticsearch Cluster via Logstash and will be documenting my escapades here. Starting with installing Logstash on a brand new minted EC2 micro instance; because I love free tier especially for evaluating software solutions.
Install Logstash
Setup yum
Import public key
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
Configure yum
Open /etc/yum.repos.d/logstash.repo using your favorite text editor and this configuration.
[logstash-2.2]
name=Logstash repository for 2.2.x packages
baseurl=http://packages.elastic.co/logstash/2.2/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
Install
yum install logstash
Loaded plugins: priorities, update-motd, upgrade-helper
logstash-2.2 | 951 B 00:00
logstash-2.2/primary | 2.2 kB 00:00
logstash-2.2 3/3
Resolving Dependencies
--> Running transaction check
---> Package logstash.noarch 1:2.2.2-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================
Installing:
logstash noarch 1:2.2.2-1 logstash-2.2 72 M
Transaction Summary
============================================================================================================================================
Install 1 Package
Total download size: 72 M
Installed size: 126 M
Is this ok [y/d/N]: y
Downloading packages:
logstash-2.2.2-1.noarch.rpm | 72 MB 00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:logstash-2.2.2-1.noarch 1/1
Verifying : 1:logstash-2.2.2-1.noarch 1/1
Installed:
logstash.noarch 1:2.2.2-1
Complete!
See nice sweet and easy installation; gone are the days of installing software that takes hours. Try some from source C++/C applications on Linux for some fun challenges!
Install dependencies for DynamoDB plugin
All we really need is git.
yum -y install git
Install plugin
git clone https://github.com/awslabs/logstash-input-dynamodb.git
cd logstash-input-dynamodb/
/opt/logstash/vendor/jruby/bin/jruby -S gem install bundler
/opt/logstash/vendor/jruby/bin/jruby -S bundle install
/opt/logstash/vendor/jruby/bin/jruby -S gem build logstash-input-dynamodb.gemspec
/opt/logstash/vendor/jruby/bin/jruby -S gem install logstash-input-dynamodb-1.0.0-java.gem
/opt/logstash/bin/plugin install --no-verify logstash-input-dynamodb
Finish building logstash
cd /opt/logstash
/opt/logstash/vendor/jruby/bin/jruby --2.0 -S gem install mime-types-data -v '3.2016.0221'
/opt/logstash/vendor/jruby/bin/jruby --2.0 -S gem install mime-types -v '3.0'
/opt/logstash/vendor/jruby/bin/jruby -S bundle install
Configure Logstash
/etc/logstash/conf.d/logstash-dynamodb.conf
input {
dynamodb{
endpoint => "dynamodb.us-east-1.amazonaws.com"
streams_endpoint => "streams.dynamodb.us-east-1.amazonaws.com"
view_type => "new_and_old_images"
aws_access_key_id => ""
aws_secret_access_key => ""
table_name => "table_name"
}
}
output {
elasticsearch {
hosts => "https://search-abc-defeg.us-east-1.es.amazonaws.com"
}
stdout { }
}
Test and start Logstash
logstash -f /etc/logstash/conf.d/logstash-dynamodb.conf
If all goes well above then you just need to start the service with
service logstash start