randy/randy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
randy/ELK-stack.md

6.8 KiB
Raw Permalink Blame History

Setting Up the ELK Stack on a Bro Cluster

Assume a running bro system, single or cluster. We want to get the ELK Stack (Elastisearch, Logstash, Kibana) running to give us a GUI view into the bro log data.

This example is on a Ubuntu 16.04 VM running bro 2.5. A lot of hints were taken from a cheat sheet Fakrul Alam gave me.

Assumptions:

  • We run as the bro user
  • The umask for bro and root are both 022 (I lost hours to this)
  • bro is the default installation, see https://git.rg.net/randy/randy/src/master/bro-cluster.md
  • We prefer packages to fetching files so unattended-upgrade will keep it fresh.
  • We take defaults as much as possible
  • bash is the assumed CLI

Install Java 8

A lot of this is written in Java. The OpenJDK that comes as a Ubuntu package is just fine.

sudo apt install default-jdk

Install Elasticsearch (from)

First get the key file and sources.d file to all the ELK packaged components. We will need them again for Logstash and Kibana.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Now we can just install it

sudo apt update
sudo apt install elasticsearch

You will need to make at least two changes to /etc/elasticsearch/elasticsearch.yml.

#cluster.name: my-application
#node.name: node-1

Then perform the systemdeath rituals

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

kopf Plugin (warning: not tested!)

If you also want the kopf plugin, it looks to be something such as this

git clone git@github.com:lmenezes/elasticsearch-kopf.git
sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install elasticsearch-kopf/v2.1.2

Install Logstash (from)

As you already have the key and sources files retrieved above, you should only need to do the following:

sudo apt install logstash

Install Translation Filter

cd /usr/share/logstash
sudo bin/logstash-plugin install logstash-filter-translate
cd

Install GeoIP Plugin

Logstash wants to add GeoIP data to IP addresses. So we need to get the MaxMind database and install it

cd /etc/logstash
sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
sudo gunzip GeoLiteCity.dat.gz

You are probably going to wnt to do this every month or two

Install Fakrul Alam's Filters to Clean bro Logfiles

Logstash needs to digest bro log files that have had headers and other bro-isms removed. Fakrul Alam has provided a nice set of filters. Install them as follows (excuse my fancy rsync options):

git clone git@github.com:fakrul/bro-elk.git

Edit the "path ->" lines in each of the files to match your configuration. At the same time, copy the filters to where they belong.

cd bro-elk
for i in *conf; do sudo sed -s 's.opt/nsm.usr/local.' $i > /etc/logstash/conf.d/$i; done
cd ..

Check that the perms are OK; they should be bro:bro 644

ls -l /etc/logstash/conf.d

And perform the systemdeath rituals

sudo systemctl daemon-reload
sudo systemctl enable logstash.service
sudo systemctl start logstash.service

Check the Logstash Set-up

First, a simple test

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -e
output {
  stdout {}
}
^D

And watch the log

tail -f /var/log/logstash/logstash-plain.log

If that looks good, run the following command so check if everything is working fine:

sudo -u logstash /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d --debug

There will be a loooong pause and then a lot of debug output. Watch for ERRORs. I have no idea what the output means. I think that, if it does not halt and catch fire, you're good to go.

Install Kibana (from)

Again, we have the key and sources files from when we installed Elastisearch. So all we have to do is

sudo apt install kibana

Edit /etc/kibana/kibana.yml and change server-host: to match your server's FQDN or IP address.

And perform the systemdeath rituals

sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service

Check that all Daemons are Running

psg     () { /bin/ps -Awwj | grep "$@" | grep -v grep; }
psg logstash
logstash  5402  4.1  9.1 4268076 371448 ?      SNsl 00:47   3:26 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
psg elasticsearch
elastic+  5358  1.0 58.1 4220876 2353588 ?     Ssl  00:47   0:51 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.2.2.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
psg kibana
kibana    5467  0.5  2.3 1263108 93244 ?       Ssl  00:47   0:25 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Connect to the Kibana Web GUI

You should now be able to connect to the GUI

http://your.host.name:5601

To Be Done

Restrict Access to the Service

How the heck to do the equialent of .htaccess?

Run it over TLS

2017.03.11