6.3 KiB
Setting Up a Bro Cluster on a Ganeti Cluster
Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.
A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this
Global LAN
147.28.0.0/24
+---------------+---------------+--------> Global
| | | Internet
| | |
+----+----+ +----+----+ +----+----+
| eth0 | | eth0 | | eth0 |
| | | | | |
| | | | | |
| bro0 | | bro1 | | bro2 |
| | | | | |
| | | | | |
| eth1 | | eth1 | | eth1 |
+----+----+ +----+----+ +----+----+
| | |
| | |
+---------------+---------------+
DRBD Closed LAN
10.0.0.0/24
We will monitor the global segment and use the Closed LAN for inter-bro traffic.
On Each Bro VM, Create a Second Interface
We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...
So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.
Bridge Each VM onto the DRBD LAN
On the Ganeti master, add the DRDB Closed LAN to each Bro instance.
gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net
gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net
gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net
Tell Each Bro Node About the Backdoor LAN
Edit each of the bro node's /etc/network/interfaces to add the new interface.
auto eth1
iface eth1 inet static
address 10.0.0.10/24
and so for each bro node.
On all bro nodes, add entries in /etc/hosts so the back LAN will have names
# BRO Backdoor LAN
#
10.0.0.10 bro0.backlan
10.0.0.11 bro1.backlan
10.0.0.12 bro2.backlan
Reboot the Instances so they Get the New Configurations
The instances must be rebooted from the ganeti master, not from within the instance
gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net
Log in to each and ping the others to make sure the configuration has been successful.
Create bro User and Give it Perms
On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.
adduser bro
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
chown -R bro:bro ~bro
On all nodes, add bro user to /etc/sudoers
bro ALL=(ALL) NOPASSWD: ALL
Set Up Credentials
Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.
ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
rsync -vPaHxRSzr .ssh bro1.backlan:
rsync -vPaHxRSzr .ssh bro2.backlan:
Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys
ssh -i .ssh/ed25519.pub bro0.backlan
ssh -i .ssh/ed25519.pub bro1.backlan
ssh -i .ssh/ed25519.pub bro2.backlan
Install Bro on the Bro Manager Node
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
git clone --recursive git://git.bro.org/bro
./configure
make
make install
This takes a while.
Fix the $PATH in .bashrc or whatever
export PATH=/usr/local/bro/bin:$PATH
Configure the Cluster
This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.
cat /usr/local/bro/etc/node.cfg << EOF
# Example BroControl node configuration.
#
# This example has a standalone node ready to go except for possibly changing
# the sniffing interface.
# This is a complete standalone configuration. Most likely you will
# only need to change the interface.
#[bro]
#type=standalone
#host=localhost
#interface=eth0
## Below is an example clustered configuration. If you use this,
## remove the [bro] node above.
[logger]
type=logger
host=bro0.backlan
#
[manager]
type=manager
host=bro0.backlan
#
[proxy-1]
type=proxy
host=bro0.backlan
#
[worker-0]
type=worker
host=bro0.backlan
interface=eth0
#
[worker-1]
type=worker
host=bro1.backlan
interface=eth0
#
[worker-2]
type=worker
host=bro2.backlan
interface=eth0
EOF
Configure broctl.cfg
Make it so that Bro can be Promiscuous on all Nodes
Hack the following into /usr/local/bro/etc/broctl.cfg on the master.
###############################################
# Hacks
### clean up setcap problem
### https://github.com/PingTrip/broctl-setcap
#
setcap.enabled=1
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats
Bro 2.5 Forgot Sendmail Configuration
Hack the following into /usr/local/bro/etc/broctl.cfg on the master.
### sendmail not configured
#
SendMail = /usr/sbin/sendmail
And you probably want to fix up the MailTo
MailTo = randy@psg,com
Give bro User Access to the Ethernet
Allow the bro user to control network devices. The setcap will be done later.
gpasswd -a bro netdev
Configure networks.cfg for the LAN You Want to Monitor
cat > /usr/local/bro/etc/networks.cfg << EOF
# List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.
147.28.0.0/24
EOF
Prepare the Worker Nodes
Make it so bro user can write to /usr/local/bro on all nodes
sudo mkdir /usr/local/bro
sudo chown bro:bro /usr/local/bro
Test
Go for broke
broctl deploy
And start debugging.
It is Working, so Cron Watcher
Add the following to the bro user's crontab:
*/5 * * * * /usr/local/bro/bin/broctl cron
Note that you can disable and enable the cron watcher
broctl cron disable
broctl cron enable