269 lines
No EOL
6.3 KiB
Markdown
269 lines
No EOL
6.3 KiB
Markdown
# Setting Up a Bro Cluster on a Ganeti Cluster
|
|
|
|
Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.
|
|
|
|
A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this
|
|
|
|
```
|
|
Global LAN
|
|
147.28.0.0/24
|
|
+---------------+---------------+--------> Global
|
|
| | | Internet
|
|
| | |
|
|
+----+----+ +----+----+ +----+----+
|
|
| eth0 | | eth0 | | eth0 |
|
|
| | | | | |
|
|
| | | | | |
|
|
| bro0 | | bro1 | | bro2 |
|
|
| | | | | |
|
|
| | | | | |
|
|
| eth1 | | eth1 | | eth1 |
|
|
+----+----+ +----+----+ +----+----+
|
|
| | |
|
|
| | |
|
|
+---------------+---------------+
|
|
DRBD Closed LAN
|
|
10.0.0.0/24
|
|
```
|
|
|
|
We will monitor the global segment and use the Closed LAN for inter-bro traffic.
|
|
|
|
## On Each Bro VM, Create a Second Interface
|
|
|
|
We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...
|
|
|
|
So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.
|
|
|
|
### Bridge Each VM onto the DRBD LAN
|
|
|
|
On the Ganeti master, add the DRDB Closed LAN to each Bro instance.
|
|
|
|
```
|
|
gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net
|
|
gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net
|
|
gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net
|
|
```
|
|
|
|
### Tell Each Bro Node About the Backdoor LAN
|
|
|
|
Edit each of the bro node's `/etc/network/interfaces` to add the new interface.
|
|
|
|
```
|
|
auto eth1
|
|
iface eth1 inet static
|
|
address 10.0.0.10/24
|
|
```
|
|
|
|
and so for each bro node.
|
|
|
|
On all bro nodes, add entries in `/etc/hosts` so the back LAN will have names
|
|
|
|
```
|
|
# BRO Backdoor LAN
|
|
#
|
|
10.0.0.10 bro0.backlan
|
|
10.0.0.11 bro1.backlan
|
|
10.0.0.12 bro2.backlan
|
|
```
|
|
|
|
### Reboot the Instances so they Get the New Configurations
|
|
|
|
The instances must be rebooted from the ganeti master, not from within the instance
|
|
|
|
```
|
|
gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net
|
|
```
|
|
|
|
Log in to each and ping the others to make sure the configuration has been successful.
|
|
|
|
## Create bro User and Give it Perms
|
|
|
|
On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.
|
|
|
|
```
|
|
adduser bro
|
|
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
|
|
chown -R bro:bro ~bro
|
|
```
|
|
|
|
On all nodes, add bro user to `/etc/sudoers`
|
|
|
|
```
|
|
bro ALL=(ALL) NOPASSWD: ALL
|
|
```
|
|
|
|
## Set Up Credentials
|
|
|
|
Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.
|
|
|
|
```
|
|
ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
|
|
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
|
|
rsync -vPaHxRSzr .ssh bro1.backlan:
|
|
rsync -vPaHxRSzr .ssh bro2.backlan:
|
|
```
|
|
|
|
Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys
|
|
|
|
```
|
|
ssh -i .ssh/ed25519.pub bro0.backlan
|
|
ssh -i .ssh/ed25519.pub bro1.backlan
|
|
ssh -i .ssh/ed25519.pub bro2.backlan
|
|
```
|
|
|
|
## Install Bro on the Bro Manager Node
|
|
|
|
[I use the excellent Bro Doc](https://www.bro.org/sphinx/install/install.html)
|
|
|
|
```
|
|
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
|
|
git clone --recursive git://git.bro.org/bro
|
|
./configure
|
|
make
|
|
make install
|
|
```
|
|
|
|
This takes a while.
|
|
|
|
Fix the $PATH in .bashrc or whatever
|
|
|
|
```
|
|
export PATH=/usr/local/bro/bin:$PATH
|
|
```
|
|
|
|
## Configure the Cluster
|
|
|
|
This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.
|
|
|
|
```
|
|
cat /usr/local/bro/etc/node.cfg << EOF
|
|
# Example BroControl node configuration.
|
|
#
|
|
# This example has a standalone node ready to go except for possibly changing
|
|
# the sniffing interface.
|
|
|
|
# This is a complete standalone configuration. Most likely you will
|
|
# only need to change the interface.
|
|
#[bro]
|
|
#type=standalone
|
|
#host=localhost
|
|
#interface=eth0
|
|
|
|
## Below is an example clustered configuration. If you use this,
|
|
## remove the [bro] node above.
|
|
|
|
[logger]
|
|
type=logger
|
|
host=bro0.backlan
|
|
#
|
|
[manager]
|
|
type=manager
|
|
host=bro0.backlan
|
|
#
|
|
[proxy-1]
|
|
type=proxy
|
|
host=bro0.backlan
|
|
#
|
|
[worker-0]
|
|
type=worker
|
|
host=bro0.backlan
|
|
interface=eth0
|
|
#
|
|
[worker-1]
|
|
type=worker
|
|
host=bro1.backlan
|
|
interface=eth0
|
|
#
|
|
[worker-2]
|
|
type=worker
|
|
host=bro2.backlan
|
|
interface=eth0
|
|
EOF
|
|
```
|
|
|
|
## Configure `broctl.cfg`
|
|
|
|
### Make it so that Bro can be Promiscuous on all Nodes
|
|
|
|
Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.
|
|
|
|
```
|
|
###############################################
|
|
# Hacks
|
|
|
|
### clean up setcap problem
|
|
### https://github.com/PingTrip/broctl-setcap
|
|
#
|
|
setcap.enabled=1
|
|
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats
|
|
```
|
|
|
|
### Bro 2.5 Forgot Sendmail Configuration
|
|
|
|
Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.
|
|
|
|
```
|
|
### sendmail not configured
|
|
#
|
|
SendMail = /usr/sbin/sendmail
|
|
```
|
|
|
|
And you probably want to fix up the MailTo
|
|
|
|
```
|
|
MailTo = randy@psg,com
|
|
```
|
|
|
|
## Give bro User Access to the Ethernet
|
|
|
|
Allow the bro user to control network devices. The `setcap` will be done later.
|
|
|
|
```
|
|
gpasswd -a bro netdev
|
|
```
|
|
|
|
## Configure `networks.cfg` for the LAN You Want to Monitor
|
|
|
|
```
|
|
cat > /usr/local/bro/etc/networks.cfg << EOF
|
|
# List of local networks in CIDR notation, optionally followed by a
|
|
# descriptive tag.
|
|
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.
|
|
|
|
147.28.0.0/24
|
|
EOF
|
|
```
|
|
|
|
## Prepare the Worker Nodes
|
|
|
|
Make it so bro user can write to `/usr/local/bro` on all nodes
|
|
|
|
```
|
|
sudo mkdir /usr/local/bro
|
|
sudo chown bro:bro /usr/local/bro
|
|
```
|
|
|
|
## Test
|
|
|
|
Go for broke
|
|
|
|
```
|
|
broctl deploy
|
|
```
|
|
|
|
And start debugging.
|
|
|
|
## It is Working, so Cron Watcher
|
|
|
|
Add the following to the bro user's crontab:
|
|
|
|
```
|
|
*/5 * * * * /usr/local/bro/bin/broctl cron
|
|
```
|
|
|
|
Note that you can disable and enable the cron watcher
|
|
|
|
```
|
|
broctl cron disable
|
|
broctl cron enable
|
|
``` |