# Setting Up a Bro Cluster on a Ganeti Cluster

Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.

A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication.  In this example, it looks like this

```
                  Global LAN
                147.28.0.0/24
     +---------------+---------------+-------->  Global
     |               |               |          Internet
     |               |               |
+----+----+     +----+----+     +----+----+
|  eth0   |     |  eth0   |     |  eth0   |
|         |     |         |     |         |
|         |     |         |     |         |
|  bro0   |     |  bro1   |     |  bro2   |
|         |     |         |     |         |
|         |     |         |     |         |
|  eth1   |     |  eth1   |     |  eth1   |
+----+----+     +----+----+     +----+----+
     |               |               |
     |               |               |
     +---------------+---------------+
              DRBD Closed LAN
                10.0.0.0/24
```

We will monitor the global segment and use the Closed LAN for inter-bro traffic.

## On Each Bro VM, Create a Second Interface

We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...

So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.

### Bridge Each VM onto the DRBD LAN

On the Ganeti master, add the DRDB Closed LAN to each Bro instance.

```
gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net
gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net
gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net
```

### Tell Each Bro Node About the Backdoor LAN

Edit each of the bro node's `/etc/network/interfaces` to add the new interface.

```
auto eth1
iface eth1 inet static
  address 10.0.0.10/24
```

and so for each bro node.

On all bro nodes, add entries in `/etc/hosts` so the back LAN will have names

```
# BRO Backdoor LAN
#
10.0.0.10       bro0.backlan
10.0.0.11       bro1.backlan
10.0.0.12       bro2.backlan
```

### Reboot the Instances so they Get the New Configurations

The instances must be rebooted from the ganeti master, not from within the instance

```
gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net
```

Log in to each and ping the others to make sure the configuration has been successful.

## Create bro User and Give it Perms

On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.

```
adduser bro
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
chown -R bro:bro ~bro
```

On all nodes, add bro user to `/etc/sudoers`

```
bro     ALL=(ALL) NOPASSWD: ALL
```

## Set Up Credentials

Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.

```
ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
rsync -vPaHxRSzr .ssh bro1.backlan:
rsync -vPaHxRSzr .ssh bro2.backlan:
```

Introduce the accounts to each other.  On all three nodes, run the following and confirm the host keys

```
ssh -i .ssh/ed25519.pub bro0.backlan
ssh -i .ssh/ed25519.pub bro1.backlan
ssh -i .ssh/ed25519.pub bro2.backlan
```

## Install Bro on the Bro Manager Node

[I use the excellent Bro Doc](https://www.bro.org/sphinx/install/install.html)

```
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
git clone --recursive git://git.bro.org/bro
./configure
make
make install
```

This takes a while.

Fix the $PATH in .bashrc or whatever

```
export PATH=/usr/local/bro/bin:$PATH
```

## Configure the Cluster

This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.

```
cat /usr/local/bro/etc/node.cfg << EOF
# Example BroControl node configuration.
#
# This example has a standalone node ready to go except for possibly changing
# the sniffing interface.

# This is a complete standalone configuration.  Most likely you will
# only need to change the interface.
#[bro]
#type=standalone
#host=localhost
#interface=eth0

## Below is an example clustered configuration. If you use this,
## remove the [bro] node above.

[logger]
type=logger
host=bro0.backlan
#
[manager]
type=manager
host=bro0.backlan
#
[proxy-1]
type=proxy
host=bro0.backlan
#
[worker-0]
type=worker
host=bro0.backlan
interface=eth0
#
[worker-1]
type=worker
host=bro1.backlan
interface=eth0
#
[worker-2]
type=worker
host=bro2.backlan
interface=eth0
EOF
```

## Configure `broctl.cfg`

### Make it so that Bro can be Promiscuous on all Nodes

Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.

```
###############################################
# Hacks

### clean up setcap problem
### https://github.com/PingTrip/broctl-setcap
#
setcap.enabled=1
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats
```

### Bro 2.5 Forgot Sendmail Configuration

Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.

```
### sendmail not configured
#
SendMail = /usr/sbin/sendmail
```

And you probably want to fix up the MailTo

```
MailTo = randy@psg,com
```

## Give bro User Access to the Ethernet

Allow the bro user to control network devices.  The `setcap` will be done later.

```
gpasswd -a bro netdev
```

## Configure `networks.cfg` for the LAN You Want to Monitor

```
cat > /usr/local/bro/etc/networks.cfg << EOF
# List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.

147.28.0.0/24
EOF
```

## Prepare the Worker Nodes

Make it so bro user can write to `/usr/local/bro` on all nodes

```
sudo mkdir /usr/local/bro
sudo chown bro:bro /usr/local/bro
```

## Test

Go for broke

```
broctl deploy
```

And start debugging.

## It is Working, so Cron Watcher

Add the following to the bro user's crontab:

```
*/5 * * * * /usr/local/bro/bin/broctl cron
```

Note that you can disable and enable the cron watcher

```
broctl cron disable
broctl cron enable
```