from wiki
This commit is contained in:
parent
828ae60924
commit
d5841f48b6
1 changed files with 269 additions and 0 deletions
269
pages/BroCluster.md
Normal file
269
pages/BroCluster.md
Normal file
|
|
@ -0,0 +1,269 @@
|
||||||
|
# Setting Up a Bro Cluster on a Ganeti Cluster
|
||||||
|
|
||||||
|
Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.
|
||||||
|
|
||||||
|
A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this
|
||||||
|
|
||||||
|
```
|
||||||
|
Global LAN
|
||||||
|
147.28.0.0/24
|
||||||
|
+---------------+---------------+--------> Global
|
||||||
|
| | | Internet
|
||||||
|
| | |
|
||||||
|
+----+----+ +----+----+ +----+----+
|
||||||
|
| eth0 | | eth0 | | eth0 |
|
||||||
|
| | | | | |
|
||||||
|
| | | | | |
|
||||||
|
| bro0 | | bro1 | | bro2 |
|
||||||
|
| | | | | |
|
||||||
|
| | | | | |
|
||||||
|
| eth1 | | eth1 | | eth1 |
|
||||||
|
+----+----+ +----+----+ +----+----+
|
||||||
|
| | |
|
||||||
|
| | |
|
||||||
|
+---------------+---------------+
|
||||||
|
DRBD Closed LAN
|
||||||
|
10.0.0.0/24
|
||||||
|
```
|
||||||
|
|
||||||
|
We will monitor the global segment and use the Closed LAN for inter-bro traffic.
|
||||||
|
|
||||||
|
## On Each Bro VM, Create a Second Interface
|
||||||
|
|
||||||
|
We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...
|
||||||
|
|
||||||
|
So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.
|
||||||
|
|
||||||
|
### Bridge Each VM onto the DRBD LAN
|
||||||
|
|
||||||
|
On the Ganeti master, add the DRDB Closed LAN to each Bro instance.
|
||||||
|
|
||||||
|
```
|
||||||
|
gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net
|
||||||
|
gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net
|
||||||
|
gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tell Each Bro Node About the Backdoor LAN
|
||||||
|
|
||||||
|
Edit each of the bro node's `/etc/network/interfaces` to add the new interface.
|
||||||
|
|
||||||
|
```
|
||||||
|
auto eth1
|
||||||
|
iface eth1 inet static
|
||||||
|
address 10.0.0.10/24
|
||||||
|
```
|
||||||
|
|
||||||
|
and so for each bro node.
|
||||||
|
|
||||||
|
On all bro nodes, add entries in `/etc/hosts` so the back LAN will have names
|
||||||
|
|
||||||
|
```
|
||||||
|
# BRO Backdoor LAN
|
||||||
|
#
|
||||||
|
10.0.0.10 bro0.backlan
|
||||||
|
10.0.0.11 bro1.backlan
|
||||||
|
10.0.0.12 bro2.backlan
|
||||||
|
```
|
||||||
|
|
||||||
|
### Reboot the Instances so they Get the New Configurations
|
||||||
|
|
||||||
|
The instances must be rebooted from the ganeti master, not from within the instance
|
||||||
|
|
||||||
|
```
|
||||||
|
gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net
|
||||||
|
```
|
||||||
|
|
||||||
|
Log in to each and ping the others to make sure the configuration has been successful.
|
||||||
|
|
||||||
|
## Create bro User and Give it Perms
|
||||||
|
|
||||||
|
On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.
|
||||||
|
|
||||||
|
```
|
||||||
|
adduser bro
|
||||||
|
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
|
||||||
|
chown -R bro:bro ~bro
|
||||||
|
```
|
||||||
|
|
||||||
|
On all nodes, add bro user to `/etc/sudoers`
|
||||||
|
|
||||||
|
```
|
||||||
|
bro ALL=(ALL) NOPASSWD: ALL
|
||||||
|
```
|
||||||
|
|
||||||
|
## Set Up Credentials
|
||||||
|
|
||||||
|
Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
|
||||||
|
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
|
||||||
|
rsync -vPaHxRSzr .ssh bro1.backlan:
|
||||||
|
rsync -vPaHxRSzr .ssh bro2.backlan:
|
||||||
|
```
|
||||||
|
|
||||||
|
Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh -i .ssh/ed25519.pub bro0.backlan
|
||||||
|
ssh -i .ssh/ed25519.pub bro1.backlan
|
||||||
|
ssh -i .ssh/ed25519.pub bro2.backlan
|
||||||
|
```
|
||||||
|
|
||||||
|
## Install Bro on the Bro Manager Node
|
||||||
|
|
||||||
|
[I use the excellent Bro Doc](https://www.bro.org/sphinx/install/install.html)
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
|
||||||
|
git clone --recursive git://git.bro.org/bro
|
||||||
|
./configure
|
||||||
|
make
|
||||||
|
make install
|
||||||
|
```
|
||||||
|
|
||||||
|
This takes a while.
|
||||||
|
|
||||||
|
Fix the $PATH in .bashrc or whatever
|
||||||
|
|
||||||
|
```
|
||||||
|
export PATH=/usr/local/bro/bin:$PATH
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configure the Cluster
|
||||||
|
|
||||||
|
This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.
|
||||||
|
|
||||||
|
```
|
||||||
|
cat /usr/local/bro/etc/node.cfg << EOF
|
||||||
|
# Example BroControl node configuration.
|
||||||
|
#
|
||||||
|
# This example has a standalone node ready to go except for possibly changing
|
||||||
|
# the sniffing interface.
|
||||||
|
|
||||||
|
# This is a complete standalone configuration. Most likely you will
|
||||||
|
# only need to change the interface.
|
||||||
|
#[bro]
|
||||||
|
#type=standalone
|
||||||
|
#host=localhost
|
||||||
|
#interface=eth0
|
||||||
|
|
||||||
|
## Below is an example clustered configuration. If you use this,
|
||||||
|
## remove the [bro] node above.
|
||||||
|
|
||||||
|
[logger]
|
||||||
|
type=logger
|
||||||
|
host=bro0.backlan
|
||||||
|
#
|
||||||
|
[manager]
|
||||||
|
type=manager
|
||||||
|
host=bro0.backlan
|
||||||
|
#
|
||||||
|
[proxy-1]
|
||||||
|
type=proxy
|
||||||
|
host=bro0.backlan
|
||||||
|
#
|
||||||
|
[worker-0]
|
||||||
|
type=worker
|
||||||
|
host=bro0.backlan
|
||||||
|
interface=eth0
|
||||||
|
#
|
||||||
|
[worker-1]
|
||||||
|
type=worker
|
||||||
|
host=bro1.backlan
|
||||||
|
interface=eth0
|
||||||
|
#
|
||||||
|
[worker-2]
|
||||||
|
type=worker
|
||||||
|
host=bro2.backlan
|
||||||
|
interface=eth0
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configure `broctl.cfg`
|
||||||
|
|
||||||
|
### Make it so that Bro can be Promiscuous on all Nodes
|
||||||
|
|
||||||
|
Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.
|
||||||
|
|
||||||
|
```
|
||||||
|
###############################################
|
||||||
|
# Hacks
|
||||||
|
|
||||||
|
### clean up setcap problem
|
||||||
|
### https://github.com/PingTrip/broctl-setcap
|
||||||
|
#
|
||||||
|
setcap.enabled=1
|
||||||
|
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bro 2.5 Forgot Sendmail Configuration
|
||||||
|
|
||||||
|
Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.
|
||||||
|
|
||||||
|
```
|
||||||
|
### sendmail not configured
|
||||||
|
#
|
||||||
|
SendMail = /usr/sbin/sendmail
|
||||||
|
```
|
||||||
|
|
||||||
|
And you probably want to fix up the MailTo
|
||||||
|
|
||||||
|
```
|
||||||
|
MailTo = randy@psg,com
|
||||||
|
```
|
||||||
|
|
||||||
|
## Give bro User Access to the Ethernet
|
||||||
|
|
||||||
|
Allow the bro user to control network devices. The `setcap` will be done later.
|
||||||
|
|
||||||
|
```
|
||||||
|
gpasswd -a bro netdev
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configure `networks.cfg` for the LAN You Want to Monitor
|
||||||
|
|
||||||
|
```
|
||||||
|
cat > /usr/local/bro/etc/networks.cfg << EOF
|
||||||
|
# List of local networks in CIDR notation, optionally followed by a
|
||||||
|
# descriptive tag.
|
||||||
|
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.
|
||||||
|
|
||||||
|
147.28.0.0/24
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prepare the Worker Nodes
|
||||||
|
|
||||||
|
Make it so bro user can write to `/usr/local/bro` on all nodes
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo mkdir /usr/local/bro
|
||||||
|
sudo chown bro:bro /usr/local/bro
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
Go for broke
|
||||||
|
|
||||||
|
```
|
||||||
|
broctl deploy
|
||||||
|
```
|
||||||
|
|
||||||
|
And start debugging.
|
||||||
|
|
||||||
|
## It is Working, so Cron Watcher
|
||||||
|
|
||||||
|
Add the following to the bro user's crontab:
|
||||||
|
|
||||||
|
```
|
||||||
|
*/5 * * * * /usr/local/bro/bin/broctl cron
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that you can disable and enable the cron watcher
|
||||||
|
|
||||||
|
```
|
||||||
|
broctl cron disable
|
||||||
|
broctl cron enable
|
||||||
|
```
|
||||||
Loading…
Add table
Add a link
Reference in a new issue