from wiki
This commit is contained in:
parent
828ae60924
commit
d5841f48b6
1 changed files with 269 additions and 0 deletions
269
pages/BroCluster.md
Normal file
269
pages/BroCluster.md
Normal file
|
|
@ -0,0 +1,269 @@
|
|||
# Setting Up a Bro Cluster on a Ganeti Cluster
|
||||
|
||||
Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.
|
||||
|
||||
A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this
|
||||
|
||||
```
|
||||
Global LAN
|
||||
147.28.0.0/24
|
||||
+---------------+---------------+--------> Global
|
||||
| | | Internet
|
||||
| | |
|
||||
+----+----+ +----+----+ +----+----+
|
||||
| eth0 | | eth0 | | eth0 |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| bro0 | | bro1 | | bro2 |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| eth1 | | eth1 | | eth1 |
|
||||
+----+----+ +----+----+ +----+----+
|
||||
| | |
|
||||
| | |
|
||||
+---------------+---------------+
|
||||
DRBD Closed LAN
|
||||
10.0.0.0/24
|
||||
```
|
||||
|
||||
We will monitor the global segment and use the Closed LAN for inter-bro traffic.
|
||||
|
||||
## On Each Bro VM, Create a Second Interface
|
||||
|
||||
We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...
|
||||
|
||||
So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.
|
||||
|
||||
### Bridge Each VM onto the DRBD LAN
|
||||
|
||||
On the Ganeti master, add the DRDB Closed LAN to each Bro instance.
|
||||
|
||||
```
|
||||
gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net
|
||||
gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net
|
||||
gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net
|
||||
```
|
||||
|
||||
### Tell Each Bro Node About the Backdoor LAN
|
||||
|
||||
Edit each of the bro node's `/etc/network/interfaces` to add the new interface.
|
||||
|
||||
```
|
||||
auto eth1
|
||||
iface eth1 inet static
|
||||
address 10.0.0.10/24
|
||||
```
|
||||
|
||||
and so for each bro node.
|
||||
|
||||
On all bro nodes, add entries in `/etc/hosts` so the back LAN will have names
|
||||
|
||||
```
|
||||
# BRO Backdoor LAN
|
||||
#
|
||||
10.0.0.10 bro0.backlan
|
||||
10.0.0.11 bro1.backlan
|
||||
10.0.0.12 bro2.backlan
|
||||
```
|
||||
|
||||
### Reboot the Instances so they Get the New Configurations
|
||||
|
||||
The instances must be rebooted from the ganeti master, not from within the instance
|
||||
|
||||
```
|
||||
gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net
|
||||
```
|
||||
|
||||
Log in to each and ping the others to make sure the configuration has been successful.
|
||||
|
||||
## Create bro User and Give it Perms
|
||||
|
||||
On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.
|
||||
|
||||
```
|
||||
adduser bro
|
||||
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
|
||||
chown -R bro:bro ~bro
|
||||
```
|
||||
|
||||
On all nodes, add bro user to `/etc/sudoers`
|
||||
|
||||
```
|
||||
bro ALL=(ALL) NOPASSWD: ALL
|
||||
```
|
||||
|
||||
## Set Up Credentials
|
||||
|
||||
Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.
|
||||
|
||||
```
|
||||
ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
|
||||
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
|
||||
rsync -vPaHxRSzr .ssh bro1.backlan:
|
||||
rsync -vPaHxRSzr .ssh bro2.backlan:
|
||||
```
|
||||
|
||||
Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys
|
||||
|
||||
```
|
||||
ssh -i .ssh/ed25519.pub bro0.backlan
|
||||
ssh -i .ssh/ed25519.pub bro1.backlan
|
||||
ssh -i .ssh/ed25519.pub bro2.backlan
|
||||
```
|
||||
|
||||
## Install Bro on the Bro Manager Node
|
||||
|
||||
[I use the excellent Bro Doc](https://www.bro.org/sphinx/install/install.html)
|
||||
|
||||
```
|
||||
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
|
||||
git clone --recursive git://git.bro.org/bro
|
||||
./configure
|
||||
make
|
||||
make install
|
||||
```
|
||||
|
||||
This takes a while.
|
||||
|
||||
Fix the $PATH in .bashrc or whatever
|
||||
|
||||
```
|
||||
export PATH=/usr/local/bro/bin:$PATH
|
||||
```
|
||||
|
||||
## Configure the Cluster
|
||||
|
||||
This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.
|
||||
|
||||
```
|
||||
cat /usr/local/bro/etc/node.cfg << EOF
|
||||
# Example BroControl node configuration.
|
||||
#
|
||||
# This example has a standalone node ready to go except for possibly changing
|
||||
# the sniffing interface.
|
||||
|
||||
# This is a complete standalone configuration. Most likely you will
|
||||
# only need to change the interface.
|
||||
#[bro]
|
||||
#type=standalone
|
||||
#host=localhost
|
||||
#interface=eth0
|
||||
|
||||
## Below is an example clustered configuration. If you use this,
|
||||
## remove the [bro] node above.
|
||||
|
||||
[logger]
|
||||
type=logger
|
||||
host=bro0.backlan
|
||||
#
|
||||
[manager]
|
||||
type=manager
|
||||
host=bro0.backlan
|
||||
#
|
||||
[proxy-1]
|
||||
type=proxy
|
||||
host=bro0.backlan
|
||||
#
|
||||
[worker-0]
|
||||
type=worker
|
||||
host=bro0.backlan
|
||||
interface=eth0
|
||||
#
|
||||
[worker-1]
|
||||
type=worker
|
||||
host=bro1.backlan
|
||||
interface=eth0
|
||||
#
|
||||
[worker-2]
|
||||
type=worker
|
||||
host=bro2.backlan
|
||||
interface=eth0
|
||||
EOF
|
||||
```
|
||||
|
||||
## Configure `broctl.cfg`
|
||||
|
||||
### Make it so that Bro can be Promiscuous on all Nodes
|
||||
|
||||
Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.
|
||||
|
||||
```
|
||||
###############################################
|
||||
# Hacks
|
||||
|
||||
### clean up setcap problem
|
||||
### https://github.com/PingTrip/broctl-setcap
|
||||
#
|
||||
setcap.enabled=1
|
||||
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats
|
||||
```
|
||||
|
||||
### Bro 2.5 Forgot Sendmail Configuration
|
||||
|
||||
Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master.
|
||||
|
||||
```
|
||||
### sendmail not configured
|
||||
#
|
||||
SendMail = /usr/sbin/sendmail
|
||||
```
|
||||
|
||||
And you probably want to fix up the MailTo
|
||||
|
||||
```
|
||||
MailTo = randy@psg,com
|
||||
```
|
||||
|
||||
## Give bro User Access to the Ethernet
|
||||
|
||||
Allow the bro user to control network devices. The `setcap` will be done later.
|
||||
|
||||
```
|
||||
gpasswd -a bro netdev
|
||||
```
|
||||
|
||||
## Configure `networks.cfg` for the LAN You Want to Monitor
|
||||
|
||||
```
|
||||
cat > /usr/local/bro/etc/networks.cfg << EOF
|
||||
# List of local networks in CIDR notation, optionally followed by a
|
||||
# descriptive tag.
|
||||
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.
|
||||
|
||||
147.28.0.0/24
|
||||
EOF
|
||||
```
|
||||
|
||||
## Prepare the Worker Nodes
|
||||
|
||||
Make it so bro user can write to `/usr/local/bro` on all nodes
|
||||
|
||||
```
|
||||
sudo mkdir /usr/local/bro
|
||||
sudo chown bro:bro /usr/local/bro
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
Go for broke
|
||||
|
||||
```
|
||||
broctl deploy
|
||||
```
|
||||
|
||||
And start debugging.
|
||||
|
||||
## It is Working, so Cron Watcher
|
||||
|
||||
Add the following to the bro user's crontab:
|
||||
|
||||
```
|
||||
*/5 * * * * /usr/local/bro/bin/broctl cron
|
||||
```
|
||||
|
||||
Note that you can disable and enable the cron watcher
|
||||
|
||||
```
|
||||
broctl cron disable
|
||||
broctl cron enable
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue