From d5841f48b631cab435dff46b96f7fa4e3adb4c46 Mon Sep 17 00:00:00 2001 From: Randy Bush Date: Sat, 4 Jul 2020 15:18:33 -0700 Subject: [PATCH] from wiki --- pages/BroCluster.md | 269 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 269 insertions(+) create mode 100644 pages/BroCluster.md diff --git a/pages/BroCluster.md b/pages/BroCluster.md new file mode 100644 index 0000000..b193fb7 --- /dev/null +++ b/pages/BroCluster.md @@ -0,0 +1,269 @@ +# Setting Up a Bro Cluster on a Ganeti Cluster + +Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2. + +A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this + +``` + Global LAN + 147.28.0.0/24 + +---------------+---------------+--------> Global + | | | Internet + | | | ++----+----+ +----+----+ +----+----+ +| eth0 | | eth0 | | eth0 | +| | | | | | +| | | | | | +| bro0 | | bro1 | | bro2 | +| | | | | | +| | | | | | +| eth1 | | eth1 | | eth1 | ++----+----+ +----+----+ +----+----+ + | | | + | | | + +---------------+---------------+ + DRBD Closed LAN + 10.0.0.0/24 +``` + +We will monitor the global segment and use the Closed LAN for inter-bro traffic. + +## On Each Bro VM, Create a Second Interface + +We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ... + +So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic. + +### Bridge Each VM onto the DRBD LAN + +On the Ganeti master, add the DRDB Closed LAN to each Bro instance. + +``` +gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net +gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net +gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net +``` + +### Tell Each Bro Node About the Backdoor LAN + +Edit each of the bro node's `/etc/network/interfaces` to add the new interface. + +``` +auto eth1 +iface eth1 inet static + address 10.0.0.10/24 +``` + +and so for each bro node. + +On all bro nodes, add entries in `/etc/hosts` so the back LAN will have names + +``` +# BRO Backdoor LAN +# +10.0.0.10 bro0.backlan +10.0.0.11 bro1.backlan +10.0.0.12 bro2.backlan +``` + +### Reboot the Instances so they Get the New Configurations + +The instances must be rebooted from the ganeti master, not from within the instance + +``` +gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net +``` + +Log in to each and ping the others to make sure the configuration has been successful. + +## Create bro User and Give it Perms + +On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files. + +``` +adduser bro +rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro +chown -R bro:bro ~bro +``` + +On all nodes, add bro user to `/etc/sudoers` + +``` +bro ALL=(ALL) NOPASSWD: ALL +``` + +## Set Up Credentials + +Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs. + +``` +ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519 +cat .ssh/id_ed25519.pub >> .ssh/authorized_keys +rsync -vPaHxRSzr .ssh bro1.backlan: +rsync -vPaHxRSzr .ssh bro2.backlan: +``` + +Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys + +``` +ssh -i .ssh/ed25519.pub bro0.backlan +ssh -i .ssh/ed25519.pub bro1.backlan +ssh -i .ssh/ed25519.pub bro2.backlan +``` + +## Install Bro on the Bro Manager Node + +[I use the excellent Bro Doc](https://www.bro.org/sphinx/install/install.html) + +``` +sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev +git clone --recursive git://git.bro.org/bro +./configure +make +make install +``` + +This takes a while. + +Fix the $PATH in .bashrc or whatever + +``` +export PATH=/usr/local/bro/bin:$PATH +``` + +## Configure the Cluster + +This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0. + +``` +cat /usr/local/bro/etc/node.cfg << EOF +# Example BroControl node configuration. +# +# This example has a standalone node ready to go except for possibly changing +# the sniffing interface. + +# This is a complete standalone configuration. Most likely you will +# only need to change the interface. +#[bro] +#type=standalone +#host=localhost +#interface=eth0 + +## Below is an example clustered configuration. If you use this, +## remove the [bro] node above. + +[logger] +type=logger +host=bro0.backlan +# +[manager] +type=manager +host=bro0.backlan +# +[proxy-1] +type=proxy +host=bro0.backlan +# +[worker-0] +type=worker +host=bro0.backlan +interface=eth0 +# +[worker-1] +type=worker +host=bro1.backlan +interface=eth0 +# +[worker-2] +type=worker +host=bro2.backlan +interface=eth0 +EOF +``` + +## Configure `broctl.cfg` + +### Make it so that Bro can be Promiscuous on all Nodes + +Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master. + +``` +############################################### +# Hacks + +### clean up setcap problem +### https://github.com/PingTrip/broctl-setcap +# +setcap.enabled=1 +setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats +``` + +### Bro 2.5 Forgot Sendmail Configuration + +Hack the following into `/usr/local/bro/etc/broctl.cfg` on the master. + +``` +### sendmail not configured +# +SendMail = /usr/sbin/sendmail +``` + +And you probably want to fix up the MailTo + +``` +MailTo = randy@psg,com +``` + +## Give bro User Access to the Ethernet + +Allow the bro user to control network devices. The `setcap` will be done later. + +``` +gpasswd -a bro netdev +``` + +## Configure `networks.cfg` for the LAN You Want to Monitor + +``` +cat > /usr/local/bro/etc/networks.cfg << EOF +# List of local networks in CIDR notation, optionally followed by a +# descriptive tag. +# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes. + +147.28.0.0/24 +EOF +``` + +## Prepare the Worker Nodes + +Make it so bro user can write to `/usr/local/bro` on all nodes + +``` +sudo mkdir /usr/local/bro +sudo chown bro:bro /usr/local/bro +``` + +## Test + +Go for broke + +``` +broctl deploy +``` + +And start debugging. + +## It is Working, so Cron Watcher + +Add the following to the bro user's crontab: + +``` +*/5 * * * * /usr/local/bro/bin/broctl cron +``` + +Note that you can disable and enable the cron watcher + +``` +broctl cron disable +broctl cron enable +``` \ No newline at end of file