# Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node

## Install Ubuntu

- Before install, it is wise to disable DHCP service so one can manually set the system IP data
- Boot Ubuntu CD/ISO
- Choose Install Ubuntu Server
- Choose US English or it will choose the wrong apt server set
- Choose American English
- CHoose Macintosh keyboard
- Select the Ethernet interface to be used, I had connected eno1
- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
- Configure Hostname
- Choose user name and password
- Do not encrypt home directory
- Let it set some stupid timezone, you can fix later
- Partition Disks
  - Choose Manual Partitioning
  - Select the first drive
  - Create a new empty partition table
  - Select each of the other Drives and Create a new empty partition table
  - Select the free space on each device and create a 256M ESP partition named /boot/efi
  - Configure Software RAID
  - Write Changes and Configure RAID
  - Create MD Device
  - Select RAID10  - with lots of disk, I chose RAID10 for speed
  - Number of Devices should be all the drives you have, but an even number
  - Number of Spare Drives is 0, unless you had an odd number of drives
  - Select all the drives (not the EFI partitions) and continue
  - Write the changes and configure RAID
  - Finish
- Configure LVM
  - Configure LVM accepting write changes to disks
  - Create volume group
    - Volume group name: ganeti
    - Devices for the new volume group: select /dev/md0
    - Keep partioning and write
  - Create Logical Volume: on ganeti, root, 16G
  - Create Logical Volume: on ganeti, swap, size of RAM
  - Create Logical Volume: on ganeti, var, 16G
  - Finish
  - Edit the Logical Volumes to be ext4 /, swap, and ext4 /var

At this point, the partitioning looks like this
 [![raid10-part.jpg](https://wiki.rg.net/chrome/site/raid10-part.jpg)](https://wiki.rg.net/chrome/site/raid10-part.jpg)  

- Finish partitioning and write changes
- Finish partitioning and write changes to disk
- Ubuntu will now install and you will do the normal mirror selection etc.
- I choose no automatic updates as I ansible that in later
- Only install SSH Server, System Utilities
- Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask
- Be sure it will not boot CD-ROM, and Reboot from the installed system

## Finish Ubuntu Installation

Install homey things (it's not a computer without emacs:)

```
apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox gcc bridge-utils vlan unbound
usermod -G sudo -a randy
```

As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot.

The next thing I do is to copy root's credentials and dot-files from a known system

Fix `/etc/ssh/sshd_config` to forbid passwords

```
PermitRootLogin without-password
PasswordAuthentication no
```

And restart the ssh daemon 

```
service ssh restart
```

You can do most of the rest from the KVM/VGA or from an ssh with password.  Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.

Set the timezone

```
dpkg-reconfigure tzdata
```

In the wonderful world of systemd, here is how to get ntpd going

```
timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF
```

On some Xenial systems, to get unbound to start reliably, I have had to

```
cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service

[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535

[Install]
WantedBy=multi-user.target
EOF
```

Fix hostname

```
echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`
```

Fix /etc/unbound/unbound.conf

```
        access-control: 127.0.0.0/8 allow
        access-control: 198.180.152.0/24 allow
        access-control: 0.0.0.0/0 refuse
        access-control: ::1 allow
        access-control: ::ffff:127.0.0.1 allow
        access-control: 2001:deb::/48
        access-control: ::0/0 refuse
```

And restart unbound

```
service unbound restart
```

I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN.  Edit `/etc/default/grub` to have

```
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=6
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8"
```

And reconfigure grub

```
# update-grub
```

Clean up from CDROM sources

```
emacs /etc/apt/sources.list
```

and delete the two CDROM entries at the top.

We want to manage the server so install ipmitool and smartmontools

```
modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool
```

Set up IPMI

```
# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list
```

Set up smartmontools to watch the disks

```
apt install smartmontools
```

And check that smartd is running

```
root      1457  0.0  0.0  25396  4512 ?        Ss   16:02   0:00 /usr/sbin/smartd -n
```

I configure smartd to do short test every day and long once a week

```
cat >> /etc/smartd.conf << EOF

/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF
```

### System Services

It is good to have a mail system to send logs to the sucker who watches such things.  I use exim4

```
apt install exim4 mailtools
dpkg-reconfigure exim4-config
```

Use internet mail
 System name is vm0.sea.rg.net
 Only listen on local interfaces
 Relay mail for noone

### Unattended Upgrades, syslog-NG, etc.

[Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade)

Install syslog-ng

```
apt-get install syslog-ng
```

Hack /etc/logrotate.d/syslog-ng

```
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng  2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
        missingok
        notifempty
        compress
+       mailfirst
+       mail randy@psg.com
        delaycompress
        sharedscripts
        postrotate
```

## Ubuntu Ganeti Specific Configuration

Edit /etc/hosts to have the real address of the host, e.g.

```
cat > /etc/hosts << EOF
127.0.0.1       localhost
147.28.0.14     vm0.sea.rg.net vm0
147.28.0.15     vm1.sea.rg.net vm1
147.28.0.16     vm2.sea.rg.net vm2
147.28.0.100    gnt0.sea.rg.net gnt0
EOF
```

## Install Ganeti

In Ubuntu 16.04, ganeti is in the package system, no need for hacks.

Check what version will be installed

```
apt-cache policy ganeti
ganeti:
  Installed: (none)
  Candidate: 2.15.2-3
  Version table:
     2.15.2-3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
```

So now we can explicitly install 2.15

```
apt install ganeti
```

And it's a non-trivial dependency tree, to say the least

```
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping
  libcurl3 python-bitarray python-cffi-backend python-crypto
  python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
  python-ipaddr python-ipaddress python-openssl python-paramiko
  python-pkg-resources python-psutil python-pyasn1 python-pycurl
  python-pyinotify python-pyparsing python-simplejson python-six socat
Suggested packages:
  ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
  python-cryptography-doc python-cryptography-vectors python-enum34-doc
  python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc
  doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc
  python-pyinotify-doc
Recommended packages:
  drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64
  | xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
  fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
  iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto
  python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
  python-ipaddr python-ipaddress python-openssl python-paramiko
  python-pkg-resources python-psutil python-pyasn1 python-pycurl
  python-pyinotify python-pyparsing python-simplejson python-six socat
0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.3 MB of archives.
After this operation, 141 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
```

Fix up drbd

```
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd      # ignore any error
modprobe drbd
```