# Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node ## Install Ubuntu - Before install, it is wise to disable DHCP service so one can manually set the system IP data - Boot Ubuntu CD/ISO - Choose Install Ubuntu Server - Choose US English or it will choose the wrong apt server set - Choose American English - CHoose Macintosh keyboard - Select the Ethernet interface to be used, I had connected eno1 - Configure Ethernet IP Address, Netmask, Gateway, and DNS Service - Configure Hostname - Choose user name and password - Do not encrypt home directory - Let it set some stupid timezone, you can fix later - Partition Disks - Choose Manual Partitioning - Select the first drive - Create a new empty partition table - Select each of the other Drives and Create a new empty partition table - Select the free space on each device and create a 256M ESP partition named /boot/efi - Configure Software RAID - Write Changes and Configure RAID - Create MD Device - Select RAID10 - with lots of disk, I chose RAID10 for speed - Number of Devices should be all the drives you have, but an even number - Number of Spare Drives is 0, unless you had an odd number of drives - Select all the drives (not the EFI partitions) and continue - Write the changes and configure RAID - Finish - Configure LVM - Configure LVM accepting write changes to disks - Create volume group - Volume group name: ganeti - Devices for the new volume group: select /dev/md0 - Keep partioning and write - Create Logical Volume: on ganeti, root, 16G - Create Logical Volume: on ganeti, swap, size of RAM - Create Logical Volume: on ganeti, var, 16G - Finish - Edit the Logical Volumes to be ext4 /, swap, and ext4 /var At this point, the partitioning looks like this [![raid10-part.jpg](https://wiki.rg.net/chrome/site/raid10-part.jpg)](https://wiki.rg.net/chrome/site/raid10-part.jpg) - Finish partitioning and write changes - Finish partitioning and write changes to disk - Ubuntu will now install and you will do the normal mirror selection etc. - I choose no automatic updates as I ansible that in later - Only install SSH Server, System Utilities - Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask - Be sure it will not boot CD-ROM, and Reboot from the installed system ## Finish Ubuntu Installation Install homey things (it's not a computer without emacs:) ``` apt update apt upgrade apt dist-upgrade apt install emacs-nox gcc bridge-utils vlan unbound usermod -G sudo -a randy ``` As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot. The next thing I do is to copy root's credentials and dot-files from a known system Fix `/etc/ssh/sshd_config` to forbid passwords ``` PermitRootLogin without-password PasswordAuthentication no ``` And restart the ssh daemon ``` service ssh restart ``` You can do most of the rest from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI. Set the timezone ``` dpkg-reconfigure tzdata ``` In the wonderful world of systemd, here is how to get ntpd going ``` timedatectl set-ntp true cat > /etc/systemd/timesyncd.conf << EOF # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See timesyncd.conf(5) for details. [Time] NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org FallbackNTP=ntp.ubuntu.com EOF ``` On some Xenial systems, to get unbound to start reliably, I have had to ``` cat > /etc/systemd/user/unbound.service << EOF [Unit] Description=Unbound is a validating, recursive, and caching DNS resolver. After=network.target networking.service [Service] Type=simple ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key ExecStartPre=/usr/local/sbin/unbound-checkconf ExecStart=/usr/local/sbin/unbound -d LimitNOFILE=102400 LimitNPROC=65535 [Install] WantedBy=multi-user.target EOF ``` Fix hostname ``` echo vm0.sea.rg.net > /etc/hostname hostname `cat /etc/hostname` ``` Fix /etc/unbound/unbound.conf ``` access-control: 127.0.0.0/8 allow access-control: 198.180.152.0/24 allow access-control: 0.0.0.0/0 refuse access-control: ::1 allow access-control: ::ffff:127.0.0.1 allow access-control: 2001:deb::/48 access-control: ::0/0 refuse ``` And restart unbound ``` service unbound restart ``` I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN. Edit `/etc/default/grub` to have ``` GRUB_HIDDEN_TIMEOUT_QUIET=false GRUB_TIMEOUT=6 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8" ``` And reconfigure grub ``` # update-grub ``` Clean up from CDROM sources ``` emacs /etc/apt/sources.list ``` and delete the two CDROM entries at the top. We want to manage the server so install ipmitool and smartmontools ``` modprobe ipmi_si modprobe ipmi_devintf echo ipmi_si >> /etc/modules echo ipmi_devintf >> /etc/modules apt install ipmitool ``` Set up IPMI ``` # turn off gratuitous arp ipmitool lan set 1 arp generate off # ip addressing ipmitool lan set 1 ipsrc static ipmitool lan set 1 ipaddr 147.28.0.240 ipmitool lan set 1 netmask 255.255.255.0 ipmitool lan set 1 defgw ipaddr 147.28.0.1 ipmitool lan print 1 # user(s) ipmitool user set name 2 ipmitool user set password 2 ipmitool user priv 2 4 ipmitool user enable 2 ipmitool user list ``` Set up smartmontools to watch the disks ``` apt install smartmontools ``` And check that smartd is running ``` root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n ``` I configure smartd to do short test every day and long once a week ``` cat >> /etc/smartd.conf << EOF /dev/sda -o on -S on -s (S/../.././02|L/../../3/03) /dev/sdb -o on -S on -s (S/../.././04|L/../../4/03) /dev/sdc -o on -S on -s (S/../.././06|L/../../5/03) /dev/sdd -o on -S on -s (S/../.././08|L/../../6/03) EOF ``` ### System Services It is good to have a mail system to send logs to the sucker who watches such things. I use exim4 ``` apt install exim4 mailtools dpkg-reconfigure exim4-config ``` Use internet mail System name is vm0.sea.rg.net Only listen on local interfaces Relay mail for noone ### Unattended Upgrades, syslog-NG, etc. [Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade) Install syslog-ng ``` apt-get install syslog-ng ``` Hack /etc/logrotate.d/syslog-ng ``` *** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000 --- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100 *************** *** 29,34 **** --- 29,36 ---- missingok notifempty compress + mailfirst + mail randy@psg.com delaycompress sharedscripts postrotate ``` ## Ubuntu Ganeti Specific Configuration Edit /etc/hosts to have the real address of the host, e.g. ``` cat > /etc/hosts << EOF 127.0.0.1 localhost 147.28.0.14 vm0.sea.rg.net vm0 147.28.0.15 vm1.sea.rg.net vm1 147.28.0.16 vm2.sea.rg.net vm2 147.28.0.100 gnt0.sea.rg.net gnt0 EOF ``` ## Install Ganeti In Ubuntu 16.04, ganeti is in the package system, no need for hacks. Check what version will be installed ``` apt-cache policy ganeti ganeti: Installed: (none) Candidate: 2.15.2-3 Version table: 2.15.2-3 500 500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages 500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages ``` So now we can explicitly install 2.15 ``` apt install ganeti ``` And it's a non-trivial dependency tree, to say the least ``` Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto python-cryptography python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr python-ipaddress python-openssl python-paramiko python-pkg-resources python-psutil python-pyasn1 python-pycurl python-pyinotify python-pyparsing python-simplejson python-six socat Suggested packages: ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc python-cryptography-doc python-cryptography-vectors python-enum34-doc python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc python-pyinotify-doc Recommended packages: drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64 | xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery The following NEW packages will be installed: fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto python-cryptography python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr python-ipaddress python-openssl python-paramiko python-pkg-resources python-psutil python-pyasn1 python-pycurl python-pyinotify python-pyparsing python-simplejson python-six socat 0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded. Need to get 19.3 MB of archives. After this operation, 141 MB of additional disk space will be used. Do you want to continue? [Y/n] ``` Fix up drbd ``` echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf rmmod drbd # ignore any error modprobe drbd ```