From 39a0f01e9bdb58803adf23f6fe99ab2f12c37c63 Mon Sep 17 00:00:00 2001 From: Randy Bush Date: Sat, 4 Jul 2020 14:57:29 -0700 Subject: [PATCH] from wiki --- pages/XenialInstall.md | 359 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 359 insertions(+) create mode 100644 pages/XenialInstall.md diff --git a/pages/XenialInstall.md b/pages/XenialInstall.md new file mode 100644 index 0000000..b237a0e --- /dev/null +++ b/pages/XenialInstall.md @@ -0,0 +1,359 @@ +# Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node + +## Install Ubuntu + +- Before install, it is wise to disable DHCP service so one can manually set the system IP data +- Boot Ubuntu CD/ISO +- Choose Install Ubuntu Server +- Choose US English or it will choose the wrong apt server set +- Choose American English +- CHoose Macintosh keyboard +- Select the Ethernet interface to be used, I had connected eno1 +- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service +- Configure Hostname +- Choose user name and password +- Do not encrypt home directory +- Let it set some stupid timezone, you can fix later +- Partition Disks + - Choose Manual Partitioning + - Select the first drive + - Create a new empty partition table + - Select each of the other Drives and Create a new empty partition table + - Select the free space on each device and create a 256M ESP partition named /boot/efi + - Configure Software RAID + - Write Changes and Configure RAID + - Create MD Device + - Select RAID10 - with lots of disk, I chose RAID10 for speed + - Number of Devices should be all the drives you have, but an even number + - Number of Spare Drives is 0, unless you had an odd number of drives + - Select all the drives (not the EFI partitions) and continue + - Write the changes and configure RAID + - Finish +- Configure LVM + - Configure LVM accepting write changes to disks + - Create volume group + - Volume group name: ganeti + - Devices for the new volume group: select /dev/md0 + - Keep partioning and write + - Create Logical Volume: on ganeti, root, 16G + - Create Logical Volume: on ganeti, swap, size of RAM + - Create Logical Volume: on ganeti, var, 16G + - Finish + - Edit the Logical Volumes to be ext4 /, swap, and ext4 /var + +At this point, the partitioning looks like this + [![raid10-part.jpg](https://wiki.rg.net/chrome/site/raid10-part.jpg)](https://wiki.rg.net/chrome/site/raid10-part.jpg) + +- Finish partitioning and write changes +- Finish partitioning and write changes to disk +- Ubuntu will now install and you will do the normal mirror selection etc. +- I choose no automatic updates as I ansible that in later +- Only install SSH Server, System Utilities +- Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask +- Be sure it will not boot CD-ROM, and Reboot from the installed system + +## Finish Ubuntu Installation + +Install homey things (it's not a computer without emacs:) + +``` +apt update +apt upgrade +apt dist-upgrade +apt install emacs-nox gcc bridge-utils vlan unbound +usermod -G sudo -a randy +``` + +As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot. + +The next thing I do is to copy root's credentials and dot-files from a known system + +Fix `/etc/ssh/sshd_config` to forbid passwords + +``` +PermitRootLogin without-password +PasswordAuthentication no +``` + +And restart the ssh daemon + +``` +service ssh restart +``` + +You can do most of the rest from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI. + +Set the timezone + +``` +dpkg-reconfigure tzdata +``` + +In the wonderful world of systemd, here is how to get ntpd going + +``` +timedatectl set-ntp true +cat > /etc/systemd/timesyncd.conf << EOF +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. +# +# Entries in this file show the compile time defaults. +# You can change settings by editing this file. +# Defaults can be restored by simply deleting this file. +# +# See timesyncd.conf(5) for details. + +[Time] +NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org +FallbackNTP=ntp.ubuntu.com +EOF +``` + +On some Xenial systems, to get unbound to start reliably, I have had to + +``` +cat > /etc/systemd/user/unbound.service << EOF +[Unit] +Description=Unbound is a validating, recursive, and caching DNS resolver. +After=network.target networking.service + +[Service] +Type=simple +ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key +ExecStartPre=/usr/local/sbin/unbound-checkconf +ExecStart=/usr/local/sbin/unbound -d +LimitNOFILE=102400 +LimitNPROC=65535 + +[Install] +WantedBy=multi-user.target +EOF +``` + +Fix hostname + +``` +echo vm0.sea.rg.net > /etc/hostname +hostname `cat /etc/hostname` +``` + +Fix /etc/unbound/unbound.conf + +``` + access-control: 127.0.0.0/8 allow + access-control: 198.180.152.0/24 allow + access-control: 0.0.0.0/0 refuse + access-control: ::1 allow + access-control: ::ffff:127.0.0.1 allow + access-control: 2001:deb::/48 + access-control: ::0/0 refuse +``` + +And restart unbound + +``` +service unbound restart +``` + +I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN. Edit `/etc/default/grub` to have + +``` +GRUB_HIDDEN_TIMEOUT_QUIET=false +GRUB_TIMEOUT=6 +GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` +GRUB_CMDLINE_LINUX_DEFAULT="" +GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8" +``` + +And reconfigure grub + +``` +# update-grub +``` + +Clean up from CDROM sources + +``` +emacs /etc/apt/sources.list +``` + +and delete the two CDROM entries at the top. + +We want to manage the server so install ipmitool and smartmontools + +``` +modprobe ipmi_si +modprobe ipmi_devintf +echo ipmi_si >> /etc/modules +echo ipmi_devintf >> /etc/modules +apt install ipmitool +``` + +Set up IPMI + +``` +# turn off gratuitous arp +ipmitool lan set 1 arp generate off +# ip addressing +ipmitool lan set 1 ipsrc static +ipmitool lan set 1 ipaddr 147.28.0.240 +ipmitool lan set 1 netmask 255.255.255.0 +ipmitool lan set 1 defgw ipaddr 147.28.0.1 +ipmitool lan print 1 +# user(s) +ipmitool user set name 2 +ipmitool user set password 2 +ipmitool user priv 2 4 +ipmitool user enable 2 +ipmitool user list +``` + +Set up smartmontools to watch the disks + +``` +apt install smartmontools +``` + +And check that smartd is running + +``` +root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n +``` + +I configure smartd to do short test every day and long once a week + +``` +cat >> /etc/smartd.conf << EOF + +/dev/sda -o on -S on -s (S/../.././02|L/../../3/03) +/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03) +/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03) +/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03) +EOF +``` + +### System Services + +It is good to have a mail system to send logs to the sucker who watches such things. I use exim4 + +``` +apt install exim4 mailtools +dpkg-reconfigure exim4-config +``` + +Use internet mail + System name is vm0.sea.rg.net + Only listen on local interfaces + Relay mail for noone + +### Unattended Upgrades, syslog-NG, etc. + +[Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade) + +Install syslog-ng + +``` +apt-get install syslog-ng +``` + +Hack /etc/logrotate.d/syslog-ng + +``` +*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000 +--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100 +*************** +*** 29,34 **** +--- 29,36 ---- + missingok + notifempty + compress ++ mailfirst ++ mail randy@psg.com + delaycompress + sharedscripts + postrotate +``` + +## Ubuntu Ganeti Specific Configuration + +Edit /etc/hosts to have the real address of the host, e.g. + +``` +cat > /etc/hosts << EOF +127.0.0.1 localhost +147.28.0.14 vm0.sea.rg.net vm0 +147.28.0.15 vm1.sea.rg.net vm1 +147.28.0.16 vm2.sea.rg.net vm2 +147.28.0.100 gnt0.sea.rg.net gnt0 +EOF +``` + +## Install Ganeti + +In Ubuntu 16.04, ganeti is in the package system, no need for hacks. + +Check what version will be installed + +``` +apt-cache policy ganeti +ganeti: + Installed: (none) + Candidate: 2.15.2-3 + Version table: + 2.15.2-3 500 + 500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages + 500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages +``` + +So now we can explicitly install 2.15 + +``` +apt install ganeti +``` + +And it's a non-trivial dependency tree, to say the least + +``` +Reading package lists... Done +Building dependency tree +Reading state information... Done +The following additional packages will be installed: + fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping + libcurl3 python-bitarray python-cffi-backend python-crypto + python-cryptography python-ecdsa python-enum34 python-fdsend python-idna + python-ipaddr python-ipaddress python-openssl python-paramiko + python-pkg-resources python-psutil python-pyasn1 python-pycurl + python-pyinotify python-pyparsing python-simplejson python-six socat +Suggested packages: + ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc + python-cryptography-doc python-cryptography-vectors python-enum34-doc + python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc + doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc + python-pyinotify-doc +Recommended packages: + drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64 + | xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery +The following NEW packages will be installed: + fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 + iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto + python-cryptography python-ecdsa python-enum34 python-fdsend python-idna + python-ipaddr python-ipaddress python-openssl python-paramiko + python-pkg-resources python-psutil python-pyasn1 python-pycurl + python-pyinotify python-pyparsing python-simplejson python-six socat +0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded. +Need to get 19.3 MB of archives. +After this operation, 141 MB of additional disk space will be used. +Do you want to continue? [Y/n] +``` + +Fix up drbd + +``` +echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf +rmmod drbd # ignore any error +modprobe drbd +``` \ No newline at end of file