9.8 KiB
9.8 KiB
Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node
Install Ubuntu
- Before install, it is wise to disable DHCP service so one can manually set the system IP data
- Boot Ubuntu CD/ISO
- Choose Install Ubuntu Server
- Choose US English or it will choose the wrong apt server set
- Choose American English
- CHoose Macintosh keyboard
- Select the Ethernet interface to be used, I had connected eno1
- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
- Configure Hostname
- Choose user name and password
- Do not encrypt home directory
- Let it set some stupid timezone, you can fix later
- Partition Disks
- Choose Manual Partitioning
- Select the first drive
- Create a new empty partition table
- Select each of the other Drives and Create a new empty partition table
- Select the free space on each device and create a 256M ESP partition named /boot/efi
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID10 - with lots of disk, I chose RAID10 for speed
- Number of Devices should be all the drives you have, but an even number
- Number of Spare Drives is 0, unless you had an odd number of drives
- Select all the drives (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
- Configure LVM
- Configure LVM accepting write changes to disks
- Create volume group
- Volume group name: ganeti
- Devices for the new volume group: select /dev/md0
- Keep partioning and write
- Create Logical Volume: on ganeti, root, 16G
- Create Logical Volume: on ganeti, swap, size of RAM
- Create Logical Volume: on ganeti, var, 16G
- Finish
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
At this point, the partitioning looks like this
- Finish partitioning and write changes
- Finish partitioning and write changes to disk
- Ubuntu will now install and you will do the normal mirror selection etc.
- I choose no automatic updates as I ansible that in later
- Only install SSH Server, System Utilities
- Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask
- Be sure it will not boot CD-ROM, and Reboot from the installed system
Finish Ubuntu Installation
Install homey things (it's not a computer without emacs:)
apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox gcc bridge-utils vlan unbound
usermod -G sudo -a randy
As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot.
The next thing I do is to copy root's credentials and dot-files from a known system
Fix /etc/ssh/sshd_config to forbid passwords
PermitRootLogin without-password
PasswordAuthentication no
And restart the ssh daemon
service ssh restart
You can do most of the rest from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.
Set the timezone
dpkg-reconfigure tzdata
In the wonderful world of systemd, here is how to get ntpd going
timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.
[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF
On some Xenial systems, to get unbound to start reliably, I have had to
cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service
[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535
[Install]
WantedBy=multi-user.target
EOF
Fix hostname
echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`
Fix /etc/unbound/unbound.conf
access-control: 127.0.0.0/8 allow
access-control: 198.180.152.0/24 allow
access-control: 0.0.0.0/0 refuse
access-control: ::1 allow
access-control: ::ffff:127.0.0.1 allow
access-control: 2001:deb::/48
access-control: ::0/0 refuse
And restart unbound
service unbound restart
I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN. Edit /etc/default/grub to have
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=6
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8"
And reconfigure grub
# update-grub
Clean up from CDROM sources
emacs /etc/apt/sources.list
and delete the two CDROM entries at the top.
We want to manage the server so install ipmitool and smartmontools
modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool
Set up IPMI
# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list
Set up smartmontools to watch the disks
apt install smartmontools
And check that smartd is running
root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n
I configure smartd to do short test every day and long once a week
cat >> /etc/smartd.conf << EOF
/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF
System Services
It is good to have a mail system to send logs to the sucker who watches such things. I use exim4
apt install exim4 mailtools
dpkg-reconfigure exim4-config
Use internet mail System name is vm0.sea.rg.net Only listen on local interfaces Relay mail for noone
Unattended Upgrades, syslog-NG, etc.
Install syslog-ng
apt-get install syslog-ng
Hack /etc/logrotate.d/syslog-ng
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
missingok
notifempty
compress
+ mailfirst
+ mail randy@psg.com
delaycompress
sharedscripts
postrotate
Ubuntu Ganeti Specific Configuration
Edit /etc/hosts to have the real address of the host, e.g.
cat > /etc/hosts << EOF
127.0.0.1 localhost
147.28.0.14 vm0.sea.rg.net vm0
147.28.0.15 vm1.sea.rg.net vm1
147.28.0.16 vm2.sea.rg.net vm2
147.28.0.100 gnt0.sea.rg.net gnt0
EOF
Install Ganeti
In Ubuntu 16.04, ganeti is in the package system, no need for hacks.
Check what version will be installed
apt-cache policy ganeti
ganeti:
Installed: (none)
Candidate: 2.15.2-3
Version table:
2.15.2-3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
So now we can explicitly install 2.15
apt install ganeti
And it's a non-trivial dependency tree, to say the least
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping
libcurl3 python-bitarray python-cffi-backend python-crypto
python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
python-ipaddr python-ipaddress python-openssl python-paramiko
python-pkg-resources python-psutil python-pyasn1 python-pycurl
python-pyinotify python-pyparsing python-simplejson python-six socat
Suggested packages:
ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
python-cryptography-doc python-cryptography-vectors python-enum34-doc
python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc
doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc
python-pyinotify-doc
Recommended packages:
drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64
| xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto
python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
python-ipaddr python-ipaddress python-openssl python-paramiko
python-pkg-resources python-psutil python-pyasn1 python-pycurl
python-pyinotify python-pyparsing python-simplejson python-six socat
0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.3 MB of archives.
After this operation, 141 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Fix up drbd
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd # ignore any error
modprobe drbd