359 lines
No EOL
9.8 KiB
Markdown
359 lines
No EOL
9.8 KiB
Markdown
# Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node
|
|
|
|
## Install Ubuntu
|
|
|
|
- Before install, it is wise to disable DHCP service so one can manually set the system IP data
|
|
- Boot Ubuntu CD/ISO
|
|
- Choose Install Ubuntu Server
|
|
- Choose US English or it will choose the wrong apt server set
|
|
- Choose American English
|
|
- CHoose Macintosh keyboard
|
|
- Select the Ethernet interface to be used, I had connected eno1
|
|
- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
|
|
- Configure Hostname
|
|
- Choose user name and password
|
|
- Do not encrypt home directory
|
|
- Let it set some stupid timezone, you can fix later
|
|
- Partition Disks
|
|
- Choose Manual Partitioning
|
|
- Select the first drive
|
|
- Create a new empty partition table
|
|
- Select each of the other Drives and Create a new empty partition table
|
|
- Select the free space on each device and create a 256M ESP partition named /boot/efi
|
|
- Configure Software RAID
|
|
- Write Changes and Configure RAID
|
|
- Create MD Device
|
|
- Select RAID10 - with lots of disk, I chose RAID10 for speed
|
|
- Number of Devices should be all the drives you have, but an even number
|
|
- Number of Spare Drives is 0, unless you had an odd number of drives
|
|
- Select all the drives (not the EFI partitions) and continue
|
|
- Write the changes and configure RAID
|
|
- Finish
|
|
- Configure LVM
|
|
- Configure LVM accepting write changes to disks
|
|
- Create volume group
|
|
- Volume group name: ganeti
|
|
- Devices for the new volume group: select /dev/md0
|
|
- Keep partioning and write
|
|
- Create Logical Volume: on ganeti, root, 16G
|
|
- Create Logical Volume: on ganeti, swap, size of RAM
|
|
- Create Logical Volume: on ganeti, var, 16G
|
|
- Finish
|
|
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
|
|
|
|
At this point, the partitioning looks like this
|
|
[](https://wiki.rg.net/chrome/site/raid10-part.jpg)
|
|
|
|
- Finish partitioning and write changes
|
|
- Finish partitioning and write changes to disk
|
|
- Ubuntu will now install and you will do the normal mirror selection etc.
|
|
- I choose no automatic updates as I ansible that in later
|
|
- Only install SSH Server, System Utilities
|
|
- Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask
|
|
- Be sure it will not boot CD-ROM, and Reboot from the installed system
|
|
|
|
## Finish Ubuntu Installation
|
|
|
|
Install homey things (it's not a computer without emacs:)
|
|
|
|
```
|
|
apt update
|
|
apt upgrade
|
|
apt dist-upgrade
|
|
apt install emacs-nox gcc bridge-utils vlan unbound
|
|
usermod -G sudo -a randy
|
|
```
|
|
|
|
As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot.
|
|
|
|
The next thing I do is to copy root's credentials and dot-files from a known system
|
|
|
|
Fix `/etc/ssh/sshd_config` to forbid passwords
|
|
|
|
```
|
|
PermitRootLogin without-password
|
|
PasswordAuthentication no
|
|
```
|
|
|
|
And restart the ssh daemon
|
|
|
|
```
|
|
service ssh restart
|
|
```
|
|
|
|
You can do most of the rest from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.
|
|
|
|
Set the timezone
|
|
|
|
```
|
|
dpkg-reconfigure tzdata
|
|
```
|
|
|
|
In the wonderful world of systemd, here is how to get ntpd going
|
|
|
|
```
|
|
timedatectl set-ntp true
|
|
cat > /etc/systemd/timesyncd.conf << EOF
|
|
# This file is part of systemd.
|
|
#
|
|
# systemd is free software; you can redistribute it and/or modify it
|
|
# under the terms of the GNU Lesser General Public License as published by
|
|
# the Free Software Foundation; either version 2.1 of the License, or
|
|
# (at your option) any later version.
|
|
#
|
|
# Entries in this file show the compile time defaults.
|
|
# You can change settings by editing this file.
|
|
# Defaults can be restored by simply deleting this file.
|
|
#
|
|
# See timesyncd.conf(5) for details.
|
|
|
|
[Time]
|
|
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
|
|
FallbackNTP=ntp.ubuntu.com
|
|
EOF
|
|
```
|
|
|
|
On some Xenial systems, to get unbound to start reliably, I have had to
|
|
|
|
```
|
|
cat > /etc/systemd/user/unbound.service << EOF
|
|
[Unit]
|
|
Description=Unbound is a validating, recursive, and caching DNS resolver.
|
|
After=network.target networking.service
|
|
|
|
[Service]
|
|
Type=simple
|
|
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
|
|
ExecStartPre=/usr/local/sbin/unbound-checkconf
|
|
ExecStart=/usr/local/sbin/unbound -d
|
|
LimitNOFILE=102400
|
|
LimitNPROC=65535
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
EOF
|
|
```
|
|
|
|
Fix hostname
|
|
|
|
```
|
|
echo vm0.sea.rg.net > /etc/hostname
|
|
hostname `cat /etc/hostname`
|
|
```
|
|
|
|
Fix /etc/unbound/unbound.conf
|
|
|
|
```
|
|
access-control: 127.0.0.0/8 allow
|
|
access-control: 198.180.152.0/24 allow
|
|
access-control: 0.0.0.0/0 refuse
|
|
access-control: ::1 allow
|
|
access-control: ::ffff:127.0.0.1 allow
|
|
access-control: 2001:deb::/48
|
|
access-control: ::0/0 refuse
|
|
```
|
|
|
|
And restart unbound
|
|
|
|
```
|
|
service unbound restart
|
|
```
|
|
|
|
I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN. Edit `/etc/default/grub` to have
|
|
|
|
```
|
|
GRUB_HIDDEN_TIMEOUT_QUIET=false
|
|
GRUB_TIMEOUT=6
|
|
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
|
|
GRUB_CMDLINE_LINUX_DEFAULT=""
|
|
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8"
|
|
```
|
|
|
|
And reconfigure grub
|
|
|
|
```
|
|
# update-grub
|
|
```
|
|
|
|
Clean up from CDROM sources
|
|
|
|
```
|
|
emacs /etc/apt/sources.list
|
|
```
|
|
|
|
and delete the two CDROM entries at the top.
|
|
|
|
We want to manage the server so install ipmitool and smartmontools
|
|
|
|
```
|
|
modprobe ipmi_si
|
|
modprobe ipmi_devintf
|
|
echo ipmi_si >> /etc/modules
|
|
echo ipmi_devintf >> /etc/modules
|
|
apt install ipmitool
|
|
```
|
|
|
|
Set up IPMI
|
|
|
|
```
|
|
# turn off gratuitous arp
|
|
ipmitool lan set 1 arp generate off
|
|
# ip addressing
|
|
ipmitool lan set 1 ipsrc static
|
|
ipmitool lan set 1 ipaddr 147.28.0.240
|
|
ipmitool lan set 1 netmask 255.255.255.0
|
|
ipmitool lan set 1 defgw ipaddr 147.28.0.1
|
|
ipmitool lan print 1
|
|
# user(s)
|
|
ipmitool user set name 2 <whatever>
|
|
ipmitool user set password 2 <secret>
|
|
ipmitool user priv 2 4
|
|
ipmitool user enable 2
|
|
ipmitool user list
|
|
```
|
|
|
|
Set up smartmontools to watch the disks
|
|
|
|
```
|
|
apt install smartmontools
|
|
```
|
|
|
|
And check that smartd is running
|
|
|
|
```
|
|
root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n
|
|
```
|
|
|
|
I configure smartd to do short test every day and long once a week
|
|
|
|
```
|
|
cat >> /etc/smartd.conf << EOF
|
|
|
|
/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
|
|
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
|
|
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
|
|
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
|
|
EOF
|
|
```
|
|
|
|
### System Services
|
|
|
|
It is good to have a mail system to send logs to the sucker who watches such things. I use exim4
|
|
|
|
```
|
|
apt install exim4 mailtools
|
|
dpkg-reconfigure exim4-config
|
|
```
|
|
|
|
Use internet mail
|
|
System name is vm0.sea.rg.net
|
|
Only listen on local interfaces
|
|
Relay mail for noone
|
|
|
|
### Unattended Upgrades, syslog-NG, etc.
|
|
|
|
[Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade)
|
|
|
|
Install syslog-ng
|
|
|
|
```
|
|
apt-get install syslog-ng
|
|
```
|
|
|
|
Hack /etc/logrotate.d/syslog-ng
|
|
|
|
```
|
|
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
|
|
--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100
|
|
***************
|
|
*** 29,34 ****
|
|
--- 29,36 ----
|
|
missingok
|
|
notifempty
|
|
compress
|
|
+ mailfirst
|
|
+ mail randy@psg.com
|
|
delaycompress
|
|
sharedscripts
|
|
postrotate
|
|
```
|
|
|
|
## Ubuntu Ganeti Specific Configuration
|
|
|
|
Edit /etc/hosts to have the real address of the host, e.g.
|
|
|
|
```
|
|
cat > /etc/hosts << EOF
|
|
127.0.0.1 localhost
|
|
147.28.0.14 vm0.sea.rg.net vm0
|
|
147.28.0.15 vm1.sea.rg.net vm1
|
|
147.28.0.16 vm2.sea.rg.net vm2
|
|
147.28.0.100 gnt0.sea.rg.net gnt0
|
|
EOF
|
|
```
|
|
|
|
## Install Ganeti
|
|
|
|
In Ubuntu 16.04, ganeti is in the package system, no need for hacks.
|
|
|
|
Check what version will be installed
|
|
|
|
```
|
|
apt-cache policy ganeti
|
|
ganeti:
|
|
Installed: (none)
|
|
Candidate: 2.15.2-3
|
|
Version table:
|
|
2.15.2-3 500
|
|
500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
|
|
500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
|
|
```
|
|
|
|
So now we can explicitly install 2.15
|
|
|
|
```
|
|
apt install ganeti
|
|
```
|
|
|
|
And it's a non-trivial dependency tree, to say the least
|
|
|
|
```
|
|
Reading package lists... Done
|
|
Building dependency tree
|
|
Reading state information... Done
|
|
The following additional packages will be installed:
|
|
fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping
|
|
libcurl3 python-bitarray python-cffi-backend python-crypto
|
|
python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
|
|
python-ipaddr python-ipaddress python-openssl python-paramiko
|
|
python-pkg-resources python-psutil python-pyasn1 python-pycurl
|
|
python-pyinotify python-pyparsing python-simplejson python-six socat
|
|
Suggested packages:
|
|
ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
|
|
python-cryptography-doc python-cryptography-vectors python-enum34-doc
|
|
python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc
|
|
doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc
|
|
python-pyinotify-doc
|
|
Recommended packages:
|
|
drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64
|
|
| xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
|
|
The following NEW packages will be installed:
|
|
fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
|
|
iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto
|
|
python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
|
|
python-ipaddr python-ipaddress python-openssl python-paramiko
|
|
python-pkg-resources python-psutil python-pyasn1 python-pycurl
|
|
python-pyinotify python-pyparsing python-simplejson python-six socat
|
|
0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded.
|
|
Need to get 19.3 MB of archives.
|
|
After this operation, 141 MB of additional disk space will be used.
|
|
Do you want to continue? [Y/n]
|
|
```
|
|
|
|
Fix up drbd
|
|
|
|
```
|
|
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
|
|
rmmod drbd # ignore any error
|
|
modprobe drbd
|
|
``` |