randy/rgnet-wiki
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

from wiki

Browse source
This commit is contained in:
Randy Bush 2020-07-04 14:57:29 -07:00
parent e5498a561a
commit 39a0f01e9b
1 changed files with 359 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

359
pages/XenialInstall.md Normal file
View file

@ -0,0 +1,359 @@
# Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node
## Install Ubuntu
- Before install, it is wise to disable DHCP service so one can manually set the system IP data
- Boot Ubuntu CD/ISO
- Choose Install Ubuntu Server
- Choose US English or it will choose the wrong apt server set
- Choose American English
- CHoose Macintosh keyboard
- Select the Ethernet interface to be used, I had connected eno1
- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
- Configure Hostname
- Choose user name and password
- Do not encrypt home directory
- Let it set some stupid timezone, you can fix later
- Partition Disks
- Choose Manual Partitioning
- Select the first drive
- Create a new empty partition table
- Select each of the other Drives and Create a new empty partition table
- Select the free space on each device and create a 256M ESP partition named /boot/efi
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID10 - with lots of disk, I chose RAID10 for speed
- Number of Devices should be all the drives you have, but an even number
- Number of Spare Drives is 0, unless you had an odd number of drives
- Select all the drives (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
- Configure LVM
- Configure LVM accepting write changes to disks
- Create volume group
- Volume group name: ganeti
- Devices for the new volume group: select /dev/md0
- Keep partioning and write
- Create Logical Volume: on ganeti, root, 16G
- Create Logical Volume: on ganeti, swap, size of RAM
- Create Logical Volume: on ganeti, var, 16G
- Finish
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
At this point, the partitioning looks like this
[![raid10-part.jpg](https://wiki.rg.net/chrome/site/raid10-part.jpg)](https://wiki.rg.net/chrome/site/raid10-part.jpg)
- Finish partitioning and write changes
- Finish partitioning and write changes to disk
- Ubuntu will now install and you will do the normal mirror selection etc.
- I choose no automatic updates as I ansible that in later
- Only install SSH Server, System Utilities
- Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask
- Be sure it will not boot CD-ROM, and Reboot from the installed system
## Finish Ubuntu Installation
Install homey things (it's not a computer without emacs:)
```
apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox gcc bridge-utils vlan unbound
usermod -G sudo -a randy
```
As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot.
The next thing I do is to copy root's credentials and dot-files from a known system
Fix `/etc/ssh/sshd_config` to forbid passwords
```
PermitRootLogin without-password
PasswordAuthentication no
```
And restart the ssh daemon
```
service ssh restart
```
You can do most of the rest from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.
Set the timezone
```
dpkg-reconfigure tzdata
```
In the wonderful world of systemd, here is how to get ntpd going
```
timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.
[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF
```
On some Xenial systems, to get unbound to start reliably, I have had to
```
cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service
[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535
[Install]
WantedBy=multi-user.target
EOF
```
Fix hostname
```
echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`
```
Fix /etc/unbound/unbound.conf
```
access-control: 127.0.0.0/8 allow
access-control: 198.180.152.0/24 allow
access-control: 0.0.0.0/0 refuse
access-control: ::1 allow
access-control: ::ffff:127.0.0.1 allow
access-control: 2001:deb::/48
access-control: ::0/0 refuse
```
And restart unbound
```
service unbound restart
```
I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN. Edit `/etc/default/grub` to have
```
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=6
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8"
```
And reconfigure grub
```
# update-grub
```
Clean up from CDROM sources
```
emacs /etc/apt/sources.list
```
and delete the two CDROM entries at the top.
We want to manage the server so install ipmitool and smartmontools
```
modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool
```
Set up IPMI
```
# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list
```
Set up smartmontools to watch the disks
```
apt install smartmontools
```
And check that smartd is running
```
root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n
```
I configure smartd to do short test every day and long once a week
```
cat >> /etc/smartd.conf << EOF
/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF
```
### System Services
It is good to have a mail system to send logs to the sucker who watches such things. I use exim4
```
apt install exim4 mailtools
dpkg-reconfigure exim4-config
```
Use internet mail
System name is vm0.sea.rg.net
Only listen on local interfaces
Relay mail for noone
### Unattended Upgrades, syslog-NG, etc.
[Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade)
Install syslog-ng
```
apt-get install syslog-ng
```
Hack /etc/logrotate.d/syslog-ng
```
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
missingok
notifempty
compress
+ mailfirst
+ mail randy@psg.com
delaycompress
sharedscripts
postrotate
```
## Ubuntu Ganeti Specific Configuration
Edit /etc/hosts to have the real address of the host, e.g.
```
cat > /etc/hosts << EOF
127.0.0.1 localhost
147.28.0.14 vm0.sea.rg.net vm0
147.28.0.15 vm1.sea.rg.net vm1
147.28.0.16 vm2.sea.rg.net vm2
147.28.0.100 gnt0.sea.rg.net gnt0
EOF
```
## Install Ganeti
In Ubuntu 16.04, ganeti is in the package system, no need for hacks.
Check what version will be installed
```
apt-cache policy ganeti
ganeti:
Installed: (none)
Candidate: 2.15.2-3
Version table:
2.15.2-3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
```
So now we can explicitly install 2.15
```
apt install ganeti
```
And it's a non-trivial dependency tree, to say the least
```
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping
libcurl3 python-bitarray python-cffi-backend python-crypto
python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
python-ipaddr python-ipaddress python-openssl python-paramiko
python-pkg-resources python-psutil python-pyasn1 python-pycurl
python-pyinotify python-pyparsing python-simplejson python-six socat
Suggested packages:
ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
python-cryptography-doc python-cryptography-vectors python-enum34-doc
python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc
doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc
python-pyinotify-doc
Recommended packages:
drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64
| xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto
python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
python-ipaddr python-ipaddress python-openssl python-paramiko
python-pkg-resources python-psutil python-pyasn1 python-pycurl
python-pyinotify python-pyparsing python-simplejson python-six socat
0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.3 MB of archives.
After this operation, 141 MB of additional disk space will be used.
Do you want to continue? [Y/n]
```
Fix up drbd
```
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd # ignore any error
modprobe drbd
```