13 KiB
13 KiB
Build a Software RAID5 Based Debian 8.0 Ganeti 2.15 Node on a UEFI BIOS
This is for UEFI based BIOS. Be sure to enable UEFI for the hard drives and for the install boot device.
Install Debian
- Before install, it is wise to disable DHCP service so one can manuall set the system IP data
- Also, the BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
- Boot Debian CD/ISO
- Choose Install Debian Server
- Choose US English or it will choose the wrong apt server set
- Choose American English
- CHoose Macintosh keyboard
- Select the Ethernet interface to be used
- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
- Configure Hostname & Domain Name
- Choose user name and password
- Let it set some stupid timezone, you can fix later
- Partition Disks
- Select the first drive
- Create a new empty partition table
- Select each of the other Drives and Create a new empty partition table
- Select the free space on each device and
- create a 256M ESP partition at the beginning
- name it /boot/efi
- Use as: EFI System Partition
- Be sure it is bootable
- Done setting up the partition
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID5 - I had problems with RAID10 on Ubuntu
- Number of Devices should be all the drives you have
- Number of Spare Drives is 0
- Select all the free partitions (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
- Configure Logical Volume Manager
- Keep current partition layout and configure LVM
- Create volume group
- Volume group name: ganeti
- Devices for the new volume group: select /dev/md0
- Continue
- Keep current partioning layout and configure LVM
- Create Logical Volume: on ganeti, root, 16G
- Create Logical Volume: on ganeti, swap, 128G (at least RAM size)
- Create Logical Volume: on ganeti, var, 16G
- Finish
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
- Finish partitioning and write changes
At this point, the partitioning looks like this
- Finish partitioning and write changes to disk
- Debian will now install and you will do the normal mirror selection etc.
- Only install SSH Server and, System Utilities
- I choose no automatic updates as I ansible that in later
- Install GRUB2, except with EFI it eems not to ask
- Be sure it will not boot CD-ROM, and Reboot from the installed system
If Debian does NOT auto-boot (thanks Joe Clark)
- Once Debian is installed, boot the server into the EFI Shell. This can be done by hitting F6 during the BIOS boot (you will be prompted on screen) to enter the boot list manager. Select the shell from the boot list.
- In the shell, run the following command:
bcfg boot dump - Look at the list for the item called "debian". Remove it using the following command:
bcfg boot rm <NUM> - Where is the option number of the debian option (this will typically be 1).
- Then add back the option in order to update the boot variable. To do that, execute the following command:
bcfg boot add <NEXT_NUM> fs0:\EFI\debian\grubx64.efi "UEFI : Debian "Where <NEXT_NUM> is the next available boot option number (this will typically be 5). - Once that is done, type
resetto reboot the host.
Finish Debian Installation
Install homey things (it's not a computer without emacs:)
The first thing I do is to copy root's credentials and dot-files from a known system
Then I fix /etc/ssh/sshd_config to forbid passwords
PermitRootLogin without-password
PasswordAuthentication no
And restart the ssh daemon
service ssh restart
You can do most of this from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.
apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox rsync gcc bridge-utils vlan sudo unbound
usermod -G sudo -a randy
Get the timezone to be sensible
dpkg-reconfigure tzdata
On some systemd systems, to get unbound to start reliably, I have had to
cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service
[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535
[Install]
WantedBy=multi-user.target
EOF```
Fix hostname
echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`
Fix /etc/unbound/unbound.conf
access-control: 127.0.0.0/8 allow
access-control: 198.180.152.0/24 allow
access-control: 0.0.0.0/0 refuse
access-control: ::1 allow
access-control: ::ffff:127.0.0.1 allow
access-control: 2001:deb::/48
access-control: ::0/0 refuse
And restart unbound
service unbound restart
Clean up from CDROM sources
emacs /etc/apt/sources.list
and delete the two CDROM entries at the top.
We want to manage the server so install ipmitool and smartmontools
modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool
Set up IPMI
# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
#ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 ipaddr 192.168.0.120
ipmitool lan set 1 netmask 255.255.255.0
#ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan set 1 defgw ipaddr 192.168.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list
Set up smartmontools to watch the disks
apt install smartmontools
And check that smartd is running
root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n
I configure smartd to do short test every day and long once a week
cat >> /etc/smartd.conf << EOF
/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF
System Services
It is good to have a mail system to send logs to the sucker who watches such things. I use exim4
apt install exim4 mailtools
dpkg-reconfigure exim4-config
Use internet mail System name is vm0.sea.rg.net Only listen on local interfaces Relay mail for noone
In the wonderful world of systemd, here is how to get ntpd going
timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.
[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF
Unattended Upgrades, syslog-NG, etc.
I actually do these with ansib;e playbooks.
Install syslog-ng
apt-get install syslog-ng
Hack /etc/logrotate.d/syslog-ng
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
missingok
notifempty
compress
+ mailfirst
+ mail randy@psg.com
delaycompress
sharedscripts
postrotate
Debian Ganeti Specific Configuration
Edit /etc/hosts to have the real address of the host, e.g.
cat > /etc/hosts << EOF
127.0.0.1 localhost
147.28.0.14 vm0.sea.rg.net vm0
147.28.0.15 vm1.sea.rg.net vm1
147.28.0.16 vm2.sea.rg.net vm2
147.28.0.100 gnt0.sea.rg.net gnt0
EOF
Fix /etc/network/interfaces
Make eth0 hang off of whatever your bridge will be called
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet manual
auto br-lan
iface br-lan inet static
address 198.180.150.29/24
dns-nameservers 127.0.0.1 198.180.150.1 147.28.0.35
dns-search psg.com rg.net rpki.net
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
iface br-lan inet6 static
address 2001:418:8006::29/64
gateway 2001:418:1::126
# for drbd
auto eth1
iface eth1 inet static
address 10.0.0.100/24
Check /etc.resolv.comf
Install Ganeti
We'll need debian backports
cat >> /etc/apt/sources.d << EOF
# backports to get ganeti
deb http://http.debian.net/debian jessie-backports main
EOF
apt update
Check what version will be installed
apt-cache policy ganeti
ganeti:
Installed: (none)
Candidate: 2.12.4-1+deb8u3
Version table:
2.15.2-1~bpo8+1 0
100 http://http.debian.net/debian/ jessie-backports/main amd64 Packages
2.12.4-1+deb8u3 0
500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
500 http://security.debian.org/ jessie/updates/main amd64 Packages
Whoops! So get explicit
apt-cache policy -t jessie-backports ganeti before to check
ganeti:
Installed: (none)
Candidate: 2.15.2-1~bpo8+1
Version table:
2.15.2-1~bpo8+1 0
990 http://http.debian.net/debian/ jessie-backports/main amd64 Packages
2.12.4-1+deb8u3 0
500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
500 http://security.debian.org/ jessie/updates/main amd64 Packages
So now we can explicitly install 2.15
apt-get install -t jessie-backports ganeti
And it's a non-trivial dependency tree, to say the least
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 libcurl3
python-bitarray python-cffi-backend python-crypto python-cryptography
python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr
python-ipaddress python-openssl python-paramiko python-pkg-resources
python-psutil python-pyasn1 python-pyinotify python-pyparsing
python-setuptools python-simplejson socat
Suggested packages:
ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
python-cryptography-doc python-cryptography-vectors python-enum34-doc
python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
python-pyinotify-doc python-setuptools-doc
Recommended packages:
drbd-utils drbd8-utils qemu-kvm xen-linux-system-amd64
xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 libcurl3
python-bitarray python-cffi-backend python-crypto python-cryptography
python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr
python-ipaddress python-openssl python-paramiko python-psutil python-pyasn1
python-pyinotify python-pyparsing python-setuptools python-simplejson socat
The following packages will be upgraded:
python-pkg-resources
1 upgraded, 25 newly installed, 0 to remove and 116 not upgraded.
Need to get 16.0 MB of archives.
After this operation, 127 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Fix up drbd
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd # ignore any error
modprobe drbd
The ganati indstall seems not to always install KVM or drbd, so do it manually
apt-get install kvm drbd-utils
And you will need the noop image type support
echo "deb http://repo.noc.grnet.gr/ wheezy main" >> /etc/apt/sources.list.d/grnet.list
wget -O - http://repo.noc.grnet.gr/grnet.gpg.key|apt-key add -
apt-get update
apt-get install ganeti-os-noop
Initialize Master
So, first we're gonna need to initialize the master node
gnt-cluster init \
--vg-name=ganeti \
--master-netdev=br-lan \
--enabled-hypervisors=kvm \
-N link=br-lan \
gnt0.sjc.arrcus.com}}}
Then verify
gnt-cluster verify
If you get errors of the form
Fri Jun 1 17:21:11 2018 - WARNING: node vm0.sjc.arrcus.com: volume ganeti/root is unknown
Fri Jun 1 17:21:11 2018 - WARNING: node vm0.sjc.arrcus.com: volume ganeti/swap is unknown
Fri Jun 1 17:21:11 2018 - WARNING: node vm0.sjc.arrcus.com: volume ganeti/var is unknown
Then
gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var