randy/rgnet-wiki
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
rgnet-wiki/pages/DebianRAIDGaneti.md
Randy Bush b3b26dada6 from wiki
2020-07-04 14:52:53 -07:00

473 lines
13 KiB
Markdown
Raw Blame History

# Build a Software RAID5 Based Debian 8.0 Ganeti 2.15 Node on a UEFI BIOS
**This is for UEFI based BIOS. Be sure to enable UEFI for the hard drives and for the install boot device.**
## Install Debian
- Before install, it is wise to disable DHCP service so one can manuall set the system IP data
- Also, the BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
- Boot Debian CD/ISO
- Choose Install Debian Server
- Choose US English or it will choose the wrong apt server set
- Choose American English
- CHoose Macintosh keyboard
- Select the Ethernet interface to be used
- Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
- Configure Hostname & Domain Name
- Choose user name and password
- Let it set some stupid timezone, you can fix later
- Partition Disks
- Select the first drive
- Create a new empty partition table
- Select each of the other Drives and Create a new empty partition table
- Select the free space on each device and
- create a 256M ESP partition at the beginning
- name it /boot/efi
- Use as: EFI System Partition
- Be sure it is bootable
- Done setting up the partition
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID5 - I had problems with RAID10 on Ubuntu
- Number of Devices should be all the drives you have
- Number of Spare Drives is 0
- Select all the free partitions (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
- Configure Logical Volume Manager
- Keep current partition layout and configure LVM
- Create volume group
- Volume group name: ganeti
- Devices for the new volume group: select /dev/md0
- Continue
- Keep current partioning layout and configure LVM
- Create Logical Volume: on ganeti, root, 16G
- Create Logical Volume: on ganeti, swap, 128G (at least RAM size)
- Create Logical Volume: on ganeti, var, 16G
- Finish
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
- Finish partitioning and write changes
At this point, the partitioning looks like this
[![partitioning.jpg](https://wiki.rg.net/chrome/site/partitioning.jpg)](https://wiki.rg.net/chrome/site/partitioning.jpg)
- Finish partitioning and write changes to disk
- Debian will now install and you will do the normal mirror selection etc.
- Only install SSH Server and, System Utilities
- I choose no automatic updates as I ansible that in later
- Install GRUB2, except with EFI it eems not to ask
- Be sure it will not boot CD-ROM, and Reboot from the installed system
## If Debian does NOT auto-boot (thanks Joe Clark)
- Once Debian is installed, boot the server into the EFI Shell. This can be done by hitting F6 during the BIOS boot (you will be prompted on screen) to enter the boot list manager. Select the shell from the boot list.
- In the shell, run the following command: `bcfg boot dump`
- Look at the list for the item called "debian". Remove it using the following command: `bcfg boot rm <NUM>`
- Where <NUM> is the option number of the debian option (this will typically be 1).
- Then add back the option in order to update the boot variable. To do that, execute the following command: `bcfg boot add <NEXT_NUM> fs0:\EFI\debian\grubx64.efi "UEFI : Debian "` Where <NEXT_NUM> is the next available boot option number (this will typically be 5).
- Once that is done, type `reset` to reboot the host.
## Finish Debian Installation
Install homey things (it's not a computer without emacs:)
The first thing I do is to copy root's credentials and dot-files from a known system
Then I fix `/etc/ssh/sshd_config` to forbid passwords
```
PermitRootLogin without-password
PasswordAuthentication no
```
And restart the ssh daemon
```
service ssh restart
```
You can do most of this from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.
```
apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox rsync gcc bridge-utils vlan sudo unbound
usermod -G sudo -a randy
```
Get the timezone to be sensible
```
dpkg-reconfigure tzdata
```
On some systemd systems, to get unbound to start reliably, I have had to
```
cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service
[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535
[Install]
WantedBy=multi-user.target
EOF```
```
Fix hostname
```
echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`
```
Fix /etc/unbound/unbound.conf
```
access-control: 127.0.0.0/8 allow
access-control: 198.180.152.0/24 allow
access-control: 0.0.0.0/0 refuse
access-control: ::1 allow
access-control: ::ffff:127.0.0.1 allow
access-control: 2001:deb::/48
access-control: ::0/0 refuse
```
And restart unbound
```
service unbound restart
```
Clean up from CDROM sources
```
emacs /etc/apt/sources.list
```
and delete the two CDROM entries at the top.
We want to manage the server so install ipmitool and smartmontools
```
modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool
```
Set up IPMI
```
# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
#ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 ipaddr 192.168.0.120
ipmitool lan set 1 netmask 255.255.255.0
#ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan set 1 defgw ipaddr 192.168.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list
```
Set up smartmontools to watch the disks
```
apt install smartmontools
```
And check that smartd is running
```
root 1457 0.0 0.0 25396 4512 ? Ss 16:02 0:00 /usr/sbin/smartd -n
```
I configure smartd to do short test every day and long once a week
```
cat >> /etc/smartd.conf << EOF
/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF
```
### System Services
It is good to have a mail system to send logs to the sucker who watches such things. I use exim4
```
apt install exim4 mailtools
dpkg-reconfigure exim4-config
```
Use internet mail
System name is vm0.sea.rg.net
Only listen on local interfaces
Relay mail for noone
In the wonderful world of systemd, here is how to get ntpd going
```
timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.
[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF
```
### Unattended Upgrades, syslog-NG, etc.
I actually do these with ansib;e playbooks.
[Install Unattended Upgrading](https://wiki.rg.net/wiki/DebianAutoUpgrade)
Install syslog-ng
```
apt-get install syslog-ng
```
Hack /etc/logrotate.d/syslog-ng
```
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
missingok
notifempty
compress
+ mailfirst
+ mail randy@psg.com
delaycompress
sharedscripts
postrotate
```
## Debian Ganeti Specific Configuration
Edit /etc/hosts to have the real address of the host, e.g.
```
cat > /etc/hosts << EOF
127.0.0.1 localhost
147.28.0.14 vm0.sea.rg.net vm0
147.28.0.15 vm1.sea.rg.net vm1
147.28.0.16 vm2.sea.rg.net vm2
147.28.0.100 gnt0.sea.rg.net gnt0
EOF
```
### Fix /etc/network/interfaces
Make eth0 hang off of whatever your bridge will be called
```
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet manual
auto br-lan
iface br-lan inet static
address 198.180.150.29/24
dns-nameservers 127.0.0.1 198.180.150.1 147.28.0.35
dns-search psg.com rg.net rpki.net
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
iface br-lan inet6 static
address 2001:418:8006::29/64
gateway 2001:418:1::126
# for drbd
auto eth1
iface eth1 inet static
address 10.0.0.100/24
```
Check /etc.resolv.comf
## Install Ganeti
We'll need debian backports
```
cat >> /etc/apt/sources.d << EOF
# backports to get ganeti
deb http://http.debian.net/debian jessie-backports main
EOF
apt update
```
Check what version will be installed
```
apt-cache policy ganeti
ganeti:
Installed: (none)
Candidate: 2.12.4-1+deb8u3
Version table:
2.15.2-1~bpo8+1 0
100 http://http.debian.net/debian/ jessie-backports/main amd64 Packages
2.12.4-1+deb8u3 0
500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
500 http://security.debian.org/ jessie/updates/main amd64 Packages
```
Whoops! So get explicit
```
apt-cache policy -t jessie-backports ganeti before to check
ganeti:
Installed: (none)
Candidate: 2.15.2-1~bpo8+1
Version table:
2.15.2-1~bpo8+1 0
990 http://http.debian.net/debian/ jessie-backports/main amd64 Packages
2.12.4-1+deb8u3 0
500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
500 http://security.debian.org/ jessie/updates/main amd64 Packages
```
So now we can explicitly install 2.15
```
apt-get install -t jessie-backports ganeti
```
And it's a non-trivial dependency tree, to say the least
```
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 libcurl3
python-bitarray python-cffi-backend python-crypto python-cryptography
python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr
python-ipaddress python-openssl python-paramiko python-pkg-resources
python-psutil python-pyasn1 python-pyinotify python-pyparsing
python-setuptools python-simplejson socat
Suggested packages:
ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
python-cryptography-doc python-cryptography-vectors python-enum34-doc
python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
python-pyinotify-doc python-setuptools-doc
Recommended packages:
drbd-utils drbd8-utils qemu-kvm xen-linux-system-amd64
xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 libcurl3
python-bitarray python-cffi-backend python-crypto python-cryptography
python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr
python-ipaddress python-openssl python-paramiko python-psutil python-pyasn1
python-pyinotify python-pyparsing python-setuptools python-simplejson socat
The following packages will be upgraded:
python-pkg-resources
1 upgraded, 25 newly installed, 0 to remove and 116 not upgraded.
Need to get 16.0 MB of archives.
After this operation, 127 MB of additional disk space will be used.
Do you want to continue? [Y/n]
```
Fix up drbd
```
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd # ignore any error
modprobe drbd
```
The ganati indstall seems not to always install KVM or drbd, so do it manually
```
apt-get install kvm drbd-utils
```
And you will need the noop image type support
```
echo "deb http://repo.noc.grnet.gr/ wheezy main" >> /etc/apt/sources.list.d/grnet.list
wget -O - http://repo.noc.grnet.gr/grnet.gpg.key|apt-key add -
apt-get update
apt-get install ganeti-os-noop
```
## Initialize Master
So, first we're gonna need to initialize the master node
```
gnt-cluster init \
--vg-name=ganeti \
--master-netdev=br-lan \
--enabled-hypervisors=kvm \
-N link=br-lan \
gnt0.sjc.arrcus.com}}}
```
Then verify
```
gnt-cluster verify
```
If you get errors of the form
```
Fri Jun 1 17:21:11 2018 - WARNING: node vm0.sjc.arrcus.com: volume ganeti/root is unknown
Fri Jun 1 17:21:11 2018 - WARNING: node vm0.sjc.arrcus.com: volume ganeti/swap is unknown
Fri Jun 1 17:21:11 2018 - WARNING: node vm0.sjc.arrcus.com: volume ganeti/var is unknown
```
Then
```
gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var
```
Reference in a new issue View git blame Copy permalink