Randy Bush 09b52369b7 have smartd send email to root

2024-03-01 14:52:05 -08:00

13 KiB

Raw Permalink Blame History

Build a Software RAID10 Based Debian 11 Ganeti 3.0 Node

This is for a UEFI based BIOS. Be sure to enable UEFI for the hard drives and for the install boot device.

If you need an image which supports proprietary hardware, look in Unofficial non-free images including firmware packages

Install Debian

Before install, it is wise to disable DHCP service so one can manuall set the system IP data. Debian 9 seems reluctant to let one go into manual net configuration
Also, the BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
Also be sure virtualization is enabled for CPUs
Boot Debian CD/ISO, possibly by modifying the CMOS setup
Choose Install Debian Server
Choose your favorite language etc.
Select the Ethernet interface to be used
If there is no DHCP, configure Ethernet IP Address, netmask, gateway, and DNS Service
Configure hostname & domain name
Choose user name and password
Let it set some stupid timezone, you can fix later
Choose manual disk partitioning
Partition Disks
- For each drive (except the boot usb, silly boy)
  - Select it
  - Create a new empty partition table
  - Select the free space
  - create a 1G ESP partition at the beginning
  - name it /boot/efi
  - Use as: EFI System Partition
  - Be sure it is bootable
  - Done setting up the partition
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID10
- Number of Devices should be all the drives you have
- Number of Spare Drives is 0
- Select all the free partitions (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
Configure Logical Volume Manager
- Write the changes and configure LVM
- Create volume group
  - Volume group name: ganeti
  - Devices for the new volume group: select /dev/md0
  - Continue
  - Keep current partioning layout and configure LVM
- Write the changes and configure LVM
- Create Logical Volume: on ganeti, root, 64G
- Create Logical Volume: on ganeti, swap, 256G (at least RAM size)
- Finish
- Edit the Logical Volumes to be ext4 / and swap and root's mount point
- Finish partitioning and write changes
Finish partitioning and write changes to disk
Debian will now install and you will do the normal mirror selection etc.
Only install SSH Server and, System Utilities
I choose no automatic updates as I ansible that in later
Install GRUB2. If the system is EFI, it may not even ask
Be sure it will not boot CD-ROM, and Reboot from the installed system

If Debian does NOT auto-boot (thanks Joe Clark)

Once Debian is installed, boot the server into the EFI Shell. This can be done by hitting F6 during the BIOS boot (you will be prompted on screen) to enter the boot list manager. Select the shell from the boot list.
In the shell, run the following command: bcfg boot dump
Look at the list for the item called "debian". Remove it using the following command: bcfg boot rm <NUM>
Where is the option number of the debian option (this will typically be 1).
Then add back the option in order to update the boot variable. To do that, execute the following command: bcfg boot add <NEXT_NUM> fs0:\EFI\debian\grubx64.efi "UEFI : Debian " Where <NEXT_NUM> is the next available boot option number (this will typically be 5).
Once that is done, type reset to reboot the host.

First Configuration

The first thing is set the password to something nasty. And remember to also set the user password.

I also have to get the network configured properly. In my example

cat > /etc/network/interfaces << EOF
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens13

iface ens13 inet static
    address 147.28.0.9/24
    gateway 147.28.0.1
    dns-nameservers 147.28.0.14
    dns-search rg.net psg.com

iface ens13 inet6 static
    address 2001:418:1::9/64
    gateway 2001:418:1::1
EOF

Then, if systemctl restart networking does not work, reboot to get the configuration.

Customize VM

If it does not run Emacs, it is not a computer. And install unbound so we have a local nameserver, and ntpd so we have a local chime. We will do detailed configuration of unbound and chrony later.

First hack so that we do not pull in unnecessary wannabe depends.

cat > /etc/apt/apt.conf.d/01norecommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF

Then install needed things

apt install emacs-nox unbound chrony rsync sudo screen debian-goodies bsd-mailx gcc bridge-utils vlan

Hack /etc/sudoers to make it passwordless for members of the sudo group.

%sudo   ALL=(ALL) NOPASSWD: SETENV: ALL

Add the initial user to /etc/group in group sudo

/usr/sbin/usermod -a -G sudo randy

I want to disable ssh passwording as soon as reasonable, so I install root's ssh credentials.

cd /root
mkdir .ssh
chmod 700 .ssh
cat > .ssh/authorized_keys << EOF
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID9DzOVboz/9daQwQRPkxD/D4PZ4ZVNgMLqd+zc79Gl+ randy@ryuu.psg.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8yvE/UBwHdLVDk708/U81ZQdtDLF9Y7grMkVBfUoclF4BGFCcH1bIv6y0p1I/2PATSRnK/B5ndvWP/SrQ6y0DqnQugHKoFMR3idkzVagjjXOhcZkpitcp5hRCdjn/y/88QEiIKItEM66q48nW0OtCgcq7ENXQuYUVt0e6kh6UKRDfrPP3i2D7m8o3/W+xYt8QkYZ0iFanCfhxHAP7w3SOezmThu+p7V6yxVcy+G0WTvPHrs2JlgSaI+4SoJ2btPgqIrmKu1ILfyoAr9/RjSVHr74LijsdUKec7OXnKnV2v1VfMRglBXFgXVaWyX/adA2tMuh9nMk3Y3a3+xDqPIz6RnzQAupqUmPZ/1qbFIrVSNmhygWTC9jgemaNNC/B1nFG7em6m+fmxKjauuWoIdpybaofWiJElUDBBT4bHU+L8NF7zbNROt1GtaDnbL4W9/asqzPu+aJy7L6uJQHtkc/eoXsheajztFC9I5qxW/29ja+FcQsVoDJpe96XmBtJ7elOGXaq3YtDRURwrdu9fGUBDWqwP1IFECVkeVh/NrXbQRHtgp4v533D+0MubA+hDhVTsUk6G93WLyX3usUprU1cJjyhrVzZ9U23eCPOj8J18Ixw4sJFuNeUhSCaN6j+rmOyGor3PYYysKTp1VfLNAYdcc/AYC0jDy9mYjjfCDwKfQ== randy@ryuu.psg.com
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP2qXA3uRFGv7Q0yolQtC5bcF7kVW94c/aOhf0xbU3i8 randy@kappa.local
EOF
chmod 600 .ssh/authorized_keys

Copy the ssh authority to the user account

rsync -vlpPStgoHxr .ssh ~randy
chown -R randy:randy ~randy

Disallow plaintext password logins by editing /etc/ssh/sshd_config as root

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
PasswordAuthentication no

#UsePAM yes
UsePAM no

Take the new sshd config

systemctl restart ssh

I want my standard dot files, so I go to a recently customized Debian system and rsync them over.

rsync -vlpPStgoHx .bashrc .emacs .exrc .forward .inputrc randy@example.rg.net:
ssh randy@example.rg.net chown -R randy:randy .

And a bit of decoration from the other system

rsync -vlpPStgoHx /usr/bin/xtitle root@example.rg.net:/usr/bin/xtitle

Configure GRUB and the Network

Configure GRUB for

longer timeout
enable the serial console

Edit /etc/default/grub

GRUB_TIMEOUT=6
...
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"

...
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

And remember to update GRUB

update-grub

At this point you might want to confirm that efibootmgr works.

efibootmgr

should display a bunch of text about BootOrder and a series of BootNNNN entries. If it does not, you are probably not running in EFI mode, which means this instal may be borked already but almost certainly will be if you update grub at this point, so you might want to figure that out first.

Update /etc/network/interfaces to add the bridged LAN

auto eno1
iface eno1 inet manual

auto br-lan
iface br-lan inet static
        address         147.28.0.14/24
        gateway         147.28.0.1
        dns-nameserver  127.0.0.1
        dns-search      rg.net
        bridge_ports    eno1
        bridge_stp      off
        bridge_fd       0
        bridge_maxwait  0

iface eno1 inet6 static
        address         2001:418:1::14
        gateway         2001:418:1::1

auto eno2
iface eno2 inet static
        address         10.0.0.10x

Use Ansible to Install and Configure a Bunch of Things

Now the barrage of ansible, which for me is based on my Mac laptop.

ansible-playbook -l vm4.sea.rg.net no-depends.yml unattended-upgrades.yml chrony.yml rsyslog.yml snmpd.yml postfix.yml unbound.yml

Clean Up Unbound

To tell systemd to keep its claws off the DNS, the following hacks seem to work as of Ubuntu 18.04 after apt-get install unbound:

sudo systemctl disable systemd-resolved.service
sudo service systemd-resolved stop

Then tell unbound to regenerate its remote control junk.

sudo rm /etc/unbound/unbound_control.* /etc/unbound/unbound_server.*
sudo unbound-control-setup

Then make sure that /etc/resolv.conf points to localhost.

nameserver 127.0.0.1
nameserver ::1

Then hack /etc/network/interfaces to start with the 127.0.0.1 nameserver.

Add the VM to Infrastructure Monitoring Etc.

Create an entry in srv0.sea.rg.net:/etc/nagios4/conf.d and enable checks in the GUI

Add the device to snmp monitoring https://nms.rg.net/

Edit /etc/aliases to send root mail where you really want it to go, and run newaliases

Clean up little stuff

Edit /etc/hostname to fill out the full FQDN.

Set the timezone. All my servers are in UTC

dpkg-reconfigure tzdata

I still have the VNC console open, so it would be a good time to reboot and watch for problems.

Some system tools that are helpful

If you want ipmitool

apt install ipmitool

and

modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules

And configure IPMI

# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
#ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 ipaddr 192.168.0.190
ipmitool lan set 1 netmask 255.255.255.0
#ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan set 1 defgw ipaddr 192.168.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list

Set up smartmontools to watch the disks

apt install smartmontools

I configure smartd to do short test every day and long once a week

cat >> /etc/smartd.conf << EOF

/dev/sda -m root -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -m root -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -m root -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -m root -o on -S on -s (S/../.././08|L/../../6/03)
EOF

And remember to comment out that first DEVICESCAN. Then

systemctl restart smartmontools.service

Debian Ganeti Specific Configuration

Edit /etc/hosts to have the real address of the host, e.g.

cat > /etc/hosts << EOF
127.0.0.1       localhost
147.28.0.14     vm0.sea.rg.net vm0
147.28.0.15     vm1.sea.rg.net vm1
147.28.0.16     vm2.sea.rg.net vm2
147.28.0.29     vm3.sea.rg.net vm3
147.28.0.9      vm4.sea.rg.net vm4
147.28.0.46     vm5.sea.rg.net vm5
147.28.0.100 	gnt0.sea.rg.net gnt0
EOF

Install Ganeti

Ganeti comes with the standard distribution.

apt install ganeti ganeti-os-noop drbd-utils qemu-kvm qemu-utils

And make drbd happy

echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd      # ignore any error
modprobe drbd

Initialize Master

Before you enable secondary-ips, be sure you can ping between them,

If this is the first node in the cluster, initialize the master node

gnt-cluster init \
    --vg-name=ganeti \
    --master-netdev=br-lan \
    --enabled-hypervisors=kvm \
    --secondary-ip=10.0.0.10x \
    -H kvm:vnc_password_file=/etc/ganeti/vnc-cluster-password \
    -N link=br-lan \
    gnt0.sea.rg.net

Otherwise, add an Nth node to the cluster. On the master,

gnt-node add --secondary-ip=10.0.0.10x vm1.sjc.arrcus.com

Then verify

gnt-cluster verify

If you get errors of the form

Parameter 'kernel_path' fails validation: not found or not a file (current value: '/boot/vmlinuz-3-kvmU')

then

gnt-cluster modify --hypervisor-parameters kvm:kernel_path=

If you get whining of the form

WARNING: node vm4.sea.rg.net: volume ganeti/root is unknown
WARNING: node vm4.sea.rg.net: volume ganeti/swap is unknown

Then

gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap

And verify again.

Add the VNC password

echo "YouReallyShouldNotUseThisOne" > /etc/ganeti/vnc-cluster-password
chmod 600 /etc/ganeti/vnc-cluster-password
chown root:root /etc/ganeti/vnc-cluster-password

2023.11.23

13 KiB Raw Permalink Blame History