randy/randy

Fork 0

Randy Bush 06aac4b029

catch up

2022-05-25 17:54:16 -07:00

12 KiB

Raw Permalink Blame History

Build a Software RAID5 Based Debian 9.0 Ganeti 2.15 Node

This is for a UEFI based BIOS. Be sure to enable UEFI for the hard drives and for the install boot device.

Install Debian

Before install, it is wise to disable DHCP service so one can manuall set the system IP data. Debian 9 seems reluctant to let one go into manual net configuration
Also, the BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
Boot Debian CD/ISO, possibly by modifying the CMOS setup
Choose Install Debian Server
Choose your favorite language etc.
Select the Ethernet interface to be used
If there is no DHCP, configure Ethernet IP Address, netmask, gateway, and DNS Service
Configure hostname & domain name
Choose user name and password
Let it set some stupid timezone, you can fix later
Choose manual disk partitioning
Partition Disks
- For each drive (except the boot usb, silly boy)
  - Select it
  - Create a new empty partition table
  - Select the free space
  - create a 512M ESP partition at the beginning
  - name it /boot/efi
  - Use as: EFI System Partition
  - Be sure it is bootable
  - Done setting up the partition
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID5 or RAID6 - I had problems with RAID10 on Ubuntu
- Number of Devices should be all the drives you have
- Number of Spare Drives is 0
- Select all the free partitions (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
Configure Logical Volume Manager
- Keep current partition layout and configure LVM
- Create volume group
  - Volume group name: ganeti
  - Devices for the new volume group: select /dev/md0
  - Continue
  - Keep current partioning layout and configure LVM
- Create Logical Volume: on ganeti, root, 16G
- Create Logical Volume: on ganeti, swap, 128G (at least RAM size)
- Create Logical Volume: on ganeti, var, 16G
- Finish
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
- Finish partitioning and write changes At this point, the partitioning looks like this
Finish partitioning and write changes to disk
Debian will now install and you will do the normal mirror selection etc.
Only install SSH Server and, System Utilities
I choose no automatic updates as I ansible that in later
Install GRUB2. If the system is EFI, it may not even ask
Be sure it will not boot CD-ROM, and Reboot from the installed system

If Debian does NOT auto-boot (thanks Joe Clark)

Once Debian is installed, boot the server into the EFI Shell. This can be done by hitting F6 during the BIOS boot (you will be prompted on screen) to enter the boot list manager. Select the shell from the boot list.
In the shell, run the following command: bcfg boot dump
Look at the list for the item called "debian". Remove it using the following command: bcfg boot rm <NUM>
Where is the option number of the debian option (this will typically be 1).
Then add back the option in order to update the boot variable. To do that, execute the following command: bcfg boot add <NEXT_NUM> fs0:\EFI\debian\grubx64.efi "UEFI : Debian " Where <NEXT_NUM> is the next available boot option number (this will typically be 5).
Once that is done, type reset to reboot the host.

Customize the Host

If it does not run Emacs, it is not a computer. And install unbound so we have a local nameserver, and ntpd so we have a local chime. We will do detailed configuration of unbound and chrony later.

apt install emacs-nox unbound chrony rsync sudo screen debian-goodies gcc bridge-utils vlan
chown -R unbound:unbound /etc/unbound/

I set root's password to something nasty so that I can get in over the serial console if things get wedged during operation.

Hack /etc/sudoers to make it passwordless for members of the sudo group.

%sudo   ALL=(ALL) NOPASSWD: SETENV: ALL

Add randy and sra to /etc/group sudo:

I want to disable ssh passwording as soon as reasonable, so I install root's ssh credentials. The ones in this example are not real.

cd /root
mkdir .ssh
chmod 700 .ssh
cat > .ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NE5AAAAIDzOVboz/9daQwQRPkxD/TD4PZ4ZVNgMLqd+zc79Gl+ randy@ryuu.psg.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8yvE/UBwHdLVDk708/U81ZQdtDLF9Y7grMkVBfUoclF4BGFCcH1bIv6y0p1I/2PATSRnK/B5ndvWP/SrQ6y0DqnQugHKoFMR3idkzVagjjXOhcZkpitcp5hRCdjn/y/88QEiIKItEM66q48nW0OtCgcq7ENXQuYUVt0e6kh6UKRDfrPP3i2D7m8o3/W+xYt8QkYZ0iFanCfhxHAP7w3SOezmThu+p7V6yxVc+G0WTvPHrs2JlgSyaI+4SoJ2btPgqIrmKu1ILfyoAr9/RjSVHr74LijsdUKec7OXnKnV2v1VfMRglBXFgXVaWyX/adA2tMuh9nMk3Y3a3+xDqPIz6RnzQAupqUmPZ/1qbFIrVSNmhygWTC9jgemaNNC/B1nFG7em6m+fmxoIdpybaofWiJElUDBBT4bHU+L8NF7zbNROt1GtaDnbL4W9/asqzPu+aJy7L6uJQHtkc/eoXsheajztFC9I5qxW/29ja+FcQsVoDJpe96XmBtJ7elOGXaq3YtDRURwrdu9fGUBDWqwP1IFECVkeVh/NrXbQRHtgp4v533D+0MubA+hDhVTsUk6G93WLyX3usUprU1cJjyhrVzZ9U23eCPOj8J18Ixw4sJFuNeUhSCaN6j+rmOyGor3PYYysKTp1VfLNAYdcc/AYC0jDy9mYjjfCDwKfQ== randy@ryuu.psg.com
chmod 600 .ssh/authorized_keys

Copy the ssh authority to the user account

rsync -vlpPStgoHxr .??* ~randy
chown -R randy:randy ~randy

Disallow plaintext password logins by editing /etc/ssh/sshd_config as root

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
PasswordAuthentication no

Take the new sshd config

service ssh restart

I want my standard dot files, so I go to a recently customized Debian system and rsync them over.

rsync -vlpPStgoHx .bashrc .emacs .exrc .forward .inputrc randy@example.rg.net:
ssh randy@example.rg.net chown -R randy:randy .

And a bit of decoration from the other system

rsync -vlpPStgoHx /usr/bin/xtitle root@example.rg.net:/usr/bin/xtitle

Undo the root login hack to /etc/ssh/sshd_config

#PermitRootLogin prohibit-password
...
PasswordAuthentication no

Take the new sshd config

service ssh restart

Configure GRUB and the Network

Configure GRUB for

longer timeout
enable the serial console

Edit /etc/default/grub

GRUB_TIMEOUT=6
...
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"

...
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

At this point you might want to confirm that efibootmgr works.

efibootmgr

should display a bunch of text about BootOrder and a series of BootNNNN entries. If it does not, you are probably not running in EFI mode, which means this instal may be borked already but almost certainly will be if you update grub at this point, so you might want to figure that out first.

And remember to update GRUB

update-grub

systemd can be really unhelpful with the console. The following incantation helps

cat > /lib/systemd/system/ttyS0.service << EOF
[Unit]
Description=Serial Console Service

[Service]
ExecStart=/sbin/getty -L 115200 ttyS0 vt102
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable ttyS0
systemctl start ttyS0

Update /etc/network/interfaces to add the bridged LAN

auto eno1
iface eno1 inet manual

auto br-lan
iface br-lan inet static
address         10.9.42.3/16
gateway         10.9.1.1
dns-nameserver  127.0.0.1
dns-search      arrcus.com sjc.arrcus.com
bridge_ports    eno1
bridge_stp      off
bridge_fd       0
bridge_maxwait  0

Last Cosmetics Before Reboot

Fix DNS if behind firewall

There is a hidden DNS service inside the firewall, which you will want unbound to know. So copy the file over and restart unbound

rsync server207:/etc/unbound/unbound.conf.d/stubs.conf /etc/unbound/unbound.conf.d/
service unbound restart

On some systemd systems, to get unbound to start reliably, I have had to

cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service

[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535

[Install]
WantedBy=multi-user.target
EOF

Clean up little stuff

Edit /etc/hostname to fill out the full FQDN.

Set the timezone. All my servers are in UTC

dpkg-reconfigure tzdata

I still have the VNC console open, so it would be a good time to reboot and watch for problems.

Some system tools that are helpful

If you want ipmitool

modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool

And configure IPMI

# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
#ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 ipaddr 192.168.0.120
ipmitool lan set 1 netmask 255.255.255.0
#ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan set 1 defgw ipaddr 192.168.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list

Set up smartmontools to watch the disks

apt install smartmontools

I configure smartd to do short test every day and long once a week

cat >> /etc/smartd.conf << EOF

/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF

System Services

It is good to have a mail system to send logs to the sucker who watches such things. I use exim4

apt install exim4 mailtools
dpkg-reconfigure exim4-config

Use internet mail
System name is vm0.sea.rg.net
Only listen on local interfaces
Relay mail for noone

Use Ansible to Install and Configure a Bunch of Things

I actually do these with ansible playbooks.

Install Unattended Upgrading

Install syslog-ng

apt-get install syslog-ng

Hack /etc/logrotate.d/syslog-ng

*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng  2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
    missingok
    notifempty
    compress
+       mailfirst
+       mail randy@psg.com
    delaycompress
    sharedscripts
    postrotate

Debian Ganeti Specific Configuration

Edit /etc/hosts to have the real address of the host, e.g.

cat > /etc/hosts << EOF
127.0.0.1       localhost
147.28.0.14     vm0.sea.rg.net vm0
147.28.0.15     vm1.sea.rg.net vm1
147.28.0.16     vm2.sea.rg.net vm2
147.28.0.100    gnt0.sea.rg.net gnt0
EOF

Install Ganeti

Ganeti comes with the standard distribution.

apt-get install ganeti ganeti-os-noop

Fix up drbd

echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd      # ignore any error
modprobe drbd

The ganati indstall seems not to always install KVM or drbd, so do it manually

apt-get install kvm drbd-utils

Initialize Master

If this is the first node in the cluster, initialize the master node

gnt-cluster init \
	--vg-name=ganeti \
	--master-netdev=br-lan \
	--enabled-hypervisors=kvm \
	-N link=br-lan \
	gnt0.sjc.arrcus.com

Otherswise, add an Nth node to the cluster. On the master,

gnt-node add vm1.sjc.arrcus.com

Then verify

gnt-cluster verify

If you get errors of the form

Fri Jun  1 17:21:11 2018   - WARNING: node vm0.sjc.arrcus.com: volume ganeti/root is unknown
Fri Jun  1 17:21:11 2018   - WARNING: node vm0.sjc.arrcus.com: volume ganeti/swap is unknown
Fri Jun  1 17:21:11 2018   - WARNING: node vm0.sjc.arrcus.com: volume ganeti/var is unknown

Then

gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var

And verify again.

2020.05.19

12 KiB Raw Permalink Blame History