randy/randy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
randy/debian11-raid.md

15 KiB
Raw Permalink Blame History

ZFS Raid System on Debian 11 Bullseye

For this example, I will use the vm/host name x.rg.net. Modify to suit your needs.

Before I start the install, I put the new host in the DNS so that it has time to propagate.

Install Debian 11 on base drives

CRITICAL: Use the firmware-11.4.0-amd64-netinst.iso Debian installation ISO, not the normal debian- install ISO.

For the Dell R530, drivers for the PERC disk controller need to be set to JBOD, well non-RAID, in the iDRAC config.

The Unreadable Debian Install Screen

I use the Advanced options and Accessible high contrast install (see here for some ideas so that i can actually see the install screens instead of being blasted by electric blue and red.

Create the VM and do the Debian install

  • The BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
  • Before install, it is wise to disable DHCP service so one can manuall set the system IP data.
  • Boot Debian CD/ISO, likely by mounting as a virtual CD/ISO
  • Choose Install Debian Server
  • Choose your favorite language etc.
  • Select the Ethernet interface to be used
  • If there is no DHCP, configure Ethernet IP Address, netmask, gateway, and DNS Service
  • Configure hostname & domain name
  • Choose user name and password
  • Let it set some stupid timezone, you can fix later
  • Choose manual disk partitioning
  • Partition Disks
    • For each drive
      • Select it
      • Create a new empty partition table
      • Select the free space
      • create a 1G ESP partition at the beginning
      • name it /boot/efi
      • Use as: EFI System Partition
      • Be sure it is bootable
      • Done setting up the partition
      • Select the free space
      • create new partition using all remaining space
      • ext4
      • name it raid or whatever
      • Use as: raid Partition
      • Done setting up the partition
    • Configure Software RAID
    • Write Changes and Configure RAID
    • Create MD Device
    • Select RAID1
    • Number of Devices should be both boot drives, i.e. 2
    • Number of Spare Drives is 0
    • Select the free partitions (not the EFI partitions) and continue
    • Write the changes and configure RAID
    • Finish
  • Debian will now install and you will do the normal mirror selection etc.
  • Only install SSH Server and, System Utilities
  • I choose no automatic updates as I ansible that in later
  • Install GRUB2. If the system is EFI, it may not even ask
  • Be sure it will not boot CD-ROM, and Reboot from the installed system

Reboot the VM

You may need to disable booting from the virtual CD/ISO

Because I did not use DHCP but set the address manually, I know where the host is. I immediately ssh into the new system as the new user.

On the other hand, if the IP address was assigned by DHCP, you'll have to stay with the VNC session and set up networking as below.

First Configuration

The first thing is set the password to something nasty. And remember to also set the user password.

fouxle

I also have to get the network configured properly. In my example

cat > /etc/network/interfaces << EOF
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens13

iface ens13 inet static
    address 147.28.0.9/24
    gateway 147.28.0.1
    dns-nameservers 147.28.0.14
    dns-search rg.net psg.com

iface ens13 inet6 static
    address 2001:418:1::9/64
    gateway 2001:418:1::1
EOF

Then I reboot to get the configuration.

Customize VM

If it does not run Emacs, it is not a computer. And install unbound so we have a local nameserver, and ntpd so we have a local chime. We will do detailed configuration of unbound and chrony later.

First hack so that we do not pull in unnecessary wannabe depends.

cat > /etc/apt/apt.conf.d/01norecommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF

Then install needed things

apt install emacs-nox unbound chrony rsync sudo screen debian-goodies bsd-mailx

Hack /etc/sudoers to make it passwordless for members of the sudo group.

%sudo   ALL=(ALL) NOPASSWD: SETENV: ALL

Add the initial user to /etc/group in group sudo

/usr/sbin/usermod -a -G sudo randy

I want to disable ssh passwording as soon as reasonable, so I install root's ssh credentials.

cd /root
mkdir .ssh
chmod 700 .ssh
cat > .ssh/authorized_keys << EOF
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID9DzOVboz/9daQwQRPkxD/D4PZ4ZVNgMLqd+zc79Gl+ randy@ryuu.psg.com
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAww1WTCBezBK1snfUKi1LAwIIvQEQuhgV63toCMnTrYZfkp9/jyI1WVmKf4FIiodJWWeUtjCkt2E7t0oPZ7BxNbP8bn6HTso8qTexng8kM23UCgdzorp0ofatO0esdIdbuXd8qu9uJRlJn1TwVYeY7qY10eF60CbSZawuwSsM0iq0EbKTJQkHBtytd1zC7a7geGfieR4D3sHB9b33YhUgP9b1EO+CYtS8Qbb8xbMboD1eIB0UH0jcSUVZvb2DRtnsyIVQioW6zaAbIbFz7x+5zJBsFjkDvjR/80atOvNYJTyYZRD71D3kBu2peYup2XrsTPvrsKh5hJCVlJFCuFQaNQ== randy@psg.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8yvE/UBwHdLVDk708/U81ZQdtDLF9Y7grMkVBfUoclF4BGFCcH1bIv6y0p1I/2PATSRnK/B5ndvWP/SrQ6y0DqnQugHKoFMR3idkzVagjjXOhcZkpitcp5hRCdjn/y/88QEiIKItEM66q48nW0OtCgcq7ENXQuYUVt0e6kh6UKRDfrPP3i2D7m8o3/W+xYt8QkYZ0iFanCfhxHAP7w3SOezmThu+p7V6yxVcy+G0WTvPHrs2JlgSaI+4SoJ2btPgqIrmKu1ILfyoAr9/RjSVHr74LijsdUKec7OXnKnV2v1VfMRglBXFgXVaWyX/adA2tMuh9nMk3Y3a3+xDqPIz6RnzQAupqUmPZ/1qbFIrVSNmhygWTC9jgemaNNC/B1nFG7em6m+fmxKjauuWoIdpybaofWiJElUDBBT4bHU+L8NF7zbNROt1GtaDnbL4W9/asqzPu+aJy7L6uJQHtkc/eoXsheajztFC9I5qxW/29ja+FcQsVoDJpe96XmBtJ7elOGXaq3YtDRURwrdu9fGUBDWqwP1IFECVkeVh/NrXbQRHtgp4v533D+0MubA+hDhVTsUk6G93WLyX3usUprU1cJjyhrVzZ9U23eCPOj8J18Ixw4sJFuNeUhSCaN6j+rmOyGor3PYYysKTp1VfLNAYdcc/AYC0jDy9mYjjfCDwKfQ== randy@ryuu.psg.com
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP2qXA3uRFGv7Q0yolQtC5bcF7kVW94c/aOhf0xbU3i8 randy@kappa.local
EOF
chmod 600 .ssh/authorized_keys

Copy the ssh authority to the user account

rsync -vlpPStgoHxr .ssh ~randy
chown -R randy:randy ~randy

Disallow plaintext password logins by editing /etc/ssh/sshd_config as root

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
PasswordAuthentication no
#UsePAM yes
UsePAM no

Take the new sshd config

systemctl restart ssh

I want my standard dot files, so I go to a recently customized Xenial system and rsync them over.

rsync -vlpPStgoHx .bashrc .emacs .exrc .forward .inputrc randy@x.rg.net:
ssh randy@x.rg.net chown -R randy:randy .

And a bit of decoration from the other system

rsync -vlpPStgoHx /usr/bin/xtitle root@x.rg.net:/usr/bin/xtitle

Configure GRUB and the Network

Configure GRUB for

  • longer timeout
  • serial console
  • use ntp not kvm clock

Edit /etc/default/grub GRUB_TIMEOUT=6 ... GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

And remember to update GRUB

update-grub

Last Cosmetics Before Reboot

Check that /etc/hostname has the full FQDN.

Set the timezone. All my servers are in UTC

dpkg-reconfigure tzdata

I still have the VNC console open, so it would be a good time to reboot and watch for problems. I could tell the VM to reboot, or whack it from the Ganeti master. To ensure it is really clean, I do the latter. I do not force the timeout so I can ensure that APCI works.

gnt-instance reboot x.rg.net

Use Ansible to Install and Configure a Bunch of Things

Now the barrage of ansible, which for me is based on my Mac laptop.

ansible-playbook -l x.rg.net no-depends.yml unattended-upgrades.yml chrony.yml rsyslog.yml snmpd.yml postfix.yml unbound.yml

Clean Up Unbound

To tell systemd to keep its claws off the DNS, the following hacks seem to work as of Ubuntu 18.04 after apt-get install unbound:

sudo systemctl disable systemd-resolved.service
sudo service systemd-resolved stop

Then tell unbound to regenerate its remote control junk.

sudo rm /etc/unbound/unbound_control.* /etc/unbound/unbound_server.*
sudo unbound-control-setup

Then make sure that /etc/resolv.conf points to localhost.

nameserver 127.0.0.1
nameserver ::1

Then hack /etc/network/interfaces to start with the 127.0.0.1 nameserver.

Add the VM to Infrastructure Monitoring Etc.

Create an entry in srv0.iad.rg.net:/etc/nagios4/conf.d and enable checks in the GUI

Add the device to snmp monitoring https://nms.rg.net/

Edit /etc/aliases to send root mail where you really want it to go, and run newaliases

Install smart disk monitoring

apt install smartmontools

Add the drives to SmartMon

cat >> /etc/smartd.conf << EOF
/dev/sda -o on -S on -s (S/../.././02|L/../../0/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../3/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../6/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../9/03)
/dev/sde -o on -S on -s (S/../.././02|L/../../12/03)
/dev/sdf -o on -S on -s (S/../.././04|L/../../15/03)
/dev/sdg -o on -S on -s (S/../.././06|L/../../18/03)
/dev/sdh -o on -S on -s (S/../.././08|L/../../21/03)
EOF

And remember to comment out the "DEVICESCAN" in /etc/smartd.conf. And

systemctl restart smartd.service

Install ZFS

If there is a smart disk controller, make sure all drives are JBODs, or in DELL terms, non-raid. The latter may only be done in BIOS set-up on console; the web-based iDRAC did not work.

Recipe taken from The Deban ZFS Wiki

Be sure backports is in apt.

codename=$(lsb_release -cs);echo "deb http://deb.debian.org/debian $codename-backports main contrib non-free" | sudo tee -a /etc/apt/sources.list

sudo apt update
sudo apt install linux-headers-amd64
sudo apt install -t bullseye-backports zfsutils-linux zfs-dkms

That last, zfs-dkms, is because we configured apt not to install 'recommends'. Building the DKMS module is long and complex. And be careful not to infringe on anyone's software property rights :)

I got

DKMS: install completed.
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.10.0-14-amd64
W: Possible missing firmware /lib/firmware/tigon/tg3_tso5.bin for module tg3
W: Possible missing firmware /lib/firmware/tigon/tg3_tso.bin for module tg3
W: Possible missing firmware /lib/firmware/tigon/tg3.bin for module tg3

which seemed to do no damage.

If you do not want to reboot in order to use zfs

sudo /sbin/modprobe zfs

To cause the modules to be loaded on boot

echo zfs >> /etc/modules

Find Array Members

$ ls -l /dev/disk/by-id/ | grep sd[a-z]$
lrwxrwxrwx 1 root root  9 Jun  7 00:21 ata-ST16000NM000J-2TW103_ZR706BX0 -> ../../sdf
lrwxrwxrwx 1 root root  9 Jun  7 00:21 ata-WDC_WD161KRYZ-01AGBB0_2PH69HJJ -> ../../sdg
lrwxrwxrwx 1 root root  9 Jun  7 00:21 ata-WDC_WD161KRYZ-01AGBB0_2PH9NJNT -> ../../sde
lrwxrwxrwx 1 root root  9 Jun  7 00:21 ata-WDC_WD161KRYZ-01AGBB0_2PH9PMWT -> ../../sdc
lrwxrwxrwx 1 root root  9 Jun  7 00:21 ata-WDC_WD161KRYZ-01AGBB0_2PHA0RMT -> ../../sdd
lrwxrwxrwx 1 root root  9 Jun  7 00:21 scsi-35000c50085b7a307 -> ../../sda
lrwxrwxrwx 1 root root  9 Jun  7 00:21 scsi-35000c50085b7ab0b -> ../../sdb
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000c50085b7a307 -> ../../sda
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000c50085b7ab0b -> ../../sdb
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000c500e337c4d2 -> ../../sdf
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000cca2c1d0f565 -> ../../sdg
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000cca2c1d27bf0 -> ../../sde
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000cca2c1d28015 -> ../../sdc
lrwxrwxrwx 1 root root  9 Jun  7 00:21 wwn-0x5000cca2c1d2a5f4 -> ../../sdd

and be sure that sda and sdb are not included, as they are likely the md0 OpSys RAID1 array.

Figure Out Pre-Creation Configuration Parameters for Array

First we want to figure out the ashift zfs parameter, as ashift may only be set on pool creation. It needs to be set as 2^N of physical sector size. So i measured the drives.

# for i in 2 3 4 5 6 7; do smartctl -a -d megaraid,$i /dev/bus/0 | grep 'Sector Sizes'; done
Sector Sizes:     512 bytes logical, 4096 bytes physical
Sector Sizes:     512 bytes logical, 4096 bytes physical
Sector Sizes:     512 bytes logical, 4096 bytes physical
Sector Sizes:     512 bytes logical, 4096 bytes physical
Sector Sizes:     512 bytes logical, 4096 bytes physical

Given a 4k physical sector size, ashift should be 12 because 2^12 == 4k.

Create the Array

It is also good to read How to create RAID 10 – Striped Mirror Vdev ZPool On Ubuntu Linux for reasons explained in detail in ZFS: You should use mirror vdevs, not RAIDZ.

Jim Salter, the author of the second cite above, also wrote a great general ZFS article for Ars Technica.

Note that we had an odd number of 16TB drives. As one of us was strongly attached to the striped mirror approach, we kept the fifth as a spare.

So, create the tank array, remembering ashift and to use the extra drive as a hot spare.

sudo zpool create tank mirror wwn-0x5000cca2c1d28015 wwn-0x5000cca2c1d2a5f4 -o ashift=12
sudo zpool add tank mirror wwn-0x5000cca2c1d27bf0 wwn-0x5000c500e337c4d2
sudo zpool add tank spare wwn-0x5000cca2c1d0f565

Turn off atime as it is an unneeded load.

sudo zfs set atime=off tank

And check that it is happy

zpool status tank
  pool: tank
 state: ONLINE
config:

        tank                        ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x5000cca2c1d28015  ONLINE       0     0     0
            wwn-0x5000cca2c1d2a5f4  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            wwn-0x5000cca2c1d27bf0  ONLINE       0     0     0
            wwn-0x5000c500e337c4d2  ONLINE       0     0     0
        spares
          wwn-0x5000cca2c1d0f565    AVAIL   

errors: No known data errors

ZFS Tuning

hak has a document which recommends looking at these variables

$ zpool get ashift tank              
NAME  PROPERTY  VALUE   SOURCE
tank  ashift    12      local

hak wants compression, so

sudo zfs set compression=lz4 tank

$ zfs get all 2>&1 | egrep 'compression|atime|relatime'
tank         compression           lz4                    local
tank         atime                 off                    local
tank         relatime              off                    default

Compression and relatime settings are performance tweaks and non-controversial for 95% of use cases. They may be turned on/off after creation and are inheritable.

It is not clear what one really wants to do with these.

Set Up Application Filesystem(s)

zfs create tank/backup
zfs set mountpoint=/backup tank/backup

And then populate it

cd /backup
mkdir arrcus ietf randy zita inarra calia cryptech
chown -R backup:backup /backup

2023.09.06