Randy Bush 34386af4df add joe clarke to admins

2026-09-07 15:51:11 -07:00

15 KiB

Raw Permalink Blame History

Build a Debian 13 ProxMox Node with ZFS store

This is for a UEFI based BIOS. Be sure to enable UEFI for the hard drives and for the install boot device. Be sure to enable Virtualization in the BIOS CPU settings.

Overview

Essentially, install Debian13, Trixie

Partioning, for each drive
- 1G EFI boot partition
- 32G EXT4 root partition
- some size swap partition, opinions vary
- the rest, #4, one large unused partition, proxmox will zfs them later
If you have multiple drives, and you should, use Linux software RAID to put the root partitions in RAID1 or RAID10, depending on how many drives you have.

Then install ProxMox

Install per Install Proxmox VE on Debian 13 Trixie
Enable Free (No-Subscription) Repositories in Proxmox VE
CreateZFS and add all those unused #4 partitions

Install Debian

Before install, it is wise to disable DHCP service so one can manuall set the system IP data. Debian 9 seems reluctant to let one go into manual net configuration
Also, the BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
Also be sure virtualization is enabled for CPUs
Boot Debian CD/ISO, possibly by modifying the CMOS setup or BootMenu
Choose Install Debian Server
Choose your favorite language etc.
Select the Ethernet interface to be used
If there is no DHCP, configure Ethernet IP Address, netmask, gateway, and DNS Service
Configure hostname & domain name
Choose user name and password
Let it set some stupid timezone, you can fix later
Choose manual disk partitioning
Partition Disks
- For each drive (except the boot usb, silly boy)
  - Select it
  - Create a new empty partition table
  - Select the free space
  - create a 1G ESP partition at the beginning
  - name it /boot/efi
  - Use as: EFI System Partition
  - Be sure it is bootable
  - Done setting up the partition
  - create a 32G ext4 partition at the beginning
  - name it boot
  - Use as: /
  - Done setting up the partition
  - create a 32G swap partition at the beginning
  - name it swap
  - Use as: swap
  - Done setting up the partition
  - create a partition using the rest of the disk
  - name it zfs
  - Use as: unused
  - Done setting up the partition
- Configure Software RAID
- Write Changes and Configure RAID
- Create MD Device
- Select RAID10
- Number of Devices should be all the drives you have
- Number of Spare Drives is 0
- Select all the boot (#2) partitions (not the EFI partitions) and continue
- Write the changes and configure RAID
- Finish
- Select the RAID10 #1 partition
  - type EXT4
  - mount point /
  - Done
- Finish partitioning and write changes
Finish partitioning and write changes to disk
Debian will now install and you will do the normal mirror selection etc.
Only install SSH Server and, System Utilities
Install GRUB2. If the system is EFI, it may not even ask
Be sure it will not boot CD-ROM, and Reboot into the installed system

First Configuration

The first thing is set the password to something nasty. And remember to also set the user password.

geshyeeth3

I also have to get the network configured properly. In my example

cat > /etc/network/interfaces << EOF
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens13

iface ens13 inet static
    address 147.28.0.9/24
    gateway 147.28.0.1
    dns-nameservers 147.28.0.14
    dns-search rg.net psg.com

iface ens13 inet6 static
    address 2001:418:1::9/64
    gateway 2001:418:1::1
EOF

Then, if systemctl restart networking does not work, reboot to get the configuration.

Customize VM

If it does not run Emacs, it is not a computer. And install unbound so we have a local nameserver, and ntpd so we have a local chime. We will do detailed configuration of unbound and chrony later.

First hack so that we do not pull in unnecessary wannabe depends.

cat > /etc/apt/apt.conf.d/01norecommend << EOF
APT::Install-Recommends "0";
APT::Install-Suggests "0";
EOF

Then install needed things

apt install emacs-nox unbound chrony rsync sudo screen debian-goodies bsd-mailx gcc bridge-utils vlan

Fix sudoers

cat > /etc/sudoers.d/local << EOF
%sudo   ALL=(ALL) NOPASSWD: SETENV: ALL
EOF
/usr/sbin/usermod -a -G sudo randy

I want to disable ssh passwording as soon as reasonable, so I install root's and my ssh credentials.

cd /root
mkdir .ssh
chmod 700 .ssh
cat > .ssh/authorized_keys << EOF
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID9DzOVboz/9daQwQRPkxD/D4PZ4ZVNgMLqd+zc79Gl+ randy@ryuu.psg.com
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP2qXA3uRFGv7Q0yolQtC5bcF7kVW94c/aOhf0xbU3i8 randy@kappa.local
EOF
chmod 600 .ssh/authorized_keys
rsync -vlpPStgoHxr .ssh ~randy
chown -R randy:randy ~randy

Disallow plaintext password logins by creating /etc/ssh/sshd_config.d/local and restart sshd

cat > /etc/ssh/sshd_config.d/99sshd-local.conf << EOF
# Local modifications to /etc/ssh/sshd_config.  Not much left that we
# need to do here anymore, as upstream configuration has gradually
# drifted towards the configuration we were already using.  The two
# exceptions are PAM, which we don't need and which has a large and
# somewhat open-ended attack surface, and tunneled plaintext
# passwords, which we just don't allow at all.

#UsePAM no
PasswordAuthentication no
KbdInteractiveAuthentication no
EOF
systemctl restart ssh

I want my standard dot files, so I go to a recently customized Debian system and rsync them over.

rsync -vlpPStgoHx .bashrc .emacs .exrc .forward .inputrc randy@example.rg.net:
ssh randy@example.rg.net chown -R randy:randy .

And a bit of decoration from the other system

rsync -vlpPStgoHx /usr/bin/xtitle root@example.rg.net:/usr/bin/xtitle

Configure GRUB and the Network

Configure GRUB for

longer timeout
enable the serial console

GRUB_TIMEOUT=6
...
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
...
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

And remember to update GRUB

update-grub

You probably want to do that on all the drives. e.g.

for i in b c d; do
grub-install /dev/sd$i
done

At this point you might want to confirm that efibootmgr works.

efibootmgr

should display a bunch of text about BootOrder and a series of BootNNNN entries. If it does not, you are probably not running in EFI mode, which means this instal may be borked already but almost certainly will be if you update grub at this point, so you might want to figure that out first.

I tend to do this later, but Update /etc/network/interfaces to add two bridged LANs

# The loopback network interface
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto eno3
iface eno3 inet manual

auto eno4
iface eno4 inet manual

auto br-lan
iface br-lan inet static
        address 192.168.0.102/24
        gateway 192.168.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge_maxwait  0
        dns-nameservers 127.0.0.1 198.180.150.10
        dns-search rg.net

iface br-lan inet6 static
        address 2001:418:3807::32/64
        gateway 2001:418:3807::1

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.102/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        bridge_maxwait  0

Use Ansible to Install and Configure a Bunch of Things

Now the barrage of ansible, which for me is based on my Mac laptop.

ansible-playbook -l vm1.sea.rg.net no-depends.yml unattended-upgrades.yml chrony.yml rsyslog.yml snmpd.yml postfix.yml unbound.yml

Clean Up Unbound

Make sure that /etc/resolv.conf points to localhost.

nameserver 127.0.0.1
nameserver ::1

Then hack /etc/network/interfaces to start with the 127.0.0.1 nameserver.

For simple test that unbound is working, try

dig @127.0.0.1 psg.com. a

Add the VM to Infrastructure Monitoring Etc.

Create an entry in srv0.sea.rg.net:/etc/nagios4/conf.d and enable checks in the GUI

Add the device to snmp monitoring https://nms.rg.net/

Edit /etc/aliases to send root mail where you really want it to go, and run newaliases

Clean up little stuff

Edit /etc/hostname to fill out the full FQDN.

Set the timezone. All my servers are in UTC

dpkg-reconfigure tzdata

I still have the VNC console open, so it would be a good time to reboot and watch for problems.

Some system tools that are helpful

If you want ipmitool

apt install ipmitool

and

modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules

And configure IPMI

# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
#ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 ipaddr 192.168.0.190
ipmitool lan set 1 netmask 255.255.255.0
#ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan set 1 defgw ipaddr 192.168.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list
ipmitool lan print 1

Set up smartmontools to watch the disks

apt install smartmontools

I configure smartd to do short test every day and long once a week

cat >> /etc/smartd.conf << EOF

/dev/sda -m root -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -m root -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -m root -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -m root -o on -S on -s (S/../.././08|L/../../6/03)
EOF

And remember to comment out that first DEVICESCAN. Then

systemctl restart smartmontools.service

Debian ProxMox Specific Configuration

Follow Install Proxmox VE on Debian 13 Trixie

WARNING!!!

Before "Install the Proxmox VE Packages" check that unbound is working, i.e. do host addresses resolve.

If unbound does not work, after the mid-install reboot, it is probably fighting with apparmor. i just nuke it

aa-teardown
aa-status 
systemctl disable apparmor

and then

systemctl restart unbound.service

Edit /etc/hosts to have the real address of the host, e.g.

cat /etc/hosts
127.0.0.1       localhost
10.13.100.14    pv0.sjc.arrcus.com      pv0

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Beware that network interfaces are likely to change with ProxMox kernel. So leaving VNC/IPMI console open is wise.

You do not need to reinstall postfix or chrony

Enable Free (No-Subscription) Repositories in Proxmox VE

Install ZFS

Riffed off Debian Wiki

sudo apt update
sudo apt install linux-headers-amd64 zfsutils-linux

Dealing with a Web Proxy

If you have to traverse proxy, you can use IPTables

/sbin/iptables -F
/sbin/iptables -t nat -F
/sbin/iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8006

Or NFTables

nft add table nat
nft 'add chain nat prerouting { type nat hook prerouting priority -100; }'
nft add rule nat prerouting tcp dport 443 redirect to 8006

Log In

You should now be able to log in to the gooey as root with the host's root password. E.g. https://vm2.dfw.rg.net:8006

Cluster Issues

I found it more reliable to join/create the cluster before creating the ZFS image pools. I do not know why.

WARNING!

But only create cluster if this is the first node in the cluter.

I found this page useful when i hit

'/etc/pve/nodes/vm22/pve-ssl.pem' does not exist! (500)

When adding nodes to a cluster.

Set Up ZFS Image Pool

I have found it is less pain if the new node joins the cluster before creating the ZFS pool.

Remember, partition 4 was saved for ZFS. So, in the gooey, select the server, then ZFS, and then CreateZFS. I used RAIDz2 as a compromise. It should offer you a list of the #4 partition(s). Accept the offer. I name the pool images.

If you manually create the zfs pool a la zpool create then you will need to manually add it to the storage pool with

pvesm add zfspool images --pool images

Separate Replication LAN

If you come from Ganeti, you'll have the habit of having replication on a separate, back end, LAN. ProxMox really wants replication over the vmbr bridged LAN, which is the only LAN in the ProxMox doc. The hack I use is to let that happen, and to have a separate br-lan bridged LAN for the 'front panel payload traffic.

auto br-lan
iface br-lan inet static
        address 10.13.100.14/16
        gateway 10.13.1.1
        bridge-ports eno1np0
        bridge-stp off
        bridge-fd 0
        dns-nameservers 127.0.0.1 10.9.42.25
        dns-search      sjc.arrcus.com
        bridge_maxwait  0

auto vmbr0
iface vmbr0 inet static
        address 10.42.42.100/24
        bridge-ports eno2np1
        bridge-stp off
        bridge-fd 0
        bridge_maxwait  0

You will have to take care when creating VMs to put the VM's payload on the br-lan a la

qm create --net0 virtio,bridge=br-lan,firewall=1 ...

Replication LAN probably needs set-up

cat > /etc/pve/datacenter.cfg << EOF
crs: ha-rebalance-on-start=1,ha=static
ha: shutdown_policy=migrate
keyboard: en-us
migration: insecure,network=10.0.0.0/24
EOF

User Management

Create the admins group

pveum group add admins

Elevate the admins group

pveum acl modify / -group admins -role Administrator

Create Usrrs

pveum user add randy@pam --email "randy@pg.com" --firstname Randy --lastname Bush --groups admins
pveum user add sra@pam --email "sra@hactrn.net" --firstname Rob -lastname Austein --groups admins
pveum user add hak@pam --email "hak@flyingpoodle.com" --firstname Hans --lastname Kuhn --groups admins
pveum user add jclarke@pam --email "jclarke@cisco.com" --firstname Joe --lastname Clarke --groups admins

You will want mail to the root account to work. The long and complex story is in the fine manual. Or you can just

pveum user modify root@pam --email root@rg.net

VM Creation

ISO images tend to be in /var/lib/vz/template/iso/. You can populate them manually or use the GUI by double clicking the Server and choosing local storage and then ISO Images.

At this point, you should be able to CreateVM using an ISO from an ISO you just downloaded the library.

2026.09.07

15 KiB Raw Permalink Blame History