# Build a Debian/Ganeti System and Convert libvirt Images ## Initialize Hardware Boot into MegaRaid BIOS and configure the four drives as one RAID5 for the 1RU and RAID6 for the 2RUs. ### Install Debian - Boot Debian CD/ISO - Choose Install - Choose English, UK (so you can get UCT) - Choose American English - Name the host - Choose root password - Choose user name and password - Partition - Choose Manual Partitioning - Select the drive - Create new empty partition table - Select Free Space - Create new partition, primary, 1GB, begining, bios - Done - Select Free Space - Create new partition, primary, 256MB, begining, /boot, ext4, bootable - Done - Select Free Space again - Create a new partition - Accept whatever size is shown (the rest of the disk) - Primary, physical volume for LVM - Done - Configure LVM - Configure LVM accepting write changes to disks - Create volume group - Volume group name: ganeti - Devices for the new volume group: select only the LVM partition - Create Logical Volume: on ganeti, root, 16G - Create Logical Volume: on ganeti, swap, 16G - Create Logical Volume: on ganeti, var, 16G - Edit the Logical Volumes to be ext4 /, swap, and ext4 /var - Finish partitioning and write changes - Finish partitioning and write changes to disk - Be sure it will not boot CD-ROM, and Reboot from the installed system ## Finish Debian Installation Clean up from CDROM sources ``` vi /etc/apt/sources.list ``` and delete the two CDROM entries at the top Install homey things (it's not a computer without emacs:) ``` apt-get update apt-get upgrade apt-get install emacs23-nox apt-get install rsync apt-get install gcc apt-get install bridge-utils vlan apt-get install sudo apt-get install unbound usermod -G sudo -a randy ``` Fix hostname ``` echo vm0.bknix.co.th > /etc/hostname hostname `cat /etc/hostname` ``` Fix /etc/unbound/unbound.conf ``` access-control: 127.0.0.0/8 allow access-control: 203.159.68.0/22 allow access-control: 0.0.0.0/0 refuse access-control: ::1 allow access-control: ::ffff:127.0.0.1 allow access-control: 2001:deb::/48 access-control: ::0/0 refuse ``` ### Make ISO FileSystem ``` lvcreate -n ISOs -L 128G ganeti mkdir /ISOs mkfs /dev/mapper/ganeti-ISOs ``` and then add it to /etc/fstab ``` /dev/mapper/ganeti-ISOs /ISOs ext4 defaults 0 2 ``` and then you can mount it ``` mount /ISOs ``` ### Unattended Upgrades, syslog-NG, etc. [Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade) Install syslog-ng ``` apt-get install syslog-ng ``` Hack /etc/logrotate.d/syslog-ng ``` *** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000 --- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100 *************** *** 29,34 **** --- 29,36 ---- missingok notifempty compress + mailfirst + mail randy@psg.com delaycompress sharedscripts postrotate ``` Silence log spew to console tty by hacking /etc/syslog-ng/syslog-ng.conf ``` # log { source(s_src); filter(f_console); destination(d_console_all); # destination(d_xconsole); }; # log { source(s_src); filter(f_crit); destination(d_console); }; ``` Fix Exim ``` dpkg-reconfigure exim4-config ``` Use internet mail System name is vm0.bknix.co.th Only listen on local interfaces Relay mail for noone ## Debian Ganeti Specific Configuration Edit /etc/hosts to have the real address of the host, e.g. ``` 127.0.0.1 localhost 198.180.150.8 vm0.iad.rg.net vm0 198.180.150.4 vm1.iad.rg.net vm1 198.180.150.100 gnt0.iad.rg.net gnt0 ``` ### Fix /etc/network/interfaces Make eth0 hang off of whatever your bridge will be called ``` # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet manual auto br-lan iface br-lan inet static address 198.180.150.4 netmask 255.255.255.128 gateway 198.180.150.126 dns-nameservers 127.0.0.1 198.180.150.1 147.28.0.35 dns-search psg.com rg.net rpki.net bridge_ports eth0 bridge_stp off bridge_fd 0 bridge_maxwait 0 iface br-lan inet6 static address 2001:418:8006::4 netmask 64 gateway 2001:418:1::126 auto eth0.255 iface eth0.255 inet manual auto br-svc iface br-svc inet manual bridge_ports eth0.255 bridge_stp off bridge_fd 0 bridge_maxwait 0 auto eth1 iface eth1 inet manual auto br-hack iface br-hack inet static address 10.0.0.101 netmask 255.255.255.0 bridge_ports eth1 bridge_stp off bridge_fd 0 bridge_maxwait 0 ``` Check /etc.resolv.comf In theory, this looks like ``` -------------+-------------- | br-lan | this host +---------+---------+ | eth0 | | | |eth0.255 eth0.101| +--------+----------+ | br-hack | +------> to other ganeti hosts ``` ## Install Ganeti Set up to get Ganeti from backports on all three servers, vm0, vm1, and vm2 ``` cat >> /etc/apt/sources.list.d/wheezy-backports.list deb http://cdn.debian.net/debian/ wheezy-backports main ``` And then install it on all three servers ``` apt-get update apt-get install ganeti/wheezy-backports ``` Fix up drbd ``` echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf rmmod drbd # ignore any error modprobe drbd ``` ### Initialize and Build Ganeti Cluster ``` gnt-cluster init \ --master-netdev=br-internal \ --enabled-hypervisors=kvm \ -H kvm:kernel_path="",initrd_path="" \ --vg-name=ganeti \ -N link=br-cluster \ -s 10.0.0.101 \ gnt0.iad.rg.net ``` If it barfs, try ``` gnt-cluster destroy --yes-do-it ``` then re-run gnt-cluster init ... If you get into gnutls problems check apt-get upgrade and dist-upgrade ### Check That Cluster is Happy Fix VNC passwording ``` echo 'fnorgille' > /etc/ganeti/vnc-cluster-password gnt-cluster modify -H kvm:vnc_password_file=/etc/ganeti/vnc-cluster-password ``` Test that the cluster was built happily ``` gnt-cluster verify ``` If you get ``` Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/root is unknown Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/swap is unknown Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/var is unknown ``` That's OK. To make it go away ``` gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var ``` Now add vm1 and vm2 to the cluster by the following on vm0 ``` gnt-node add -s 10.0.1.11 vm1.bknix.co.th gnt-node add -s 10.0.1.12 vm2.bknix.co.th ``` which should result in ``` vm0.bknix.co.th:/root# gnt-node add -s 10.0.1.11 vm1.bknix.co.th -- WARNING -- Performing this operation is going to replace the ssh daemon keypair on the target machine (vm1.bknix.co.th) with the ones of the current one and grant full intra-cluster ssh root access to/from it The authenticity of host 'vm1.bknix.co.th (203.159.71.11)' can't be established. ECDSA key fingerprint is 2f:48:91:f7:f2:dd:c4:95:da:30:d2:f4:92:6a:53:ec. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'vm1.bknix.co.th' (ECDSA) to the list of known hosts. Restarting OpenBSD Secure Shell server: sshd. Fri Aug 22 05:37:53 2014 - INFO: Node will be a master candidate ``` ## Convert a libvirt qcow2 to a ganeti instance get noop os type, v4 only! ``` echo "deb http://repo.noc.grnet.gr/ wheezy main" >> /etc/apt/sources.list.d/grnet.list wget -O - http://repo.noc.grnet.gr/grnet.gpg.key|apt-key add - apt-get update apt-get install ganeti-os-noop ``` Find the disk size the qcow2 will require ``` qemu-img info .qcow2 ``` Create the empty Ganeti instance ``` ./do-add 32 4 ``` where the do-add script is ``` #!/bin/sh # makeVM diskGB ramGB nameFQDN DISK=$1 RAM=$2 NAME=$3 NODE=vm1.iad.rg.net gnt-instance add \ -t raw \ -o noop \ -s ${DISK}G \ -B maxmem=${RAM}G,minmem=$((${RAM}/2))G \ -n $NODE \ -H kvm:vnc_bind_address=0.0.0.0 \ --no-install \ --no-start \ --no-ip-check \ --no-name-check \ ${NAME} ``` then ### Hint: Copying Images Between Systems rsync and so forth spend a lot of time in crypto etc. If you are copying an image across a local LAN, `netcat` is your friend. On the source ``` nc -l -p 42420 < path-to/name-of.qcow2 ``` On the destination ``` nc -w3 source.fqdn 42420 > name-of.qcow2 ``` Of course, do an md5 or sha compare of the source and destination files before using them. ### Actually Convert qcow2 to a Ganeti Image ``` gnt-instance activate-disks ``` and you will get the disk ID of the instance ``` vm1.iad.rg.net:disk/0:/dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0 ``` convert the qcow2 directly to the instance image ``` qemu-img convert .qcow2 /dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0 ``` If qemu-convert wants more target disk space then you thought you had in the source disk, you will get ``` qemu-img: /dev/ganeti/c316d162-af4d-4ce0-8d1f-7b0b01919053.disk0: error while converting raw: No space left on device ``` The only solution seems to be to allocate more disk in the do-add Now you can start the converted image ``` gnt-instance start ``` ### Moving Secondary Disk Drives from libvert to a Ganeti Instance First, create the empty disk on the Ganeti image ``` gnt-instance modify --disk add:size=1024G ``` Then find out the magic filename of the disk ``` gnt-instance activate-disks vm1.iad.rg.net:disk/0:/dev/ganeti/15df702f-3535-4a99-90e1-42d5ae00f469.disk0 vm1.iad.rg.net:disk/1:/dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1 ``` disk0 is the base Ganeti image disk, disk1 has been added. The libvert secondary disk is a raw image, not a qcow. So you do a move directly into place. On the libvert side ``` nc -l -p 42420 < path-to-disk.sdb ``` On the Ganeti side ``` nc -w3 vm0.iad 42420 > /dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1 ``` When it is finished, you need to ``` gnt-instance restart ```