diff --git a/pages/BuildAndConvertLibirt.md b/pages/BuildAndConvertLibirt.md new file mode 100644 index 0000000..2f0d329 --- /dev/null +++ b/pages/BuildAndConvertLibirt.md @@ -0,0 +1,478 @@ +# Build a Debian/Ganeti System and Convert libvirt Images + +## Initialize Hardware + +Boot into MegaRaid BIOS and configure the four drives as one RAID5 for the 1RU and RAID6 for the 2RUs. + +### Install Debian + +- Boot Debian CD/ISO +- Choose Install +- Choose English, UK (so you can get UCT) +- Choose American English +- Name the host +- Choose root password +- Choose user name and password +- Partition + - Choose Manual Partitioning + - Select the drive + - Create new empty partition table + - Select Free Space + - Create new partition, primary, 1GB, begining, bios + - Done + - Select Free Space + - Create new partition, primary, 256MB, begining, /boot, ext4, bootable + - Done + - Select Free Space again + - Create a new partition + - Accept whatever size is shown (the rest of the disk) + - Primary, physical volume for LVM + - Done +- Configure LVM + - Configure LVM accepting write changes to disks + - Create volume group + - Volume group name: ganeti + - Devices for the new volume group: select only the LVM partition + - Create Logical Volume: on ganeti, root, 16G + - Create Logical Volume: on ganeti, swap, 16G + - Create Logical Volume: on ganeti, var, 16G + - Edit the Logical Volumes to be ext4 /, swap, and ext4 /var + - Finish partitioning and write changes +- Finish partitioning and write changes to disk + +- Be sure it will not boot CD-ROM, and Reboot from the installed system + +## Finish Debian Installation + +Clean up from CDROM sources + +``` +vi /etc/apt/sources.list +``` + +and delete the two CDROM entries at the top + +Install homey things (it's not a computer without emacs:) + +``` +apt-get update +apt-get upgrade +apt-get install emacs23-nox +apt-get install rsync +apt-get install gcc +apt-get install bridge-utils vlan +apt-get install sudo +apt-get install unbound +usermod -G sudo -a randy +``` + +Fix hostname + +``` +echo vm0.bknix.co.th > /etc/hostname +hostname `cat /etc/hostname` +``` + +Fix /etc/unbound/unbound.conf + +``` + access-control: 127.0.0.0/8 allow + access-control: 203.159.68.0/22 allow + access-control: 0.0.0.0/0 refuse + access-control: ::1 allow + access-control: ::ffff:127.0.0.1 allow + access-control: 2001:deb::/48 + access-control: ::0/0 refuse +``` + +### Make ISO FileSystem + +``` +lvcreate -n ISOs -L 128G ganeti +mkdir /ISOs +mkfs /dev/mapper/ganeti-ISOs +``` + +and then add it to /etc/fstab + +``` +/dev/mapper/ganeti-ISOs /ISOs ext4 defaults 0 2 +``` + +and then you can mount it + +``` +mount /ISOs +``` + +### Unattended Upgrades, syslog-NG, etc. + +[Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade) + +Install syslog-ng + +``` +apt-get install syslog-ng +``` + +Hack /etc/logrotate.d/syslog-ng + +``` +*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000 +--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100 +*************** +*** 29,34 **** +--- 29,36 ---- + missingok + notifempty + compress ++ mailfirst ++ mail randy@psg.com + delaycompress + sharedscripts + postrotate +``` + +Silence log spew to console tty by hacking /etc/syslog-ng/syslog-ng.conf + +``` +# log { source(s_src); filter(f_console); destination(d_console_all); +# destination(d_xconsole); }; +# log { source(s_src); filter(f_crit); destination(d_console); }; +``` + +Fix Exim + +``` +dpkg-reconfigure exim4-config +``` + +Use internet mail + System name is vm0.bknix.co.th + Only listen on local interfaces + Relay mail for noone + +## Debian Ganeti Specific Configuration + +Edit /etc/hosts to have the real address of the host, e.g. + +``` +127.0.0.1 localhost +198.180.150.8 vm0.iad.rg.net vm0 +198.180.150.4 vm1.iad.rg.net vm1 +198.180.150.100 gnt0.iad.rg.net gnt0 +``` + +### Fix /etc/network/interfaces + +Make eth0 hang off of whatever your bridge will be called + +``` +# This file describes the network interfaces available on your system +# and how to activate them. For more information, see interfaces(5). + +# The loopback network interface +auto lo +iface lo inet loopback + +# The primary network interface +auto eth0 +iface eth0 inet manual + +auto br-lan +iface br-lan inet static + address 198.180.150.4 + netmask 255.255.255.128 + gateway 198.180.150.126 + dns-nameservers 127.0.0.1 198.180.150.1 147.28.0.35 + dns-search psg.com rg.net rpki.net + bridge_ports eth0 + bridge_stp off + bridge_fd 0 + bridge_maxwait 0 + +iface br-lan inet6 static + address 2001:418:8006::4 + netmask 64 + gateway 2001:418:1::126 + +auto eth0.255 +iface eth0.255 inet manual + +auto br-svc +iface br-svc inet manual + bridge_ports eth0.255 + bridge_stp off + bridge_fd 0 + bridge_maxwait 0 + +auto eth1 +iface eth1 inet manual + +auto br-hack +iface br-hack inet static + address 10.0.0.101 + netmask 255.255.255.0 + bridge_ports eth1 + bridge_stp off + bridge_fd 0 + bridge_maxwait 0 +``` + +Check /etc.resolv.comf + +In theory, this looks like + +``` + -------------+-------------- + | + br-lan + | this host + +---------+---------+ + | eth0 | + | | + |eth0.255 eth0.101| + +--------+----------+ + | + br-hack + | + +------> to other ganeti hosts +``` + +## Install Ganeti + +Set up to get Ganeti from backports on all three servers, vm0, vm1, and vm2 + +``` +cat >> /etc/apt/sources.list.d/wheezy-backports.list +deb http://cdn.debian.net/debian/ wheezy-backports main +``` + +And then install it on all three servers + +``` +apt-get update +apt-get install ganeti/wheezy-backports +``` + +Fix up drbd + +``` +echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf +rmmod drbd # ignore any error +modprobe drbd +``` + +### Initialize and Build Ganeti Cluster + +``` +gnt-cluster init \ + --master-netdev=br-internal \ + --enabled-hypervisors=kvm \ + -H kvm:kernel_path="",initrd_path="" \ + --vg-name=ganeti \ + -N link=br-cluster \ + -s 10.0.0.101 \ + gnt0.iad.rg.net +``` + +If it barfs, try + +``` +gnt-cluster destroy --yes-do-it +``` + +then re-run gnt-cluster init ... + +If you get into gnutls problems check apt-get upgrade and dist-upgrade + +### Check That Cluster is Happy + +Fix VNC passwording + +``` +echo 'fnorgille' > /etc/ganeti/vnc-cluster-password +gnt-cluster modify -H kvm:vnc_password_file=/etc/ganeti/vnc-cluster-password +``` + +Test that the cluster was built happily + +``` +gnt-cluster verify +``` + +If you get + +``` +Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/root is unknown +Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/swap is unknown +Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/var is unknown +``` + +That's OK. To make it go away + +``` +gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var +``` + +Now add vm1 and vm2 to the cluster by the following on vm0 + +``` +gnt-node add -s 10.0.1.11 vm1.bknix.co.th +gnt-node add -s 10.0.1.12 vm2.bknix.co.th +``` + +which should result in + +``` +vm0.bknix.co.th:/root# gnt-node add -s 10.0.1.11 vm1.bknix.co.th +-- WARNING -- +Performing this operation is going to replace the ssh daemon keypair +on the target machine (vm1.bknix.co.th) with the ones of the current one +and grant full intra-cluster ssh root access to/from it + +The authenticity of host 'vm1.bknix.co.th (203.159.71.11)' can't be established. +ECDSA key fingerprint is 2f:48:91:f7:f2:dd:c4:95:da:30:d2:f4:92:6a:53:ec. +Are you sure you want to continue connecting (yes/no)? yes +Warning: Permanently added 'vm1.bknix.co.th' (ECDSA) to the list of known hosts. +Restarting OpenBSD Secure Shell server: sshd. +Fri Aug 22 05:37:53 2014 - INFO: Node will be a master candidate +``` + +## Convert a libvirt qcow2 to a ganeti instance + +get noop os type, v4 only! + +``` +echo "deb http://repo.noc.grnet.gr/ wheezy main" >> /etc/apt/sources.list.d/grnet.list +wget -O - http://repo.noc.grnet.gr/grnet.gpg.key|apt-key add - +apt-get update +apt-get install ganeti-os-noop +``` + +Find the disk size the qcow2 will require + +``` +qemu-img info .qcow2 +``` + +Create the empty Ganeti instance + +``` +./do-add 32 4 +``` + +where the do-add script is + +``` +#!/bin/sh + +# makeVM diskGB ramGB nameFQDN + +DISK=$1 +RAM=$2 +NAME=$3 +NODE=vm1.iad.rg.net + +gnt-instance add \ + -t raw \ + -o noop \ + -s ${DISK}G \ + -B maxmem=${RAM}G,minmem=$((${RAM}/2))G \ + -n $NODE \ + -H kvm:vnc_bind_address=0.0.0.0 \ + --no-install \ + --no-start \ + --no-ip-check \ + --no-name-check \ + ${NAME} +``` + +then + +### Hint: Copying Images Between Systems + +rsync and so forth spend a lot of time in crypto etc. If you are copying an image across a local LAN, `netcat` is your friend. + On the source + +``` +nc -l -p 42420 < path-to/name-of.qcow2 +``` + +On the destination + +``` +nc -w3 source.fqdn 42420 > name-of.qcow2 +``` + +Of course, do an md5 or sha compare of the source and destination files before using them. + +### Actually Convert qcow2 to a Ganeti Image + +``` +gnt-instance activate-disks +``` + +and you will get the disk ID of the instance + +``` +vm1.iad.rg.net:disk/0:/dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0 +``` + +convert the qcow2 directly to the instance image + +``` +qemu-img convert .qcow2 /dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0 +``` + +If qemu-convert wants more target disk space then you thought you had in the source disk, you will get + +``` +qemu-img: /dev/ganeti/c316d162-af4d-4ce0-8d1f-7b0b01919053.disk0: error while converting raw: No space left on device +``` + +The only solution seems to be to allocate more disk in the do-add + +Now you can start the converted image + +``` +gnt-instance start +``` + +### Moving Secondary Disk Drives from libvert to a Ganeti Instance + +First, create the empty disk on the Ganeti image + +``` +gnt-instance modify --disk add:size=1024G +``` + +Then find out the magic filename of the disk + +``` +gnt-instance activate-disks +vm1.iad.rg.net:disk/0:/dev/ganeti/15df702f-3535-4a99-90e1-42d5ae00f469.disk0 +vm1.iad.rg.net:disk/1:/dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1 +``` + +disk0 is the base Ganeti image disk, disk1 has been added. + + The libvert secondary disk is a raw image, not a qcow. So you do a move directly into place. + + On the libvert side + +``` +nc -l -p 42420 < path-to-disk.sdb +``` + +On the Ganeti side + +``` +nc -w3 vm0.iad 42420 > /dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1 +``` + +When it is finished, you need to + +``` +gnt-instance restart +``` \ No newline at end of file