478 lines
No EOL
10 KiB
Markdown
478 lines
No EOL
10 KiB
Markdown
# Build a Debian/Ganeti System and Convert libvirt Images
|
|
|
|
## Initialize Hardware
|
|
|
|
Boot into MegaRaid BIOS and configure the four drives as one RAID5 for the 1RU and RAID6 for the 2RUs.
|
|
|
|
### Install Debian
|
|
|
|
- Boot Debian CD/ISO
|
|
- Choose Install
|
|
- Choose English, UK (so you can get UCT)
|
|
- Choose American English
|
|
- Name the host
|
|
- Choose root password
|
|
- Choose user name and password
|
|
- Partition
|
|
- Choose Manual Partitioning
|
|
- Select the drive
|
|
- Create new empty partition table
|
|
- Select Free Space
|
|
- Create new partition, primary, 1GB, begining, bios
|
|
- Done
|
|
- Select Free Space
|
|
- Create new partition, primary, 256MB, begining, /boot, ext4, bootable
|
|
- Done
|
|
- Select Free Space again
|
|
- Create a new partition
|
|
- Accept whatever size is shown (the rest of the disk)
|
|
- Primary, physical volume for LVM
|
|
- Done
|
|
- Configure LVM
|
|
- Configure LVM accepting write changes to disks
|
|
- Create volume group
|
|
- Volume group name: ganeti
|
|
- Devices for the new volume group: select only the LVM partition
|
|
- Create Logical Volume: on ganeti, root, 16G
|
|
- Create Logical Volume: on ganeti, swap, 16G
|
|
- Create Logical Volume: on ganeti, var, 16G
|
|
- Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
|
|
- Finish partitioning and write changes
|
|
- Finish partitioning and write changes to disk
|
|
|
|
- Be sure it will not boot CD-ROM, and Reboot from the installed system
|
|
|
|
## Finish Debian Installation
|
|
|
|
Clean up from CDROM sources
|
|
|
|
```
|
|
vi /etc/apt/sources.list
|
|
```
|
|
|
|
and delete the two CDROM entries at the top
|
|
|
|
Install homey things (it's not a computer without emacs:)
|
|
|
|
```
|
|
apt-get update
|
|
apt-get upgrade
|
|
apt-get install emacs23-nox
|
|
apt-get install rsync
|
|
apt-get install gcc
|
|
apt-get install bridge-utils vlan
|
|
apt-get install sudo
|
|
apt-get install unbound
|
|
usermod -G sudo -a randy
|
|
```
|
|
|
|
Fix hostname
|
|
|
|
```
|
|
echo vm0.bknix.co.th > /etc/hostname
|
|
hostname `cat /etc/hostname`
|
|
```
|
|
|
|
Fix /etc/unbound/unbound.conf
|
|
|
|
```
|
|
access-control: 127.0.0.0/8 allow
|
|
access-control: 203.159.68.0/22 allow
|
|
access-control: 0.0.0.0/0 refuse
|
|
access-control: ::1 allow
|
|
access-control: ::ffff:127.0.0.1 allow
|
|
access-control: 2001:deb::/48
|
|
access-control: ::0/0 refuse
|
|
```
|
|
|
|
### Make ISO FileSystem
|
|
|
|
```
|
|
lvcreate -n ISOs -L 128G ganeti
|
|
mkdir /ISOs
|
|
mkfs /dev/mapper/ganeti-ISOs
|
|
```
|
|
|
|
and then add it to /etc/fstab
|
|
|
|
```
|
|
/dev/mapper/ganeti-ISOs /ISOs ext4 defaults 0 2
|
|
```
|
|
|
|
and then you can mount it
|
|
|
|
```
|
|
mount /ISOs
|
|
```
|
|
|
|
### Unattended Upgrades, syslog-NG, etc.
|
|
|
|
[Install Unattended Upgrading](https://wiki.rg.net/wiki/UbuntuAutoUpgrade)
|
|
|
|
Install syslog-ng
|
|
|
|
```
|
|
apt-get install syslog-ng
|
|
```
|
|
|
|
Hack /etc/logrotate.d/syslog-ng
|
|
|
|
```
|
|
*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
|
|
--- /etc/logrotate.d/syslog-ng 2014-07-31 14:00:08.148813531 +0100
|
|
***************
|
|
*** 29,34 ****
|
|
--- 29,36 ----
|
|
missingok
|
|
notifempty
|
|
compress
|
|
+ mailfirst
|
|
+ mail randy@psg.com
|
|
delaycompress
|
|
sharedscripts
|
|
postrotate
|
|
```
|
|
|
|
Silence log spew to console tty by hacking /etc/syslog-ng/syslog-ng.conf
|
|
|
|
```
|
|
# log { source(s_src); filter(f_console); destination(d_console_all);
|
|
# destination(d_xconsole); };
|
|
# log { source(s_src); filter(f_crit); destination(d_console); };
|
|
```
|
|
|
|
Fix Exim
|
|
|
|
```
|
|
dpkg-reconfigure exim4-config
|
|
```
|
|
|
|
Use internet mail
|
|
System name is vm0.bknix.co.th
|
|
Only listen on local interfaces
|
|
Relay mail for noone
|
|
|
|
## Debian Ganeti Specific Configuration
|
|
|
|
Edit /etc/hosts to have the real address of the host, e.g.
|
|
|
|
```
|
|
127.0.0.1 localhost
|
|
198.180.150.8 vm0.iad.rg.net vm0
|
|
198.180.150.4 vm1.iad.rg.net vm1
|
|
198.180.150.100 gnt0.iad.rg.net gnt0
|
|
```
|
|
|
|
### Fix /etc/network/interfaces
|
|
|
|
Make eth0 hang off of whatever your bridge will be called
|
|
|
|
```
|
|
# This file describes the network interfaces available on your system
|
|
# and how to activate them. For more information, see interfaces(5).
|
|
|
|
# The loopback network interface
|
|
auto lo
|
|
iface lo inet loopback
|
|
|
|
# The primary network interface
|
|
auto eth0
|
|
iface eth0 inet manual
|
|
|
|
auto br-lan
|
|
iface br-lan inet static
|
|
address 198.180.150.4
|
|
netmask 255.255.255.128
|
|
gateway 198.180.150.126
|
|
dns-nameservers 127.0.0.1 198.180.150.1 147.28.0.35
|
|
dns-search psg.com rg.net rpki.net
|
|
bridge_ports eth0
|
|
bridge_stp off
|
|
bridge_fd 0
|
|
bridge_maxwait 0
|
|
|
|
iface br-lan inet6 static
|
|
address 2001:418:8006::4
|
|
netmask 64
|
|
gateway 2001:418:1::126
|
|
|
|
auto eth0.255
|
|
iface eth0.255 inet manual
|
|
|
|
auto br-svc
|
|
iface br-svc inet manual
|
|
bridge_ports eth0.255
|
|
bridge_stp off
|
|
bridge_fd 0
|
|
bridge_maxwait 0
|
|
|
|
auto eth1
|
|
iface eth1 inet manual
|
|
|
|
auto br-hack
|
|
iface br-hack inet static
|
|
address 10.0.0.101
|
|
netmask 255.255.255.0
|
|
bridge_ports eth1
|
|
bridge_stp off
|
|
bridge_fd 0
|
|
bridge_maxwait 0
|
|
```
|
|
|
|
Check /etc.resolv.comf
|
|
|
|
In theory, this looks like
|
|
|
|
```
|
|
-------------+--------------
|
|
|
|
|
br-lan
|
|
| this host
|
|
+---------+---------+
|
|
| eth0 |
|
|
| |
|
|
|eth0.255 eth0.101|
|
|
+--------+----------+
|
|
|
|
|
br-hack
|
|
|
|
|
+------> to other ganeti hosts
|
|
```
|
|
|
|
## Install Ganeti
|
|
|
|
Set up to get Ganeti from backports on all three servers, vm0, vm1, and vm2
|
|
|
|
```
|
|
cat >> /etc/apt/sources.list.d/wheezy-backports.list
|
|
deb http://cdn.debian.net/debian/ wheezy-backports main
|
|
```
|
|
|
|
And then install it on all three servers
|
|
|
|
```
|
|
apt-get update
|
|
apt-get install ganeti/wheezy-backports
|
|
```
|
|
|
|
Fix up drbd
|
|
|
|
```
|
|
echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
|
|
rmmod drbd # ignore any error
|
|
modprobe drbd
|
|
```
|
|
|
|
### Initialize and Build Ganeti Cluster
|
|
|
|
```
|
|
gnt-cluster init \
|
|
--master-netdev=br-internal \
|
|
--enabled-hypervisors=kvm \
|
|
-H kvm:kernel_path="",initrd_path="" \
|
|
--vg-name=ganeti \
|
|
-N link=br-cluster \
|
|
-s 10.0.0.101 \
|
|
gnt0.iad.rg.net
|
|
```
|
|
|
|
If it barfs, try
|
|
|
|
```
|
|
gnt-cluster destroy --yes-do-it
|
|
```
|
|
|
|
then re-run gnt-cluster init ...
|
|
|
|
If you get into gnutls problems check apt-get upgrade and dist-upgrade
|
|
|
|
### Check That Cluster is Happy
|
|
|
|
Fix VNC passwording
|
|
|
|
```
|
|
echo 'fnorgille' > /etc/ganeti/vnc-cluster-password
|
|
gnt-cluster modify -H kvm:vnc_password_file=/etc/ganeti/vnc-cluster-password
|
|
```
|
|
|
|
Test that the cluster was built happily
|
|
|
|
```
|
|
gnt-cluster verify
|
|
```
|
|
|
|
If you get
|
|
|
|
```
|
|
Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/root is unknown
|
|
Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/swap is unknown
|
|
Fri Mar 14 05:20:55 2014 - ERROR: node deb64.psg.com: volume ganeti/var is unknown
|
|
```
|
|
|
|
That's OK. To make it go away
|
|
|
|
```
|
|
gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var
|
|
```
|
|
|
|
Now add vm1 and vm2 to the cluster by the following on vm0
|
|
|
|
```
|
|
gnt-node add -s 10.0.1.11 vm1.bknix.co.th
|
|
gnt-node add -s 10.0.1.12 vm2.bknix.co.th
|
|
```
|
|
|
|
which should result in
|
|
|
|
```
|
|
vm0.bknix.co.th:/root# gnt-node add -s 10.0.1.11 vm1.bknix.co.th
|
|
-- WARNING --
|
|
Performing this operation is going to replace the ssh daemon keypair
|
|
on the target machine (vm1.bknix.co.th) with the ones of the current one
|
|
and grant full intra-cluster ssh root access to/from it
|
|
|
|
The authenticity of host 'vm1.bknix.co.th (203.159.71.11)' can't be established.
|
|
ECDSA key fingerprint is 2f:48:91:f7:f2:dd:c4:95:da:30:d2:f4:92:6a:53:ec.
|
|
Are you sure you want to continue connecting (yes/no)? yes
|
|
Warning: Permanently added 'vm1.bknix.co.th' (ECDSA) to the list of known hosts.
|
|
Restarting OpenBSD Secure Shell server: sshd.
|
|
Fri Aug 22 05:37:53 2014 - INFO: Node will be a master candidate
|
|
```
|
|
|
|
## Convert a libvirt qcow2 to a ganeti instance
|
|
|
|
get noop os type, v4 only!
|
|
|
|
```
|
|
echo "deb http://repo.noc.grnet.gr/ wheezy main" >> /etc/apt/sources.list.d/grnet.list
|
|
wget -O - http://repo.noc.grnet.gr/grnet.gpg.key|apt-key add -
|
|
apt-get update
|
|
apt-get install ganeti-os-noop
|
|
```
|
|
|
|
Find the disk size the qcow2 will require
|
|
|
|
```
|
|
qemu-img info <foo>.qcow2
|
|
```
|
|
|
|
Create the empty Ganeti instance
|
|
|
|
```
|
|
./do-add 32 4 <foo>
|
|
```
|
|
|
|
where the do-add script is
|
|
|
|
```
|
|
#!/bin/sh
|
|
|
|
# makeVM diskGB ramGB nameFQDN
|
|
|
|
DISK=$1
|
|
RAM=$2
|
|
NAME=$3
|
|
NODE=vm1.iad.rg.net
|
|
|
|
gnt-instance add \
|
|
-t raw \
|
|
-o noop \
|
|
-s ${DISK}G \
|
|
-B maxmem=${RAM}G,minmem=$((${RAM}/2))G \
|
|
-n $NODE \
|
|
-H kvm:vnc_bind_address=0.0.0.0 \
|
|
--no-install \
|
|
--no-start \
|
|
--no-ip-check \
|
|
--no-name-check \
|
|
${NAME}
|
|
```
|
|
|
|
then
|
|
|
|
### Hint: Copying Images Between Systems
|
|
|
|
rsync and so forth spend a lot of time in crypto etc. If you are copying an image across a local LAN, `netcat` is your friend.
|
|
On the source
|
|
|
|
```
|
|
nc -l -p 42420 < path-to/name-of.qcow2
|
|
```
|
|
|
|
On the destination
|
|
|
|
```
|
|
nc -w3 source.fqdn 42420 > name-of.qcow2
|
|
```
|
|
|
|
Of course, do an md5 or sha compare of the source and destination files before using them.
|
|
|
|
### Actually Convert qcow2 to a Ganeti Image
|
|
|
|
```
|
|
gnt-instance activate-disks <foo>
|
|
```
|
|
|
|
and you will get the disk ID of the instance
|
|
|
|
```
|
|
vm1.iad.rg.net:disk/0:/dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0
|
|
```
|
|
|
|
convert the qcow2 directly to the instance image
|
|
|
|
```
|
|
qemu-img convert <foo>.qcow2 /dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0
|
|
```
|
|
|
|
If qemu-convert wants more target disk space then you thought you had in the source disk, you will get
|
|
|
|
```
|
|
qemu-img: /dev/ganeti/c316d162-af4d-4ce0-8d1f-7b0b01919053.disk0: error while converting raw: No space left on device
|
|
```
|
|
|
|
The only solution seems to be to allocate more disk in the do-add
|
|
|
|
Now you can start the converted image
|
|
|
|
```
|
|
gnt-instance start <foo>
|
|
```
|
|
|
|
### Moving Secondary Disk Drives from libvert to a Ganeti Instance
|
|
|
|
First, create the empty disk on the Ganeti image
|
|
|
|
```
|
|
gnt-instance modify --disk add:size=1024G <instance-name>
|
|
```
|
|
|
|
Then find out the magic filename of the disk
|
|
|
|
```
|
|
gnt-instance activate-disks <instance-name>
|
|
vm1.iad.rg.net:disk/0:/dev/ganeti/15df702f-3535-4a99-90e1-42d5ae00f469.disk0
|
|
vm1.iad.rg.net:disk/1:/dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1
|
|
```
|
|
|
|
disk0 is the base Ganeti image disk, disk1 has been added.
|
|
|
|
The libvert secondary disk is a raw image, not a qcow. So you do a move directly into place.
|
|
|
|
On the libvert side
|
|
|
|
```
|
|
nc -l -p 42420 < path-to-disk.sdb
|
|
```
|
|
|
|
On the Ganeti side
|
|
|
|
```
|
|
nc -w3 vm0.iad 42420 > /dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1
|
|
```
|
|
|
|
When it is finished, you need to
|
|
|
|
```
|
|
gnt-instance restart <instance-name>
|
|
``` |