diff --git a/pages/ClusterUpgrde-2.12.md b/pages/ClusterUpgrde-2.12.md new file mode 100644 index 0000000..1304661 --- /dev/null +++ b/pages/ClusterUpgrde-2.12.md @@ -0,0 +1,226 @@ +# Upgrading a Cluster from Debian7 to Debian8 and Ganeti from 2.11 to 2.12 + +## Being sure we're ready to Upgrade + +the cluster before i broke it + +``` +vm1.sql1.ietf.org:/root# ./do-stat +Node DTotal DFree MTotal MNode MFree Pinst Sinst +vm1.sql1.ietf.org 7.3T 6.5T 110.3G 20.0G 94.0G 8 0 +vm2.sql1.ietf.org 7.3T 7.1T 126.1G 769M 125.4G 0 3 +vm3.sql1.ietf.org 7.3T 6.9T 126.1G 766M 125.2G 0 4 + +Instance Primary_node Secondary_Nodes gMaxMem DiskUsage +hackathon.sql1.ietf.org vm1.sql1.ietf.org 4.0G 2.1G +management.sql1.ietf.org vm1.sql1.ietf.org vm2.sql1.ietf.or 16.0G 48.1G +netdot.sql1.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 40.1G +noc.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 256.1G +permatrac.sql1.ietf.org vm1.sql1.ietf.org vm2.sql1.ietf.or 4.0G 16.1G +services-1.sql1.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 8.1G +services-2.sql1.ietf.org vm1.sql1.ietf.org vm2.sql1.ietf.or 4.0G 8.1G +trac.sql1.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 8.1G +``` + +and it verified clean + +``` +vm1.sql1.ietf.org:/root# gnt-cluster verify +Submitted jobs 95884, 95885 +Waiting for job 95884 ... +Fri Jun 17 23:35:25 2016 * Verifying cluster config +Fri Jun 17 23:35:25 2016 * Verifying cluster certificate files +Fri Jun 17 23:35:25 2016 * Verifying hypervisor parameters +Fri Jun 17 23:35:25 2016 * Verifying all nodes belong to an existing group +Waiting for job 95885 ... +Fri Jun 17 23:35:25 2016 * Verifying group 'default' +Fri Jun 17 23:35:25 2016 * Gathering data (3 nodes) +Fri Jun 17 23:35:26 2016 * Gathering disk information (3 nodes) +Fri Jun 17 23:35:26 2016 * Verifying configuration file consistency +Fri Jun 17 23:35:26 2016 * Verifying node status +Fri Jun 17 23:35:26 2016 * Verifying instance status +Fri Jun 17 23:35:26 2016 * Verifying orphan volumes +Fri Jun 17 23:35:27 2016 * Verifying N+1 Memory redundancy +Fri Jun 17 23:35:27 2016 * Other Notes +Fri Jun 17 23:35:27 2016 - NOTICE: 1 non-redundant instance(s) found. +Fri Jun 17 23:35:27 2016 * Hooks Results +``` + +i did apt-get update and upgrade to get wheezy current. i did not reboot. + +## Prepare to Upgrade Debian 7 to Debian 8 + +i s/wheezy/jessie/g in /etc/apt/sources.list and distributed it to all servers with copyfile + +``` +deb http://ftp.us.debian.org/debian/ jessie main contrib non-free +deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free +deb http://security.debian.org/ jessie/updates main +deb-src http://security.debian.org/ jessie/updates main +``` + +i pushed the hack to force apt to use v4 only, as i had failures over v6 with the green cluster. + +``` + - name: copy 99force-ipv4 + copy: src=files/linux/99force-ipv4 + dest=/etc/apt/apt.conf.d/99force-ipv4 + owner=root + group=root + mode=0644 +# cat files/linux/99force-ipv4 +Acquire::ForceIPv4 "true"; +``` + +i shut down the VMs + +``` +gnt-instance shutdown --all +``` + +## Upgrade Debian 7 to Debian 8 + +i ran + +``` +apt-get update +apt-get upgrade +``` + +on all three nodes + +followed by + +``` +apt-get dist-upgrade +``` + +i chose the new maintainer version of /etc/lvm/lvm.conf + +i rebooted vm1. it worked. so i rebooted vm2 an dvm3. + +ganeti came up with vms, so i did another + +``` +gnt-instance shutdown --all +``` + +## Preparing to Upgrade Ganeti + +i did another + +``` +# gnt-cluster verify +Submitted jobs 95915, 95916 +Waiting for job 95915 ... +Sat Jun 18 00:50:15 2016 * Verifying cluster config +Sat Jun 18 00:50:15 2016 * Verifying cluster certificate files +Sat Jun 18 00:50:15 2016 * Verifying hypervisor parameters +Sat Jun 18 00:50:15 2016 * Verifying all nodes belong to an existing group +Waiting for job 95916 ... +Sat Jun 18 00:50:16 2016 * Verifying group 'default' +Sat Jun 18 00:50:16 2016 * Gathering data (3 nodes) +Sat Jun 18 00:50:18 2016 * Gathering disk information (3 nodes) +Sat Jun 18 00:50:18 2016 * Verifying configuration file consistency +Sat Jun 18 00:50:18 2016 * Verifying node status +Sat Jun 18 00:50:18 2016 * Verifying instance status +Sat Jun 18 00:50:19 2016 * Verifying orphan volumes +Sat Jun 18 00:50:19 2016 * Verifying N+1 Memory redundancy +Sat Jun 18 00:50:19 2016 * Other Notes +Sat Jun 18 00:50:19 2016 - NOTICE: 1 non-redundant instance(s) found. +Sat Jun 18 00:50:19 2016 * Hooks Results +``` + +i installed 2.12 on all nodes + +``` +# apt-get install ganeti +``` + +which was a noop due to previous upgrade + +i checked that 2.12 was ready and waiting on all nodes + +``` +# dpkg --list |grep ganeti +ii ganeti 2.12.4-1+deb8u3 all cluster virtualization manager +ii ganeti-2.11 2.11.6-1~bpo70+1 all cluster virtualization manager - Python components +ii ganeti-2.12 2.12.4-1+deb8u3 all cluster virtualization manager - Python components +ii ganeti-haskell-2.11 2.11.6-1~bpo70+1 amd64 cluster virtualization manager - Haskell components +ii ganeti-haskell-2.12 2.12.4-1+deb8u3 amd64 cluster virtualization manager - Haskell components +ii ganeti-htools-2.11 2.11.6-1~bpo70+1 amd64 cluster virtualization manager - tools for Ganeti 2.11 +ii ganeti-htools-2.12 2.12.4-1+deb8u3 amd64 cluster virtualization manager - tools for Ganeti 2.12 +ii ganeti-instance-debootstrap 0.14-2 all debootstrap-based instance OS definition for ganeti +ii ganeti-instance-image 0.6-1+grnet17 all image based instance OS definition for ganeti +ii ganeti-os-noop 0.1-3 all A dummy no-op OS provider for ganeti +``` + +and that i was still really running 2.11 + +``` +# ls -l /etc/ganeti/share +lrwxrwxrwx 1 root root 22 Jul 22 2015 /etc/ganeti/share -> /usr/share/ganeti/2.11/ +``` + +## Do the Ganeti Upgrade + +and then, the moment we had all been awaiting + +``` +vm1.sql1.ietf.org:/home/randy# gnt-cluster upgrade --to 2.12 +Draining queue +Pausing the watcher for one hour. +Stopping daemons on master node. +Stopping daemons everywhere. +Backing up configuration as /var/backups/ganeti1466213017.tar +Switching to version 2.12 on all nodes +Upgrading configuration +Ensuring directories everywhere. +Starting daemons everywhere. +Redistributing the configuration. +Restarting daemons everywhere. +Undraining the queue. +Running post-upgrade hooks +Unpasuing the watcher. +Verifying cluster. +``` + +## Inspect Cluster and Restart VMs + +but i wanted to see for myself + +``` +# gnt-cluster verify +Submitted jobs 95924, 95925 +Waiting for job 95924 ... +Sat Jun 18 01:25:47 2016 * Verifying cluster config +Sat Jun 18 01:25:47 2016 * Verifying cluster certificate files +Sat Jun 18 01:25:47 2016 * Verifying hypervisor parameters +Sat Jun 18 01:25:47 2016 * Verifying all nodes belong to an existing group +Waiting for job 95925 ... +Sat Jun 18 01:25:49 2016 * Verifying group 'default' +Sat Jun 18 01:25:49 2016 * Gathering data (3 nodes) +Sat Jun 18 01:25:49 2016 * Gathering information about nodes (3 nodes) +Sat Jun 18 01:25:51 2016 * Gathering disk information (3 nodes) +Sat Jun 18 01:25:51 2016 * Verifying configuration file consistency +Sat Jun 18 01:25:51 2016 * Verifying node status +Sat Jun 18 01:25:51 2016 * Verifying instance status +Sat Jun 18 01:25:51 2016 * Verifying orphan volumes +Sat Jun 18 01:25:51 2016 * Verifying N+1 Memory redundancy +Sat Jun 18 01:25:51 2016 * Other Notes +Sat Jun 18 01:25:51 2016 - NOTICE: 1 non-redundant instance(s) found. +Sat Jun 18 01:25:52 2016 * Hooks Results +``` + +and confirm which version we are running + +``` +# ls -l /etc/ganeti/share +lrwxrwxrwx 1 root root 22 Jul 22 2015 /etc/ganeti/share -> /usr/share/ganeti/2.11/ +``` + +and finally + +``` +# gnt-instance start --all +``` \ No newline at end of file