from wiki
This commit is contained in:
parent
a8ffa2adf3
commit
cb59ae54a1
1 changed files with 226 additions and 0 deletions
226
pages/ClusterUpgrde-2.12.md
Normal file
226
pages/ClusterUpgrde-2.12.md
Normal file
|
|
@ -0,0 +1,226 @@
|
||||||
|
# Upgrading a Cluster from Debian7 to Debian8 and Ganeti from 2.11 to 2.12
|
||||||
|
|
||||||
|
## Being sure we're ready to Upgrade
|
||||||
|
|
||||||
|
the cluster before i broke it
|
||||||
|
|
||||||
|
```
|
||||||
|
vm1.sql1.ietf.org:/root# ./do-stat
|
||||||
|
Node DTotal DFree MTotal MNode MFree Pinst Sinst
|
||||||
|
vm1.sql1.ietf.org 7.3T 6.5T 110.3G 20.0G 94.0G 8 0
|
||||||
|
vm2.sql1.ietf.org 7.3T 7.1T 126.1G 769M 125.4G 0 3
|
||||||
|
vm3.sql1.ietf.org 7.3T 6.9T 126.1G 766M 125.2G 0 4
|
||||||
|
|
||||||
|
Instance Primary_node Secondary_Nodes gMaxMem DiskUsage
|
||||||
|
hackathon.sql1.ietf.org vm1.sql1.ietf.org 4.0G 2.1G
|
||||||
|
management.sql1.ietf.org vm1.sql1.ietf.org vm2.sql1.ietf.or 16.0G 48.1G
|
||||||
|
netdot.sql1.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 40.1G
|
||||||
|
noc.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 256.1G
|
||||||
|
permatrac.sql1.ietf.org vm1.sql1.ietf.org vm2.sql1.ietf.or 4.0G 16.1G
|
||||||
|
services-1.sql1.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 8.1G
|
||||||
|
services-2.sql1.ietf.org vm1.sql1.ietf.org vm2.sql1.ietf.or 4.0G 8.1G
|
||||||
|
trac.sql1.ietf.org vm1.sql1.ietf.org vm3.sql1.ietf.or 4.0G 8.1G
|
||||||
|
```
|
||||||
|
|
||||||
|
and it verified clean
|
||||||
|
|
||||||
|
```
|
||||||
|
vm1.sql1.ietf.org:/root# gnt-cluster verify
|
||||||
|
Submitted jobs 95884, 95885
|
||||||
|
Waiting for job 95884 ...
|
||||||
|
Fri Jun 17 23:35:25 2016 * Verifying cluster config
|
||||||
|
Fri Jun 17 23:35:25 2016 * Verifying cluster certificate files
|
||||||
|
Fri Jun 17 23:35:25 2016 * Verifying hypervisor parameters
|
||||||
|
Fri Jun 17 23:35:25 2016 * Verifying all nodes belong to an existing group
|
||||||
|
Waiting for job 95885 ...
|
||||||
|
Fri Jun 17 23:35:25 2016 * Verifying group 'default'
|
||||||
|
Fri Jun 17 23:35:25 2016 * Gathering data (3 nodes)
|
||||||
|
Fri Jun 17 23:35:26 2016 * Gathering disk information (3 nodes)
|
||||||
|
Fri Jun 17 23:35:26 2016 * Verifying configuration file consistency
|
||||||
|
Fri Jun 17 23:35:26 2016 * Verifying node status
|
||||||
|
Fri Jun 17 23:35:26 2016 * Verifying instance status
|
||||||
|
Fri Jun 17 23:35:26 2016 * Verifying orphan volumes
|
||||||
|
Fri Jun 17 23:35:27 2016 * Verifying N+1 Memory redundancy
|
||||||
|
Fri Jun 17 23:35:27 2016 * Other Notes
|
||||||
|
Fri Jun 17 23:35:27 2016 - NOTICE: 1 non-redundant instance(s) found.
|
||||||
|
Fri Jun 17 23:35:27 2016 * Hooks Results
|
||||||
|
```
|
||||||
|
|
||||||
|
i did apt-get update and upgrade to get wheezy current. i did not reboot.
|
||||||
|
|
||||||
|
## Prepare to Upgrade Debian 7 to Debian 8
|
||||||
|
|
||||||
|
i s/wheezy/jessie/g in /etc/apt/sources.list and distributed it to all servers with copyfile
|
||||||
|
|
||||||
|
```
|
||||||
|
deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
|
||||||
|
deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free
|
||||||
|
deb http://security.debian.org/ jessie/updates main
|
||||||
|
deb-src http://security.debian.org/ jessie/updates main
|
||||||
|
```
|
||||||
|
|
||||||
|
i pushed the hack to force apt to use v4 only, as i had failures over v6 with the green cluster.
|
||||||
|
|
||||||
|
```
|
||||||
|
- name: copy 99force-ipv4
|
||||||
|
copy: src=files/linux/99force-ipv4
|
||||||
|
dest=/etc/apt/apt.conf.d/99force-ipv4
|
||||||
|
owner=root
|
||||||
|
group=root
|
||||||
|
mode=0644
|
||||||
|
# cat files/linux/99force-ipv4
|
||||||
|
Acquire::ForceIPv4 "true";
|
||||||
|
```
|
||||||
|
|
||||||
|
i shut down the VMs
|
||||||
|
|
||||||
|
```
|
||||||
|
gnt-instance shutdown --all
|
||||||
|
```
|
||||||
|
|
||||||
|
## Upgrade Debian 7 to Debian 8
|
||||||
|
|
||||||
|
i ran
|
||||||
|
|
||||||
|
```
|
||||||
|
apt-get update
|
||||||
|
apt-get upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
on all three nodes
|
||||||
|
|
||||||
|
followed by
|
||||||
|
|
||||||
|
```
|
||||||
|
apt-get dist-upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
i chose the new maintainer version of /etc/lvm/lvm.conf
|
||||||
|
|
||||||
|
i rebooted vm1. it worked. so i rebooted vm2 an dvm3.
|
||||||
|
|
||||||
|
ganeti came up with vms, so i did another
|
||||||
|
|
||||||
|
```
|
||||||
|
gnt-instance shutdown --all
|
||||||
|
```
|
||||||
|
|
||||||
|
## Preparing to Upgrade Ganeti
|
||||||
|
|
||||||
|
i did another
|
||||||
|
|
||||||
|
```
|
||||||
|
# gnt-cluster verify
|
||||||
|
Submitted jobs 95915, 95916
|
||||||
|
Waiting for job 95915 ...
|
||||||
|
Sat Jun 18 00:50:15 2016 * Verifying cluster config
|
||||||
|
Sat Jun 18 00:50:15 2016 * Verifying cluster certificate files
|
||||||
|
Sat Jun 18 00:50:15 2016 * Verifying hypervisor parameters
|
||||||
|
Sat Jun 18 00:50:15 2016 * Verifying all nodes belong to an existing group
|
||||||
|
Waiting for job 95916 ...
|
||||||
|
Sat Jun 18 00:50:16 2016 * Verifying group 'default'
|
||||||
|
Sat Jun 18 00:50:16 2016 * Gathering data (3 nodes)
|
||||||
|
Sat Jun 18 00:50:18 2016 * Gathering disk information (3 nodes)
|
||||||
|
Sat Jun 18 00:50:18 2016 * Verifying configuration file consistency
|
||||||
|
Sat Jun 18 00:50:18 2016 * Verifying node status
|
||||||
|
Sat Jun 18 00:50:18 2016 * Verifying instance status
|
||||||
|
Sat Jun 18 00:50:19 2016 * Verifying orphan volumes
|
||||||
|
Sat Jun 18 00:50:19 2016 * Verifying N+1 Memory redundancy
|
||||||
|
Sat Jun 18 00:50:19 2016 * Other Notes
|
||||||
|
Sat Jun 18 00:50:19 2016 - NOTICE: 1 non-redundant instance(s) found.
|
||||||
|
Sat Jun 18 00:50:19 2016 * Hooks Results
|
||||||
|
```
|
||||||
|
|
||||||
|
i installed 2.12 on all nodes
|
||||||
|
|
||||||
|
```
|
||||||
|
# apt-get install ganeti
|
||||||
|
```
|
||||||
|
|
||||||
|
which was a noop due to previous upgrade
|
||||||
|
|
||||||
|
i checked that 2.12 was ready and waiting on all nodes
|
||||||
|
|
||||||
|
```
|
||||||
|
# dpkg --list |grep ganeti
|
||||||
|
ii ganeti 2.12.4-1+deb8u3 all cluster virtualization manager
|
||||||
|
ii ganeti-2.11 2.11.6-1~bpo70+1 all cluster virtualization manager - Python components
|
||||||
|
ii ganeti-2.12 2.12.4-1+deb8u3 all cluster virtualization manager - Python components
|
||||||
|
ii ganeti-haskell-2.11 2.11.6-1~bpo70+1 amd64 cluster virtualization manager - Haskell components
|
||||||
|
ii ganeti-haskell-2.12 2.12.4-1+deb8u3 amd64 cluster virtualization manager - Haskell components
|
||||||
|
ii ganeti-htools-2.11 2.11.6-1~bpo70+1 amd64 cluster virtualization manager - tools for Ganeti 2.11
|
||||||
|
ii ganeti-htools-2.12 2.12.4-1+deb8u3 amd64 cluster virtualization manager - tools for Ganeti 2.12
|
||||||
|
ii ganeti-instance-debootstrap 0.14-2 all debootstrap-based instance OS definition for ganeti
|
||||||
|
ii ganeti-instance-image 0.6-1+grnet17 all image based instance OS definition for ganeti
|
||||||
|
ii ganeti-os-noop 0.1-3 all A dummy no-op OS provider for ganeti
|
||||||
|
```
|
||||||
|
|
||||||
|
and that i was still really running 2.11
|
||||||
|
|
||||||
|
```
|
||||||
|
# ls -l /etc/ganeti/share
|
||||||
|
lrwxrwxrwx 1 root root 22 Jul 22 2015 /etc/ganeti/share -> /usr/share/ganeti/2.11/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Do the Ganeti Upgrade
|
||||||
|
|
||||||
|
and then, the moment we had all been awaiting
|
||||||
|
|
||||||
|
```
|
||||||
|
vm1.sql1.ietf.org:/home/randy# gnt-cluster upgrade --to 2.12
|
||||||
|
Draining queue
|
||||||
|
Pausing the watcher for one hour.
|
||||||
|
Stopping daemons on master node.
|
||||||
|
Stopping daemons everywhere.
|
||||||
|
Backing up configuration as /var/backups/ganeti1466213017.tar
|
||||||
|
Switching to version 2.12 on all nodes
|
||||||
|
Upgrading configuration
|
||||||
|
Ensuring directories everywhere.
|
||||||
|
Starting daemons everywhere.
|
||||||
|
Redistributing the configuration.
|
||||||
|
Restarting daemons everywhere.
|
||||||
|
Undraining the queue.
|
||||||
|
Running post-upgrade hooks
|
||||||
|
Unpasuing the watcher.
|
||||||
|
Verifying cluster.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Inspect Cluster and Restart VMs
|
||||||
|
|
||||||
|
but i wanted to see for myself
|
||||||
|
|
||||||
|
```
|
||||||
|
# gnt-cluster verify
|
||||||
|
Submitted jobs 95924, 95925
|
||||||
|
Waiting for job 95924 ...
|
||||||
|
Sat Jun 18 01:25:47 2016 * Verifying cluster config
|
||||||
|
Sat Jun 18 01:25:47 2016 * Verifying cluster certificate files
|
||||||
|
Sat Jun 18 01:25:47 2016 * Verifying hypervisor parameters
|
||||||
|
Sat Jun 18 01:25:47 2016 * Verifying all nodes belong to an existing group
|
||||||
|
Waiting for job 95925 ...
|
||||||
|
Sat Jun 18 01:25:49 2016 * Verifying group 'default'
|
||||||
|
Sat Jun 18 01:25:49 2016 * Gathering data (3 nodes)
|
||||||
|
Sat Jun 18 01:25:49 2016 * Gathering information about nodes (3 nodes)
|
||||||
|
Sat Jun 18 01:25:51 2016 * Gathering disk information (3 nodes)
|
||||||
|
Sat Jun 18 01:25:51 2016 * Verifying configuration file consistency
|
||||||
|
Sat Jun 18 01:25:51 2016 * Verifying node status
|
||||||
|
Sat Jun 18 01:25:51 2016 * Verifying instance status
|
||||||
|
Sat Jun 18 01:25:51 2016 * Verifying orphan volumes
|
||||||
|
Sat Jun 18 01:25:51 2016 * Verifying N+1 Memory redundancy
|
||||||
|
Sat Jun 18 01:25:51 2016 * Other Notes
|
||||||
|
Sat Jun 18 01:25:51 2016 - NOTICE: 1 non-redundant instance(s) found.
|
||||||
|
Sat Jun 18 01:25:52 2016 * Hooks Results
|
||||||
|
```
|
||||||
|
|
||||||
|
and confirm which version we are running
|
||||||
|
|
||||||
|
```
|
||||||
|
# ls -l /etc/ganeti/share
|
||||||
|
lrwxrwxrwx 1 root root 22 Jul 22 2015 /etc/ganeti/share -> /usr/share/ganeti/2.11/
|
||||||
|
```
|
||||||
|
|
||||||
|
and finally
|
||||||
|
|
||||||
|
```
|
||||||
|
# gnt-instance start --all
|
||||||
|
```
|
||||||
Loading…
Add table
Add a link
Reference in a new issue