# Upgrading Ganeti 2.12 to 2.15 on Debian Jessie

I had 2.12 on debian jessie on three clusters so wanted to upgrade.

## apt sources

Get the correct backports on all nodes in the cluster.

```
# echo "deb http://cdn.debian.net/debian/ jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list 
```

I also had to

```
rm /etc/apt/sources.list.d/wheezy-backports.list
rm /etc/apt/sources.list.d/ganeti.list
```

Update on all nodes, and you should have 2.15 staged.

```
# apt-get update
# apt-cache policy ganeti
ganeti:
  Installed: 2.12.4-1+deb8u3
  Candidate: 2.12.4-1+deb8u3
  Version table:
     2.15.2-1~bpo8+1 0
        100 http://cdn.debian.net/debian/ jessie-backports/main amd64 Packages
 *** 2.12.4-1+deb8u3 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
        100 /var/lib/dpkg/status
```

## Get the binaries

Install new binaries on all nodes.

```
# apt-get install ganeti/jessie-backports
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '2.15.2-1~bpo8+1' (Debian Backports:jessie-backports [all]) for 'ganeti'
The following packages were automatically installed and are no longer required:
  ganeti-2.12 ganeti-haskell-2.12 ganeti-htools-2.12 libdns-export100
  libirs-export91 libisc-export95 libisccfg-export90
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
Suggested packages:
  ganeti-doc blktap-dkms molly-guard
The following NEW packages will be installed:
  ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
The following packages will be upgraded:
  ganeti
1 upgraded, 3 newly installed, 0 to remove and 55 not upgraded.
Need to get 13.8 MB of archives.
After this operation, 117 MB of additional disk space will be used.
Do you want to continue? [Y/n]
```

And you should now see that 2.15 is current, though not yet running

```
# apt-cache policy ganeti
ganeti:
  Installed: 2.15.2-1~bpo8+1
  Candidate: 2.15.2-1~bpo8+1
  Version table:
 *** 2.15.2-1~bpo8+1 0
        100 http://cdn.debian.net/debian/ jessie-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     2.12.4-1+deb8u3 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
```

## Do the Upgrade

Note that you do not have to shutdown the running VMs.

Ensure that the job queue is drained first

```
# gnt-job list
ID Status Summary
```

If it is not empty, either cancel the jobs or, if desperate, hold your  nose and

```
# rm /var/lib/ganeti/queue/job*
```

On the master node

```
# gnt-cluster upgrade --to 2.15
Draining queue
Pausing the watcher for one hour.
Stopping daemons on master node.
Stopping daemons everywhere.
Backing up configuration as /var/backups/ganeti1473770062.tar
Switching to version 2.15 on all nodes
Upgrading configuration
Ensuring directories everywhere.
Starting daemons everywhere.
Redistributing the configuration.
Restarting daemons everywhere.
Undraining the queue.
Running post-upgrade hooks
Unpausing the watcher.
Verifying cluster.
```

Note that it verified in secret.  You may want to do so explicitly

```
# gnt-cluster verify
Submitted jobs 301175, 301176
Waiting for job 301175 ...
Tue Sep 13 12:38:33 2016 * Verifying cluster config
Tue Sep 13 12:38:33 2016 * Verifying cluster certificate files
Tue Sep 13 12:38:33 2016 * Verifying hypervisor parameters
Tue Sep 13 12:38:33 2016 * Verifying all nodes belong to an existing group
Waiting for job 301176 ...
Tue Sep 13 12:38:35 2016 * Verifying group 'default'
Tue Sep 13 12:38:35 2016 * Gathering data (2 nodes)
Tue Sep 13 12:38:35 2016 * Gathering information about nodes (2 nodes)
Tue Sep 13 12:38:41 2016 * Gathering disk information (2 nodes)
Tue Sep 13 12:39:05 2016 * Verifying configuration file consistency
Tue Sep 13 12:39:05 2016 * Verifying node status
Tue Sep 13 12:39:05 2016 * Verifying instance status
Tue Sep 13 12:39:05 2016 * Verifying orphan volumes
Tue Sep 13 12:39:05 2016 * Verifying N+1 Memory redundancy
Tue Sep 13 12:39:05 2016 * Other Notes
Tue Sep 13 12:39:05 2016   - NOTICE: 2 non-redundant instance(s) found.
Tue Sep 13 12:39:05 2016 * Hooks Results
```

Confirm which version is running

```
# ls -l /etc/ganeti/share
lrwxrwxrwx 1 root root 22 Sep 13 12:34 /etc/ganeti/share -> /usr/share/ganeti/2.15/
```

## Upgrade keys

On the master node

```
# gnt-cluster renew-crypto --new-ssh-keys --no-ssh-key-check
Updating certificates now. Running "gnt-cluster verify"  is recommended after this operation.
This requires all daemons on all nodes to be restarted and may take
some time. Continue?
y/[n]/?: y
Tue Sep 13 12:41:01 2016 Renewing SSH keys
All requested certificates and keys have been replaced. Running "gnt-cluster verify" now is recommended.
```

## Verify

And, of course, always verify.  On the master

```
# gnt-cluster verify
Tue Sep 13 12:41:54 2016 * Verifying cluster config
Tue Sep 13 12:41:54 2016 * Verifying cluster certificate files
Tue Sep 13 12:41:54 2016 * Verifying hypervisor parameters
Tue Sep 13 12:41:54 2016 * Verifying all nodes belong to an existing group
Waiting for job 301181 ...
Tue Sep 13 12:41:56 2016 * Verifying group 'default'
Tue Sep 13 12:41:56 2016 * Gathering data (2 nodes)
Tue Sep 13 12:41:57 2016 * Gathering information about nodes (2 nodes)
Tue Sep 13 12:42:03 2016 * Gathering disk information (2 nodes)
Tue Sep 13 12:43:04 2016 * Verifying configuration file consistency
Tue Sep 13 12:43:04 2016 * Verifying node status
Tue Sep 13 12:43:04 2016 * Verifying instance status
Tue Sep 13 12:43:05 2016 * Verifying orphan volumes
Tue Sep 13 12:43:05 2016 * Verifying N+1 Memory redundancy
Tue Sep 13 12:43:05 2016 * Other Notes
Tue Sep 13 12:43:05 2016   - NOTICE: 2 non-redundant instance(s) found.
Tue Sep 13 12:43:05 2016 * Hooks Results
```

And Bob's your uncle!