from wiki

pages/RegenCertsEtc.md

@@ -0,0 +1,128 @@
+# Hacking a New SSL Certificate Structure for Ganeti 2.12.x
+
+If you have somehow mashed the the server/client certificates on your cluster, or otherwise gotten into a deep SSL mess, and you, as we did, found the advice at  [GanetiAndSSL](https://code.google.com/p/ganeti/wiki/GanetiAndSSL) did not work for you, then maybe the following will help. We speculate that  the damage to our certs was caused during the upgrade from 2.11 to 2.12.  There's lots of evidence from the mailing list postings that the change in security  model bit a lot of people during the upgrade process.  This recipe was hacked at some cost in source diving and certificate exploring (on a working cluster) by Rob Austein, Hans Kuhn, and Randy Bush.
+
+Be aware that this has only been tested for 2.12 and it's possible the security  model will change in the future. There are no guarantees it will even work for  your case and it is offered in the spirit of shared discovery.
+
+Apologies for the examples assuming the node names of our cluster, vm[0123].
+
+## Generate Server and Client Certificates and ssconf_master_candidates_certs
+
+Despite hints in documentation, conferences, presentations, etc.
+
+- There is no Certificate Authority.
+- All certificates are self-signed.
+- Client certificates are NOT signed by the server certificate.
+- Things break if the CN is not the same for all certificates.
+- The client.pem files we found were all ancient X.509v1 certificates with no extensions etc.
+- --no-ssl in /etc/default/ganeti didn't allow us to run without SSL
+
+The following script gets a list of {nodename, uuid} and generates certificates and a new ssconf_master_candidates_certs file.
+
+```
+#!/bin/sh -
+
+make_cert() {
+    local filename=$1 hostname=$2 
+    openssl req -batch					\
+	    -newkey rsa:2048 -keyout $filename -nodes	\
+	    -x509	     -out    $filename		\
+	    -sha1					\
+	    -days   1825				\
+	    -subj   "/CN=$hostname"
+    }
+
+make_cert server.pem ganeti.example.com
+
+rm -f ssconf_master_candidates_certs.new
+
+# the following use of gnt-node assumes a semi-working cluster if not,
+# you need to hack a file of the form {hostname uuid} for all hosts.
+# this can be rescued from the old ssconf_master_candidates_certs
+#cat nodehack |
+gnt-node list -o name,uuid --no-headers |
+while read hn uuid
+do
+    fn=${hn%%.*}.client.pem
+    make_cert $fn ganeti.example.com
+    fp="$(openssl x509 -noout -fingerprint -sha1 -in $fn | awk -F= '{print $2}')"
+    echo >> ssconf_master_candidates_certs.new "${uuid}=${fp}"
+    done
+```
+
+This generates a server.pem, a set of {nodename}.client.pem files, and a ssconf_master_candidates_certs.new file which you will distribute to the nodes in a later step.
+
+## Updating the /var/lib/ganeti/config.data
+
+The hashes of the certificates need to be corrected in the /var/lib/ganeti/config.data file.  This code needs a copy of the config.data file in your working directory.
+
+```
+#!/usr/bin/env python
+import json
+
+with open("ssconf_master_candidates_certs.new", "r") as f:
+  candidate_certs = dict(line.strip().split("=") for line in f)
+
+with open("config.data", "rb") as f:
+  config = json.load(f)
+
+config["cluster"]["candidate_certs"] = candidate_certs
+
+with open("config.data.new", "wb") as f:
+  json.dump(config, f)
+```
+
+And it generates a file config.data.new to be pushed in the next step.
+
+## Pushing the Files to Nodes
+
+The generated certificates need to be pushed to all nodes, along with the ssconf_master_candidates_certs file and the config.data.
+
+Apologies that this example uses our node names vm[0123] and is run as root.
+
+```
+#!/bin/sh
+
+for i in 0 1 2 3; do 
+
+   rsync vm$i.client.pem vm$i:/var/lib/ganeti/client.pem
+   ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/client.pem
+   ssh vm$i chmod 440 /var/lib/ganeti/client.pem
+					    
+   rsync server.pem vm$i:/var/lib/ganeti
+   ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/server.pem
+   ssh vm$i chmod 440 /var/lib/ganeti/server.pem
+
+   rsync ssconf_master_candidates_certs.new \
+   	 vm$i:/var/lib/ganeti/ssconf_master_candidates_certs
+   ssh vm$i chown root:root /var/lib/ganeti/ssconf_master_candidates_certs
+   ssh vm$i chmod 444 /var/lib/ganeti/ssconf_master_candidates_certs
+
+   rsync config.data.new vm$i:/var/lib/ganeti/config.data
+   ssh vm$i chown gnt-masterd:gnt-confd /var/lib/ganeti/config.data
+   ssh vm$i chmod 640 /var/lib/ganeti/config.data
+
+   done 
+```
+
+## Declare Victory
+
+At this point, we were able to restart ganeti on all nodes. You'll need  to log into each server to run this command, because the master isn't  able to communicate to the nodes yet.
+
+```
+service ganeti restart
+```
+
+and successfully verify the cluster
+
+```
+gnt-cluster verify
+```
+
+We were even able to generate a new generation of credentials. This  command generates a single server.pem that is distributed to all nodes,  as well as a unique per-node client.pem that is distributed to the  appropriate node. ssconf_master_candidates_certs is generated, and  finally the ganeti service is restarted on all nodes.
+
+```
+gnt-cluster renew-crypto --new-cluster-certificate
+```
+
+Oddly, the client.pem objects, presumably generated by Ganeti itself  when we told it to renew-crypto, were X.509v3 CA certificates, with BasicConstraints, AKI, and SKI extensions.  Still self-signed, and no  evidence that they signed anything else.