# Hacking a New SSL Certificate Structure for Ganeti 2.12.x

Also see [this hack](https://github.com/ganeti/ganeti/issues/1627#issuecomment-994129465).

If you have somehow mashed the the server/client certificates on your cluster, or otherwise gotten into a deep SSL mess, and you, as we did, found the advice at  [GanetiAndSSL](https://code.google.com/p/ganeti/wiki/GanetiAndSSL) did not work for you, then maybe the following will help. We speculate that  the damage to our certs was caused during the upgrade from 2.11 to 2.12.  There's lots of evidence from the mailing list postings that the change in security  model bit a lot of people during the upgrade process.  This recipe was hacked at some cost in source diving and certificate exploring (on a working cluster) by Rob Austein, Hans Kuhn, and Randy Bush.

Be aware that this has only been tested for 2.12 and it's possible the security  model will change in the future. There are no guarantees it will even work for  your case and it is offered in the spirit of shared discovery.

Apologies for the examples assuming the node names of our cluster, vm[0123].

## Generate Server and Client Certificates and ssconf_master_candidates_certs

Despite hints in documentation, conferences, presentations, etc.

- There is no Certificate Authority.
- All certificates are self-signed.
- Client certificates are NOT signed by the server certificate.
- Things break if the CN is not the same for all certificates.
- The client.pem files we found were all ancient X.509v1 certificates with no extensions etc.
- --no-ssl in /etc/default/ganeti didn't allow us to run without SSL

The following script gets a list of {nodename, uuid} and generates certificates and a new ssconf_master_candidates_certs file.

```
#!/bin/sh -

make_cert() {
    local filename=$1 hostname=$2 
    openssl req -batch					\
	    -newkey rsa:2048 -keyout $filename -nodes	\
	    -x509	     -out    $filename		\
	    -sha1					\
	    -days   1825				\
	    -subj   "/CN=$hostname"
    }

make_cert server.pem ganeti.example.com

rm -f ssconf_master_candidates_certs.new

# the following use of gnt-node assumes a semi-working cluster if not,
# you need to hack a file of the form {hostname uuid} for all hosts.
# this can be rescued from the old ssconf_master_candidates_certs
#cat nodehack |
gnt-node list -o name,uuid --no-headers |
while read hn uuid
do
    fn=${hn%%.*}.client.pem
    make_cert $fn ganeti.example.com
    fp="$(openssl x509 -noout -fingerprint -sha1 -in $fn | awk -F= '{print $2}')"
    echo >> ssconf_master_candidates_certs.new "${uuid}=${fp}"
    done
```

This generates a server.pem, a set of {nodename}.client.pem files, and a ssconf_master_candidates_certs.new file which you will distribute to the nodes in a later step.

## Updating the /var/lib/ganeti/config.data

The hashes of the certificates need to be corrected in the /var/lib/ganeti/config.data file.  This code needs a copy of the config.data file in your working directory.

```
#!/usr/bin/env python
import json

with open("ssconf_master_candidates_certs.new", "r") as f:
  candidate_certs = dict(line.strip().split("=") for line in f)

with open("config.data", "rb") as f:
  config = json.load(f)

config["cluster"]["candidate_certs"] = candidate_certs

with open("config.data.new", "wb") as f:
  json.dump(config, f)
```

And it generates a file config.data.new to be pushed in the next step.

## Pushing the Files to Nodes

The generated certificates need to be pushed to all nodes, along with the ssconf_master_candidates_certs file and the config.data.

Apologies that this example uses our node names vm[0123] and is run as root.

```
#!/bin/sh

for i in 0 1 2 3; do 

   rsync vm$i.client.pem vm$i:/var/lib/ganeti/client.pem
   ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/client.pem
   ssh vm$i chmod 440 /var/lib/ganeti/client.pem
					    
   rsync server.pem vm$i:/var/lib/ganeti
   ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/server.pem
   ssh vm$i chmod 440 /var/lib/ganeti/server.pem

   rsync ssconf_master_candidates_certs.new \
   	 vm$i:/var/lib/ganeti/ssconf_master_candidates_certs
   ssh vm$i chown root:root /var/lib/ganeti/ssconf_master_candidates_certs
   ssh vm$i chmod 444 /var/lib/ganeti/ssconf_master_candidates_certs

   rsync config.data.new vm$i:/var/lib/ganeti/config.data
   ssh vm$i chown gnt-masterd:gnt-confd /var/lib/ganeti/config.data
   ssh vm$i chmod 640 /var/lib/ganeti/config.data

   done 
```

## Declare Victory

At this point, we were able to restart ganeti on all nodes. You'll need  to log into each server to run this command, because the master isn't  able to communicate to the nodes yet.

```
service ganeti restart
```

and successfully verify the cluster

```
gnt-cluster verify
```

We were even able to generate a new generation of credentials. This  command generates a single server.pem that is distributed to all nodes,  as well as a unique per-node client.pem that is distributed to the  appropriate node. ssconf_master_candidates_certs is generated, and  finally the ganeti service is restarted on all nodes.

```
gnt-cluster renew-crypto --new-cluster-certificate
```

Oddly, the client.pem objects, presumably generated by Ganeti itself  when we told it to renew-crypto, were X.509v3 CA certificates, with BasicConstraints, AKI, and SKI extensions.  Still self-signed, and no  evidence that they signed anything else.

---
2023.06.30