randy/rgnet-wiki
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

from wiki

Browse source
This commit is contained in:
Randy Bush 2020-07-04 15:15:34 -07:00
parent 0fab894714
commit a8ffa2adf3
1 changed files with 128 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

128
pages/RegenCertsEtc.md Normal file
View file

@ -0,0 +1,128 @@
# Hacking a New SSL Certificate Structure for Ganeti 2.12.x
If you have somehow mashed the the server/client certificates on your cluster, or otherwise gotten into a deep SSL mess, and you, as we did, found the advice at [GanetiAndSSL](https://code.google.com/p/ganeti/wiki/GanetiAndSSL) did not work for you, then maybe the following will help. We speculate that the damage to our certs was caused during the upgrade from 2.11 to 2.12. There's lots of evidence from the mailing list postings that the change in security model bit a lot of people during the upgrade process. This recipe was hacked at some cost in source diving and certificate exploring (on a working cluster) by Rob Austein, Hans Kuhn, and Randy Bush.
Be aware that this has only been tested for 2.12 and it's possible the security model will change in the future. There are no guarantees it will even work for your case and it is offered in the spirit of shared discovery.
Apologies for the examples assuming the node names of our cluster, vm[0123].
## Generate Server and Client Certificates and ssconf_master_candidates_certs
Despite hints in documentation, conferences, presentations, etc.
- There is no Certificate Authority.
- All certificates are self-signed.
- Client certificates are NOT signed by the server certificate.
- Things break if the CN is not the same for all certificates.
- The client.pem files we found were all ancient X.509v1 certificates with no extensions etc.
- --no-ssl in /etc/default/ganeti didn't allow us to run without SSL
The following script gets a list of {nodename, uuid} and generates certificates and a new ssconf_master_candidates_certs file.
```
#!/bin/sh -
make_cert() {
local filename=$1 hostname=$2
openssl req -batch \
-newkey rsa:2048 -keyout $filename -nodes \
-x509 -out $filename \
-sha1 \
-days 1825 \
-subj "/CN=$hostname"
}
make_cert server.pem ganeti.example.com
rm -f ssconf_master_candidates_certs.new
# the following use of gnt-node assumes a semi-working cluster if not,
# you need to hack a file of the form {hostname uuid} for all hosts.
# this can be rescued from the old ssconf_master_candidates_certs
#cat nodehack |
gnt-node list -o name,uuid --no-headers |
while read hn uuid
do
fn=${hn%%.*}.client.pem
make_cert $fn ganeti.example.com
fp="$(openssl x509 -noout -fingerprint -sha1 -in $fn | awk -F= '{print $2}')"
echo >> ssconf_master_candidates_certs.new "${uuid}=${fp}"
done
```
This generates a server.pem, a set of {nodename}.client.pem files, and a ssconf_master_candidates_certs.new file which you will distribute to the nodes in a later step.
## Updating the /var/lib/ganeti/config.data
The hashes of the certificates need to be corrected in the /var/lib/ganeti/config.data file. This code needs a copy of the config.data file in your working directory.
```
#!/usr/bin/env python
import json
with open("ssconf_master_candidates_certs.new", "r") as f:
candidate_certs = dict(line.strip().split("=") for line in f)
with open("config.data", "rb") as f:
config = json.load(f)
config["cluster"]["candidate_certs"] = candidate_certs
with open("config.data.new", "wb") as f:
json.dump(config, f)
```
And it generates a file config.data.new to be pushed in the next step.
## Pushing the Files to Nodes
The generated certificates need to be pushed to all nodes, along with the ssconf_master_candidates_certs file and the config.data.
Apologies that this example uses our node names vm[0123] and is run as root.
```
#!/bin/sh
for i in 0 1 2 3; do
rsync vm$i.client.pem vm$i:/var/lib/ganeti/client.pem
ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/client.pem
ssh vm$i chmod 440 /var/lib/ganeti/client.pem
rsync server.pem vm$i:/var/lib/ganeti
ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/server.pem
ssh vm$i chmod 440 /var/lib/ganeti/server.pem
rsync ssconf_master_candidates_certs.new \
vm$i:/var/lib/ganeti/ssconf_master_candidates_certs
ssh vm$i chown root:root /var/lib/ganeti/ssconf_master_candidates_certs
ssh vm$i chmod 444 /var/lib/ganeti/ssconf_master_candidates_certs
rsync config.data.new vm$i:/var/lib/ganeti/config.data
ssh vm$i chown gnt-masterd:gnt-confd /var/lib/ganeti/config.data
ssh vm$i chmod 640 /var/lib/ganeti/config.data
done
```
## Declare Victory
At this point, we were able to restart ganeti on all nodes. You'll need to log into each server to run this command, because the master isn't able to communicate to the nodes yet.
```
service ganeti restart
```
and successfully verify the cluster
```
gnt-cluster verify
```
We were even able to generate a new generation of credentials. This command generates a single server.pem that is distributed to all nodes, as well as a unique per-node client.pem that is distributed to the appropriate node. ssconf_master_candidates_certs is generated, and finally the ganeti service is restarted on all nodes.
```
gnt-cluster renew-crypto --new-cluster-certificate
```
Oddly, the client.pem objects, presumably generated by Ganeti itself when we told it to renew-crypto, were X.509v3 CA certificates, with BasicConstraints, AKI, and SKI extensions. Still self-signed, and no evidence that they signed anything else.