I’m hosting an OpenShift cluster on a set of 4 Intel NUCs in the home office. I wanted to start integrating with external services, such as GitHub webhooks, but before I could do that I needed to get both DNS setup to resolve to my home network the correct way as well as secure the connections.
For the purposes of this writeup, the domain will be an example: mydomain.com
Let’s start with DNS. I work at home and although my IP address stays fairly stable, I went ahead and used a DDNS service to set everything up to auto update when an IP changes. Because it would happen and it would happen 6 minutes before I was presenting to some very important customer.
I use Google Domains to host my domain. Easy to use and supports everything I need. First thing is to create an account at a DDNS provider. I used https://www.dynu.com, and it was easy to setup. Follow the instructions at https://www.dynu.com/Resources/Tutorials/DynamicDNS/SettingUpWebsite but when you are port forwarding through your router, make sure to do it for port 443 as well. For my use case, I have an HAProxy running on RHEL 8.2 which frontends my OpenShift cluster so my port forwarding was to the static IP for that box.
Once you have the DDNS url setup and working, you can then update your Google domains to add a CNAME which points to the DDNS url. My cluster is running at *.apps.ocp.mydomain.com so we will want the CNAME to point there.
*.ocp CNAME 1h myurl.loseyourip.com.Notice the entry is just *.apps.ocp, NOT *.apps.ocp.mydomain.com. Also notice the period at the end of the url on the CNAME data field.
Stay logged in to your domain management UI for a tick because you’ll need it in the next step.
Let’s Encrypt FTW!
I love Let’s Encrypt mostly because I hate companies that try and force you to pay them to do simple things (looking at you, certficate authorities). They support wildcard certs and it’s very easy to do this now so there is NO EXCUSE to run anything without TLS.
On my RHEL box, I installed certbot, which handles all of the hard work for you. The most difficult part is that you have to assert to them that you own the domain in which you are trying to get a cert for. I used the most manual way, DNS.
Here’s the setup.
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo yum update
sudo yum install epel-release
sudo yum install certbot
certbot -d '*.apps.ocp.mydomain.com' --manual --preferred-challenges dns certonlyWhen you run the command, you’ll get the following. Leave it there. Don’t press enter.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Requesting a certificate for *.apps.ocp.snimmo.com
Performing the following challenges:
dns-01 challenge for apps.ocp.snimmo.com
Please deploy a DNS TXT record under the name
_acme-challenge.apps.ocp.snimmo.com with the following value:
******-******-******
Before continuing, verify the record is deployed.
Press Enter to ContinueGo back to your domain and add a TXT record for value above.
_acme-challenge.apps.ocp TXT 1h ******Then wait a couple of minutes. It’s DNS. It’s got to propagate. You can check the status of the propagation at https://dnschecker.org/#TXT/_acme-challenge.ocp.yourdomain.com
Once you see some of them turn green, you *should* be good to go. Go back to your bash shell and hit enter. You’ll see it confirm.
Waiting for verification…
Cleaning up challenges
IMPORTANT NOTES:
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/apps.ocp.snimmo.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/apps.ocp.snimmo.com/privkey.pem
Your certificate will expire on 2021-08-15. To obtain a new or
tweaked version of this certificate in the future, simply run
certbot again. To non-interactively renew all of your
certificates, run "certbot renew"
If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le Don’t forget to donate. It’s $5. You spend more on coffee.
OpenShift IngressController Update
Now it’s as simple as updating the IngressController with the new cert. Reference docs: https://docs.openshift.com/container-platform/4.6/security/certificates/replacing-default-ingress-certificate.html
From my RHEL box, after logging in using oc as an admin, run the following. Notice the dates as an added measure because we will need to update the cert regularly.
oc create configmap letsencrypt-ca-05172021 --from-file=ca-bundle.crt=/etc/letsencrypt/live/ocp.snimmo.com/fullchain.pem -n openshift-config
oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"letsencrypt-ca-05172021"}}}'
oc create secret tls letsencrypt-ca-secret-05172021 --cert=/etc/letsencrypt/live/apps.ocp.snimmo.com/fullchain.pem --key=/etc/letsencrypt/live/apps.ocp.snimmo.com/privkey.pem -n openshift-ingress
oc patch ingresscontroller.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "letsencrypt-ca-secret-05172021"}}}' -n openshift-ingress-operatorWhen you execute this, it’s going to basically refresh everything you can think of in your cluster. It’s almost the same amount of churn you see when doing an OpenShift cluster version update. It took my cluster about 10 minutes to get everything done. So make sure to do it during appropriate times.