Generating Let’s Encrypt certificates with Red Hat OpenShift cert-manager operator using the Cloudflare DNS solver

Configuration, System Design

Let’s Encrypt is a nonprofit certificate authority that provides free SSL/TLS certificates, which are essential for enabling secure HTTPS connections on websites. Launched in 2016 by the Internet Security Research Group (ISRG), its mission is to make encrypted connections ubiquitous on the web, enhancing privacy and security for all users. Let’s Encrypt simplifies the process of obtaining and renewing certificates through automation, making it accessible even for those without extensive technical knowledge. By offering a straightforward, cost-free way to secure websites, Let’s Encrypt plays a crucial role in promoting a safer and more privacy-respecting internet.

Focus

For the focus of this discussion, we want to be able to create and manage certificates in an automated way. To be even more blunt – we do not want anyone, especially developers, to be concerned with certificate management. The acts of certificate management is one of the clearest ways a platform engineering mindset can eliminate both technical toil but also reduce cognitive load on the organization. For our OpenShift clusters, there are typically two different types of certificates we want to manage:

  1. Cluster certificates – these are going to be certificates for the cluster ingress, which is a wildcard certificate, as well as certificates for the api endpoints.
  2. Application certificates – for each application, there may be a need for specific application certificates, especially if the cluster serves both cluster DNS requests (test.apps.ocp.example.com) but also logical DNS (test.dev.example.com).

Operator Installation

To get the party started, let’s install the OpenShift cert-manager operator.

apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cert-manager-operator-operator-group
  namespace: cert-manager-operator
spec:
  targetNamespaces:
    - cert-manager-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator-subscription
  namespace: cert-manager-operator
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Creating a ClusterIssuer

Once the operator is installed, we will now want to configure a ClusterIssuer. In our case, we are going to use Let’s Encrypt as the authority to issue our certificates and we are going to use the acme protocol to manage the lifecycle. However, one of our use cases is going to be managing the wildcard cluster ingress certificate. As part of the acme protocol, we have to be able to prove we own the domain for which we are wanting certificates issued. There are two ways to do this: HTTP or DNS. The HTTP solver allows verification because we stand up a web server to respond to the requests and therefore, we would have to own the DNS to get it to resolve correctly. But the HTTP solver doesn’t work for wildcard certs, so we need to use the DNS solver which will place a TEXT DNS entry temporarily in our domain and then query DNS for the entry.

To achieve this, you will have to use a DNS solver plugin. In our case, we are using Cloudflare for DNS so we will use the Cloudflare DNS solver plugin. But we have to provide API access to the ClusterIssuer to be able to update our DNS, so to accomplish this, you will need to go into your Cloudflare setup and generate an API token with the permissions to update that zone’s DNS records. Once you have that token, we will need to create a secret for it and place it in the cert-manager namespace.

oc create secret generic cloudflare-api-token-secret \
  -n cert-manager \
  --from-literal=api-token=<token>

Now that we have our secret, we can create our ClusterIssuer. The resource is not namespaced, as it applies to the entire cluster. 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
spec:
  acme:
    server: 'https://acme-staging-v02.api.letsencrypt.org/directory'
    privateKeySecretRef:
      name: acme-account-private-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              key: api-token
              name: cloudflare-api-token-secret

Wildcard Ingress Certificate

With our ClusterIssuer in place, we can now request a wildcard certificate for our cluster ingress.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: apps-ocp-lab-snimmo-com-certificate
  namespace: openshift-ingress
spec:
  commonName: apps.ocp.lab.snimmo.com
  dnsNames:
    - "apps.ocp.lab.snimmo.com" 
    - "*.apps.ocp.lab.snimmo.com"
  secretName: apps-ocp-lab-snimmo-com-tls
  isCA: false
  issuerRef:
    group: cert-manager.io
    name: letsencrypt-cluster-issuer
    kind: ClusterIssuer

Here are some notes about the resource:

  • Naming: your organization can setup some fairly easy patterns here to manage the consistency of naming by simply replacing the FQDN’s periods with dashes and then appending some type information.
  • Namespacing: We want the certificate to be put into a secret in the openshift-ingress namespace because that is where the resources will be updated to refer to the secret.

Next we need to download and add the Let’s Encrypt STAGING root CA to the cluster.

curl https://raw.githubusercontent.com/letsencrypt/website/main/static/certs/staging/letsencrypt-stg-root-x1.pem > /home/snimmo/Downloads/letsencryptrootca.pem

oc create configmap custom-ca \
  --from-file=ca-bundle.crt=/home/snimmo/Downloads/letsencryptrootca.pem \
  -n openshift-config

oc patch proxy/cluster --type=merge \
  --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

Once the certificate is issued, we can then update the default ingress controller to use the wildcard tls we just created. 

oc patch ingresscontroller.operator default \
     --type=merge -p \
     '{"spec":{"defaultCertificate": {"name": "apps-ocp-lab-snimmo-com-tls"}}}' \
     -n openshift-ingress-operator

API Server Certificate

We also want to replace the certificate for API access to the OpenShift cluster. This certificate is located in the openshift-config namespace so our Certificate manifest goes there.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-ocp-lab-snimmo-com-certificate
  namespace: openshift-config
spec:
  commonName: api.ocp.lab.snimmo.com
  dnsNames:
    - "api.ocp.lab.snimmo.com" 
  secretName: api-ocp-lab-snimmo-com-tls
  isCA: false
  issuerRef:
    group: cert-manager.io
    name: letsencrypt-cluster-issuer
    kind: ClusterIssuer

After applying the manifest, you can watch the Certificate request progress using the following command. 

oc get certificate -w -n openshift-config

Once the certificate is “Ready”, then we can apply the patch to the apiserver to swap out the certificates. 

oc patch apiserver cluster --type=merge \
  -p '{"spec":{"servingCerts": {"namedCertificates": [{"names": ["api.ocp.lab.snimmo.com"], "servingCertificate": {"name": "api-ocp-lab-snimmo-com-tls"}}]}}}'

Application Certificates

The cert-manager operator allows you to use annotations to produce the certificates. However, we need to take a small extra step to configure the cluster to enable this functionality on OpenShift Route objects. The documentation for this install is located at https://github.com/cert-manager/openshift-routes?tab=readme-ov-file#installation

oc apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml

Once this component is installed, we can utilize the existing ClusterIssuer to manage the certificates by simply utilizing some annotations on the route. 

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: busybox
  namespace: default
  annotations:
    cert-manager.io/common-name: apps.ocp.lab.snimmo.com
    cert-manager.io/issuer-name: letsencrypt-cluster-issuer
    cert-manager.io/issuer-kind: ClusterIssuer
spec:
  host: busybox-default.apps.ocp.lab.snimmo.com
  to:
    kind: Service
    name: busybox
    weight: 100
  port:
    targetPort: 8080
  tls:
    termination: edge
  wildcardPolicy: None

When this is applied, it is automatically updated to include all the correct certificate information. 

snimmo@snimmo-mac notes % oc get route busybox -o yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
    cert-manager.io/certificate-revision: "1"
    cert-manager.io/common-name: apps.ocp.lab.snimmo.com
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: letsencrypt-cluster-issuer
  creationTimestamp: "2024-05-16T16:54:59Z"
  name: busybox
  namespace: default
  resourceVersion: "10695190"
  uid: bcbb9a0c-ab4d-4f70-9664-cfd85a52a4a9
spec:
  host: busybox-default.apps.ocp.lab.snimmo.com
  port:
    targetPort: 8080
  tls:
    certificate: |
      -----BEGIN CERTIFICATE-----
      MIIFIzCCBAugAwIBAgISAxfYjt9t38FE6NX5rcVLtnTEMA0GCSqGSIb3DQEBCwUA
      ...
      SxAs2GWpP38NvIU2zmH0weUYhFoOxRI=
      -----END CERTIFICATE-----
      -----BEGIN CERTIFICATE-----
      MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
      ...
      MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
      nLRbwHOoq7hHwg==
      -----END CERTIFICATE-----
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAvpOTJY0AII6wH6qgsiRNFDHd9nGYDFyphmUUzZv/Xlz+0JFz
      ...
      KkX9cMbCVLj1H/F2X8dt1Zt7ZR9q8wF/wdjQQo7RWTE9uITTXegQ
      -----END RSA PRIVATE KEY-----
    termination: edge
  to:
    kind: Service
    name: busybox
    weight: 100
  wildcardPolicy: None
status:
  ingress:
  - conditions:
    - lastTransitionTime: "2024-05-16T16:54:59Z"
      status: "True"
      type: Admitted
    host: busybox-default.apps.ocp.lab.snimmo.com
    routerCanonicalHostname: router-default.apps.ocp.lab.snimmo.com
    routerName: default
    wildcardPolicy: None