Hashicorp Vault setup on OpenShift using ArgoCD

Configuration

Tools and Products:

I’m running a couple of clusters in my lab environment. One of them is a “hub” cluster which is handling all of the core, centralized services which shouldn’t be considered workloads. As part of the hub cluster setup, I wanted to have a secure KMS and there is no better choice than Hashicorp Vault. Vault has a great Helm chart for setup. Combining this chart with an ArgoCD Application CR gives you mostly everything you need. 

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hashicorp-vault-application
  namespace: openshift-gitops
spec:
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: vault
  source:
    repoURL: 'https://helm.releases.hashicorp.com'
    targetRevision: 0.28.0
    chart: vault
    helm:
      releaseName: vault
      parameters:
        - name: 'global.openshift'
          value: 'true'
        - name: 'server.ha.enabled'
          value: 'true'
        - name: 'server.ha.raft.enabled'
          value: 'true'
        - name: 'server.route.enabled'
          value: 'true'
        - name: 'server.route.host'
          value: 'vault.apps.hub.ocp.lab.snimmo.com'
        - name: 'server.route.tls.termination'
          value: 'edge'
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:
        - .webhooks[]?.clientConfig.caBundle

Notes:

  • Most of the parameters are related to running Vault on OpenShift due to OpenShift’s outstanding default security policies.

Init and Unsealing

Once the CR is applied to the cluster, the vault pods will need to be initialized and unsealed. By default, there are 3 vault pods that means that all three will need to be unsealed using the oc cli. 

oc exec -ti vault-0 -- vault operator init

Unseal Key 1: <key>
Unseal Key 2: <key>
Unseal Key 3: <key>
Unseal Key 4: <key>
Unseal Key 5: <key>

Initial Root Token: <root-token>

After you run the init command, it will print out the sealing keys and root. Put those someplace safe. Using those sealing keys, you will then go and unseal each pod and then connect the pods together. Run the unseal command below THREE TIMES using three of the unseal keys provided to you. 

oc exec -ti vault-0 -- vault operator unseal

Once the initial pod is unsealed, you will then need to go and unseal the other pods and join them with the initial vault pod. Again, you need to run the unseal command THREE TIMES for each pod. 

oc exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
oc exec -ti vault-1 -- vault operator unseal 

oc exec -ti vault-2 -- vault operator raft join http://vault-0.vault-internal:8200
oc exec -ti vault-2 -- vault operator unseal

After everything is unsealed, you will be able to login using the UI.

https://vault.apps.hub.ocp.lab.snimmo.com