Installing the HashiCorp Vault Secret CSI Driver

The HashiCorp Vault Secret CSI Driver allows you to access secrets stored in HashiCorp Vault as Kubernetes Volumes.

Prerequisites

1. An OpenShift Cluster (ROSA, ARO, OSD, and OCP 4.x all work)
2. oc
3. helm v3

Installing the Kubernetes Secret Store CSI

1. Create an OpenShift Project to deploy the CSI into

   oc new-project k8s-secrets-store-csi
   

   Copy

2. Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
   

   Copy

3. Add the Secrets Store CSI Driver to your Helm Repositories

   helm repo add secrets-store-csi-driver \
     https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   

   Copy

4. Update your Helm Repositories

   helm repo update
   

   Copy

5. Install the secrets store csi driver

   helm install -n k8s-secrets-store-csi csi-secrets-store \
     secrets-store-csi-driver/secrets-store-csi-driver \
     --version v1.3.2 \
     --set "linux.providersDir=/var/run/secrets-store-csi-providers"
   

   Copy

6. Check that the Daemonsets is running

   oc -n k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"
   

   Copy

   You should see the following

   NAME                                               READY   STATUS    RESTARTS   AGE
   csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
   csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s
   

   Copy

7. Add pod security profile label for CSI Driver

   This is required starting in OpenShift v4.13

   oc label csidriver/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
   

   Copy

Install HashiCorp Vault with CSI driver enabled

1. Add the HashiCorp Helm Repository

   helm repo add hashicorp https://helm.releases.hashicorp.com
   

   Copy

2. Update your Helm Repositories

   helm repo update
   

   Copy

3. Create a namespace for Vault

   oc new-project hashicorp-vault
   

   Copy

4. Create a SCC for the CSI driver

   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:hashicorp-vault:vault-csi-provider
   

   Copy

5. Create a values file for Helm to use

   cat << EOF > values.yaml
   global:
     openshift: true
   csi:
     enabled: true
     daemonSet:
       providersDir: /var/run/secrets-store-csi-providers
   injector:
     enabled: false
   server:
     image:
       repository: "registry.connect.redhat.com/hashicorp/vault"
       tag: "1.8.0-ubi"
     dev:
       enabled: true
   EOF
   

   Copy

6. Install Hashicorp Vault with CSI enabled

   helm install -n hashicorp-vault vault \
     hashicorp/vault --values values.yaml
   

   Copy

7. Patch the CSI daemonset

   Currently the CSI has a bug in its manifest which we need to patch

   oc patch daemonset vault-csi-provider --type='json' \
       -p='[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": true} }]'
   

   Copy

Configure Hashicorp Vault

1. Get a bash prompt inside the Vault pod

   oc exec -it vault-0 -- bash
   

   Copy

2. Create a Secret in Vault

   vault kv put secret/db-pass password="hunter2"
   

   Copy

3. Configure Vault to use Kubernetes Auth

   vault auth enable kubernetes
   

   Copy

4. Check your Cluster’s token issuer in another terminal

   oc get authentication.config cluster \
     -o json | jq -r .spec.serviceAccountIssuer
   

   Copy

5. Configure Kubernetes auth method

   If the issuer here does not match the above, update it.

   vault write auth/kubernetes/config \
   issuer="https://kubernetes.default.svc.cluster.local" \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
   

   Copy

6. Create a policy for our app

   vault policy write internal-app - <<EOF
   path "secret/data/db-pass" {
     capabilities = ["read"]
   }
   EOF
   

   Copy

7. Create an auth role to access it

   vault write auth/kubernetes/role/database \
     bound_service_account_names=webapp-sa \
     bound_service_account_namespaces=default \
     policies=internal-app \
     ttl=20m
   

   Copy

8. exit from the vault-0 pod

   exit
   

   Copy

Deploy a sample application

1. Create a SecretProviderClass in the default namespace

   cat <<EOF | oc apply -f -
   apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
   kind: SecretProviderClass
   metadata:
     name: vault-database
     namespace: default
   spec:
     provider: vault
     parameters:
       vaultAddress: "http://vault.hashicorp-vault:8200"
       roleName: "database"
       objects: |
         - objectName: "db-password"
           secretPath: "secret/data/db-pass"
           secretKey: "password"
   EOF
   

   Copy

2. Create a service account webapp-sa

   oc create serviceaccount -n default webapp-sa
   

   Copy

3. Create a Pod to use the secret

   cat << EOF | oc apply -f -
   kind: Pod
   apiVersion: v1
   metadata:
     name: webapp
     namespace: default
   spec:
     serviceAccountName: webapp-sa
     containers:
     - image: jweissig/app:0.0.1
       name: webapp
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
     volumes:
       - name: secrets-store-inline
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "vault-database"
   EOF
   

   Copy

4. Check the Pod has the secret

   oc -n default exec webapp \
     -- cat /mnt/secrets-store/db-password
   

   Copy

   The output should match

   hunter2
   

   Copy

Uninstall HashiCorp Vault with CSI driver enabled

1. Delete the pod and

   oc delete -n default pod webapp
   oc delete -n default secretproviderclass vault-database
   oc delete -n default serviceaccount webapp-sa
   

   Copy

2. Delete the Hashicorp Vault Helm

   helm delete -n hashicorp-vault vault
   

   Copy

3. Delete the SCC for Hashicorp Vault

   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:hashicorp-vault:vault-csi-provider
   

   Copy

4. Delete the Hashicorp vault project

   oc delete project hashicorp-vault
   

   Copy

Uninstalling the Kubernetes Secret Store CSI

1. Delete the secrets store csi driver

   helm delete -n k8s-secrets-store-csi csi-secrets-store
   

   Copy

2. Delete the SecurityContextConstraints

   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
   

   Copy