IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Installing the HashiCorp Vault Secret CSI Driver

The HashiCorp Vault Secret CSI Driver allows you to access secrets stored in HashiCorp Vault as Kubernetes Volumes.

Prerequisites

  1. An OpenShift Cluster (ROSA, ARO, OSD, and OCP 4.x all work)
  2. kubectl
  3. helm v3

Installing the Kubernetes Secret Store CSI

  1. Create an OpenShift Project to deploy the CSI into

    oc new-project k8s-secrets-store-csi
    
  2. Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

    oc adm policy add-scc-to-user privileged \
      system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
    
  3. Add the Secrets Store CSI Driver to your Helm Repositories

    helm repo add secrets-store-csi-driver \
      https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
    
  4. Update your Helm Repositories

    helm repo update
    
  5. Install the secrets store csi driver

    helm install -n k8s-secrets-store-csi csi-secrets-store \
      secrets-store-csi-driver/secrets-store-csi-driver \
      --version v1.0.1 \
      --set "linux.providersDir=/var/run/secrets-store-csi-providers"
    
  6. Check that the Daemonsets is running

    kubectl --namespace=k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"
    

    You should see the following

    NAME                                               READY   STATUS    RESTARTS   AGE
    csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
    csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s
    

Install HashiCorp Vault with CSI driver enabled

  1. Add the HashiCorp Helm Repository

    helm repo add hashicorp https://helm.releases.hashicorp.com
    
  2. Update your Helm Repositories

    helm repo update
    
  3. Create a namespace for Vault

    oc new-project hashicorp-vault
    
  4. Create a SCC for the CSI driver

    oc adm policy add-scc-to-user privileged \
      system:serviceaccount:hashicorp-vault:vault-csi-provider
    
  5. Create a values file for Helm to use

    cat << EOF > values.yaml
    global:
      openshift: true
    csi:
      enabled: true
      daemonSet:
        providersDir: /var/run/secrets-store-csi-providers
    injector:
      enabled: false
    server:
      image:
        repository: "registry.connect.redhat.com/hashicorp/vault"
        tag: "1.8.0-ubi"
      dev:
        enabled: true
    EOF
    
  6. Install Hashicorp Vault with CSI enabled

    helm install -n hashicorp-vault vault \
      hashicorp/vault --values values.yaml
    
  7. Patch the CSI daemonset

    Currently the CSI has a bug in its manifest which we need to patch

    kubectl patch daemonset vault-csi-provider --type='json' \
        -p='[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": true} }]'
    

Configure Hashicorp Vault

  1. Get a bash prompt inside the Vault pod

    oc exec -it vault-0 -- bash
    
  2. Create a Secret in Vault

    vault kv put secret/db-pass password="hunter2"
    
  3. Configure Vault to use Kubernetes Auth

    vault auth enable kubernetes
    
  4. Check your Cluster’s token issuer

    oc get authentication.config cluster \
      -o json | jq -r .spec.serviceAccountIssuer
    
  5. Configure Kubernetes auth method

    If the issuer here does not match the above, update it.

    vault write auth/kubernetes/config \
    issuer="https://kubernetes.default.svc.cluster.local" \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    
  6. Create a policy for our app

    vault policy write internal-app - <<EOF
    path "secret/data/db-pass" {
      capabilities = ["read"]
    }
    EOF
    
  7. Create an auth role to access it

    vault write auth/kubernetes/role/database \
      bound_service_account_names=webapp-sa \
      bound_service_account_namespaces=default \
      policies=internal-app \
      ttl=20m
    
  8. exit from the vault-0 pod

    exit
    

Deploy a sample application

  1. Create a SecretProviderClass in the default namespace

    cat <<EOF | kubectl apply -f -
    apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
    kind: SecretProviderClass
    metadata:
      name: vault-database
      namespace: default
    spec:
      provider: vault
      parameters:
        vaultAddress: "http://vault.hashicorp-vault:8200"
        roleName: "database"
        objects: |
          - objectName: "db-password"
            secretPath: "secret/data/db-pass"
            secretKey: "password"
    EOF
    
  2. Create a service account webapp-sa

    kubectl create serviceaccount -n default webapp-sa
    
  3. Create a Pod to use the secret

    cat << EOF | kubectl apply -f -
    kind: Pod
    apiVersion: v1
    metadata:
      name: webapp
      namespace: default
    spec:
      serviceAccountName: webapp-sa
      containers:
      - image: jweissig/app:0.0.1
        name: webapp
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "vault-database"
    EOF
    
  4. Check the Pod has the secret

    kubectl -n default exec webapp \
      -- cat /mnt/secrets-store/db-password
    

    The output should match

    hunter2
    

Uninstall HashiCorp Vault with CSI driver enabled

  1. Delete the pod and

    kubectl delete -n default pod webapp
    kubectl delete -n default secretproviderclass vault-database
    kubectl delete -n default serviceaccount webapp-sa
    
  2. Delete the Hashicorp Vault Helm

    helm delete -n hashicorp-vault vault
    
  3. Delete the SCC for Hashicorp Vault

    oc adm policy remove-scc-from-user privileged \
      system:serviceaccount:hashicorp-vault:vault-csi-provider
    
  4. Delete the Hashicorp vault project

    oc delete project hashicorp-vault
    

Uninstalling the Kubernetes Secret Store CSI

  1. Delete the secrets store csi driver

    helm delete -n k8s-secrets-store-csi csi-secrets-store
    
  2. Delete the SecurityContextConstraints

    oc adm policy remove-scc-from-user privileged \
      system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver