Azure Key Vault CSI on Azure Red Hat OpenShift

This document is adapted from the Azure Key Vault CSI Walkthrough specifically to run with Azure Red Hat OpenShift (ARO).

Prerequisites

1. An ARO cluster
2. The AZ CLI (logged in)
3. The OC CLI (logged in)
4. Helm 3.x CLI

Environment Variables

1. Run this command to set some environment variables to use throughout

   Note if you created the cluster from the instructions linked above these will re-use the same environment variables, or default them to openshift and eastus.

   export KEYVAULT_RESOURCE_GROUP=${AZR_RESOURCE_GROUP:-"openshift"}
   export KEYVAULT_LOCATION=${AZR_RESOURCE_LOCATION:-"eastus"}
   export KEYVAULT_NAME=secret-store-$(cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 10 | head -n 1)
   export AZ_TENANT_ID=$(az account show -o tsv --query tenantId)
   export AZ_SUB_ID=$(az account show -o tsv --query id)
   

   Copy

Installing the Kubernetes Secret Store CSI

1. Create an OpenShift Project to deploy the CSI into

   oc new-project k8s-secrets-store-csi
   

   Copy

2. Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
   

   Copy

3. Add the Secrets Store CSI Driver to your Helm Repositories

   helm repo add secrets-store-csi-driver \
     https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   

   Copy

4. Update your Helm Repositories

   helm repo update
   

   Copy

5. Install the secrets store csi driver

   helm install -n k8s-secrets-store-csi csi-secrets-store \
     secrets-store-csi-driver/secrets-store-csi-driver \
     --version v1.3.2 \
     --set "linux.providersDir=/var/run/secrets-store-csi-providers"
   

   Copy

6. Check that the Daemonsets is running

   oc -n k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"
   

   Copy

   You should see the following

   NAME                                               READY   STATUS    RESTARTS   AGE
   csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
   csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s
   

   Copy

7. Add pod security profile label for CSI Driver

   This is required starting in OpenShift v4.13

   oc label csidriver/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
   

   Copy

Deploy Azure Key Store CSI

1. Add the Azure Helm Repository

   helm repo add csi-secrets-store-provider-azure \
     https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
   

   Copy

2. Update your local Helm Repositories

   helm repo update
   

   Copy

3. Install the Azure Key Vault CSI provider

   helm install -n k8s-secrets-store-csi azure-csi-provider \
     csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
     --set linux.privileged=true --set secrets-store-csi-driver.install=false \
     --set "linux.providersDir=/var/run/secrets-store-csi-providers" \
     --version=v1.4.1
   

   Copy

4. Set SecurityContextConstraints to allow the CSI driver to run

   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:csi-secrets-store-provider-azure
   

   Copy

Create Keyvault and a Secret

1. Create a namespace for your application

   oc new-project my-application
   

   Copy

2. Create an Azure Keyvault in your Resource Group that contains ARO

   az keyvault create -n ${KEYVAULT_NAME} \
     -g ${KEYVAULT_RESOURCE_GROUP} \
     --location ${KEYVAULT_LOCATION}
   

   Copy

3. Give your user account permissions to manage secrets in Key Vault

   az role assignment create --role "Key Vault Administrator" \
    --assignee "<your-email-address>" \
    --scope “/subscriptions/$SUB_ID/resourcegroups/$KEYVAULT_RESOURCE_GROUP/providers/microsoft.keyvault/vaults/$KEYVAULT_NAME"
   

   Copy

   Replace <your-email-address> with your actual value, which is your sign-in name.

4. Create a secret in the Keyvault

   az keyvault secret set \
     --vault-name ${KEYVAULT_NAME} \
     --name secret1 --value "Hello"
   

   Copy

5. Create a Service Principal for the key Vault

   Note: If this gives you an error, you may need upgrade your Azure CLI to the latest version.

   export SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac \
     --name http://$KEYVAULT_NAME --query 'password' -otsv)"
   
   export SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list \
     --display-name http://$KEYVAULT_NAME --query '[0].appId' -otsv)"
   

   Copy

6. Give the Service Principal permissions to use secrets in Key Vault

   az role assignment create --role "Key Vault Secrets User" \
     --assignee ${SERVICE_PRINCIPAL_CLIENT_ID} \
     --scope “/subscriptions/$SUB_ID/resourcegroups/$KEYVAULT_RESOURCE_GROUP/providers/microsoft.keyvault/vaults/$KEYVAULT_NAME" 
   

   Copy

7. Create and label a secret for Kubernetes to use to access the Key Vault

   oc create secret generic secrets-store-creds \
     -n my-application \
     --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} \
     --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET}
   
   oc -n my-application label secret \
     secrets-store-creds secrets-store.csi.k8s.io/used=true
   

   Copy

Deploy an Application that uses the CSI

1. Create a Secret Provider Class to give access to this secret

   cat <<EOF | oc apply -f -
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: azure-kvname
     namespace: my-application
   spec:
     provider: azure
     parameters:
       usePodIdentity: "false"
       useVMManagedIdentity: "false"
       userAssignedIdentityID: ""
       keyvaultName: "${KEYVAULT_NAME}"
       objects: |
         array:
           - |
             objectName: secret1
             objectType: secret
             objectVersion: ""
       tenantId: "${AZ_TENANT_ID}"
   EOF
   

   Copy

2. Create a Pod that uses the above Secret Provider Class

   cat <<EOF | oc apply -f -
   kind: Pod
   apiVersion: v1
   metadata:
     name: busybox-secrets-store-inline
     namespace: my-application
   spec:
     containers:
     - name: busybox
       image: k8s.gcr.io/e2e-test-images/busybox:1.29
       command:
         - "/bin/sleep"
         - "10000"
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
     volumes:
       - name: secrets-store-inline
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "azure-kvname"
           nodePublishSecretRef:
             name: secrets-store-creds
   EOF
   

   Copy

3. Check the Secret is mounted

   oc exec busybox-secrets-store-inline -- ls /mnt/secrets-store/
   

   Copy

   Output should match:

   secret1
   

   Copy

4. Print the Secret

   oc exec busybox-secrets-store-inline \
     -- cat /mnt/secrets-store/secret1
   

   Copy

   Output should match:

   Hello
   

   Copy

Cleanup

1. Uninstall Helm

   helm uninstall -n k8s-secrets-store-csi azure-csi-provider
   

   Copy

2. Delete the app

   oc delete project my-application
   

   Copy

3. Delete the Azure Key Vault

   az keyvault delete -n ${KEYVAULT_NAME}
   

   Copy

4. Delete the Service Principal

   az ad sp delete --id ${SERVICE_PRINCIPAL_CLIENT_ID}
   

   Copy

Uninstalling the Kubernetes Secret Store CSI

1. Delete the secrets store csi driver

   helm delete -n k8s-secrets-store-csi csi-secrets-store
   

   Copy

2. Delete the SecurityContextConstraints

   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
   

   Copy