IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Azure Key Vault CSI on Azure Red Hat OpenShift

Author: Paul Czarkowski Modified: 10/13/2022

This document is adapted from the Azure Key Vault CSI Walkthrough specifically to run with Azure Red Hat OpenShift (ARO).

Prerequisites

An ARO cluster
The AZ CLI (logged in)
Helm 3.x CLI

Environment Variables

Run this command to set some environment variables to use throughout

Note if you created the cluster from the instructions linked above these will re-use the same environment variables, or default them to openshift and eastus.

 export KEYVAULT_RESOURCE_GROUP=${AZR_RESOURCE_GROUP:-"openshift"}
 export KEYVAULT_LOCATION=${AZR_RESOURCE_LOCATION:-"eastus"}
 export KEYVAULT_NAME=secret-store-$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 10 | head -n 1)
 export AZ_TENANT_ID=$(az account show -o tsv --query tenantId)

Installing the Kubernetes Secret Store CSI

Create an OpenShift Project to deploy the CSI into
```
 oc new-project k8s-secrets-store-csi
```

Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

 oc adm policy add-scc-to-user privileged \
   system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver

Add the Secrets Store CSI Driver to your Helm Repositories

 helm repo add secrets-store-csi-driver \
   https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

Update your Helm Repositories
```
 helm repo update
```

Install the secrets store csi driver

 helm install -n k8s-secrets-store-csi csi-secrets-store \
   secrets-store-csi-driver/secrets-store-csi-driver \
   --version v1.0.1 \
   --set "linux.providersDir=/var/run/secrets-store-csi-providers"

Check that the Daemonsets is running

 kubectl --namespace=k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"

You should see the following

 NAME                                               READY   STATUS    RESTARTS   AGE
 csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
 csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s

Deploy Azure Key Store CSI

Add the Azure Helm Repository

 helm repo add csi-secrets-store-provider-azure \
   https://azure.github.io/secrets-store-csi-driver-provider-azure/charts

Update your local Helm Repositories
```
 helm repo update
```

Install the Azure Key Vault CSI provider

 helm install -n k8s-secrets-store-csi azure-csi-provider \
   csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
   --set linux.privileged=true --set secrets-store-csi-driver.install=false \
   --set "linux.providersDir=/var/run/secrets-store-csi-providers" \
   --version=v1.0.1

Set SecurityContextConstraints to allow the CSI driver to run

 oc adm policy add-scc-to-user privileged \
   system:serviceaccount:k8s-secrets-store-csi:csi-secrets-store-provider-azure

Create Keyvault and a Secret

Create a namespace for your application
```
 oc new-project my-application
```

Create an Azure Keyvault in your Resource Group that contains ARO

 az keyvault create -n ${KEYVAULT_NAME} \
   -g ${KEYVAULT_RESOURCE_GROUP} \
   --location ${KEYVAULT_LOCATION}

Create a secret in the Keyvault

 az keyvault secret set \
   --vault-name ${KEYVAULT_NAME} \
   --name secret1 --value "Hello"

Create a Service Principal for the keyvault

Note: If this gives you an error, you may need upgrade your Azure CLI to the latest version.

```bash
export SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --skip-assignment --name http://$KEYVAULT_NAME --query 'password' -otsv)"
export SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name http://$KEYVAULT_NAME --query '[0].appId' -otsv)"
```

Set an Access Policy for the Service Principal

 az keyvault set-policy -n ${KEYVAULT_NAME} \
   --secret-permissions get \
   --spn ${SERVICE_PRINCIPAL_CLIENT_ID}

Create and label a secret for Kubernetes to use to access the Key Vault

 kubectl create secret generic secrets-store-creds \
   -n my-application \
   --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} \
   --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET}
 kubectl -n my-application label secret \
   secrets-store-creds secrets-store.csi.k8s.io/used=true

Deploy an Application that uses the CSI

Create a Secret Provider Class to give access to this secret

 cat <<EOF | kubectl apply -f -
 apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
 kind: SecretProviderClass
 metadata:
   name: azure-kvname
   namespace: my-application
 spec:
   provider: azure
   parameters:
     usePodIdentity: "false"
     useVMManagedIdentity: "false"
     userAssignedIdentityID: ""
     keyvaultName: "${KEYVAULT_NAME}"
     objects: |
       array:
         - |
           objectName: secret1
           objectType: secret
           objectVersion: ""
     tenantId: "${AZ_TENANT_ID}"
 EOF

Create a Pod that uses the above Secret Provider Class

 cat <<EOF | kubectl apply -f -
 kind: Pod
 apiVersion: v1
 metadata:
   name: busybox-secrets-store-inline
   namespace: my-application
 spec:
   containers:
   - name: busybox
     image: k8s.gcr.io/e2e-test-images/busybox:1.29
     command:
       - "/bin/sleep"
       - "10000"
     volumeMounts:
     - name: secrets-store-inline
       mountPath: "/mnt/secrets-store"
       readOnly: true
   volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "azure-kvname"
         nodePublishSecretRef:
           name: secrets-store-creds
 EOF

Check the Secret is mounted

 kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/

Output should match:

 secret1

Print the Secret

 kubectl exec busybox-secrets-store-inline \
   -- cat /mnt/secrets-store/secret1

Output should match:

 Hello

Cleanup

Uninstall Helm

 helm uninstall -n k8s-secrets-store-csi azure-csi-provider

Delete the app
```
 oc delete project my-application
```

Delete the Azure Key Vault

 az keyvault delete -n ${KEYVAULT_NAME}

Delete the Service Principal

 az ad sp delete --id ${SERVICE_PRINCIPAL_CLIENT_ID}

Uninstalling the Kubernetes Secret Store CSI

Delete the secrets store csi driver

 helm delete -n k8s-secrets-store-csi csi-secrets-store

Delete the SecurityContextConstraints

 oc adm policy remove-scc-from-user privileged \
   system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver