Azure Key Vault CSI on Azure Red Hat OpenShift
This content is authored by Red Hat experts, but has not yet been tested on every supported configuration.
This document is adapted from the Azure Key Vault CSI Walkthrough specifically to run with Azure Red Hat OpenShift (ARO).
Prerequisites
- An ARO cluster
- The AZ CLI (logged in)
- The OC CLI (logged in)
- Helm 3.x CLI
Environment Variables
Run this command to set some environment variables to use throughout
Note if you created the cluster from the instructions linked above these will re-use the same environment variables, or default them to
openshift
andeastus
.export KEYVAULT_RESOURCE_GROUP=${AZR_RESOURCE_GROUP:-"openshift"} export KEYVAULT_LOCATION=${AZR_RESOURCE_LOCATION:-"eastus"} export KEYVAULT_NAME=secret-store-$(cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 10 | head -n 1) export AZ_TENANT_ID=$(az account show -o tsv --query tenantId) export AZ_SUB_ID=$(az account show -o tsv --query id)
Copy
Installing the Kubernetes Secret Store CSI
Create an OpenShift Project to deploy the CSI into
oc new-project k8s-secrets-store-csi
CopySet SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)
oc adm policy add-scc-to-user privileged \ system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
CopyAdd the Secrets Store CSI Driver to your Helm Repositories
helm repo add secrets-store-csi-driver \ https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
CopyUpdate your Helm Repositories
helm repo update
CopyInstall the secrets store csi driver
helm install -n k8s-secrets-store-csi csi-secrets-store \ secrets-store-csi-driver/secrets-store-csi-driver \ --version v1.3.2 \ --set "linux.providersDir=/var/run/secrets-store-csi-providers"
CopyCheck that the Daemonsets is running
oc -n k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"
CopyYou should see the following
NAME READY STATUS RESTARTS AGE csi-secrets-store-secrets-store-csi-driver-cl7dv 3/3 Running 0 57s csi-secrets-store-secrets-store-csi-driver-gbz27 3/3 Running 0 57s
CopyAdd pod security profile label for CSI Driver
This is required starting in OpenShift v4.13 oc label csidriver/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
Copy
Deploy Azure Key Store CSI
Add the Azure Helm Repository
helm repo add csi-secrets-store-provider-azure \ https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
CopyUpdate your local Helm Repositories
helm repo update
CopyInstall the Azure Key Vault CSI provider
helm install -n k8s-secrets-store-csi azure-csi-provider \ csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \ --set linux.privileged=true --set secrets-store-csi-driver.install=false \ --set "linux.providersDir=/var/run/secrets-store-csi-providers" \ --version=v1.4.1
CopySet SecurityContextConstraints to allow the CSI driver to run
oc adm policy add-scc-to-user privileged \ system:serviceaccount:k8s-secrets-store-csi:csi-secrets-store-provider-azure
Copy
Create Keyvault and a Secret
Create a namespace for your application
oc new-project my-application
CopyCreate an Azure Keyvault in your Resource Group that contains ARO
az keyvault create -n ${KEYVAULT_NAME} \ -g ${KEYVAULT_RESOURCE_GROUP} \ --location ${KEYVAULT_LOCATION}
CopyGive your user account permissions to manage secrets in Key Vault
az role assignment create --role "Key Vault Administrator" \ --assignee "<your-email-address>" \ --scope “/subscriptions/$SUB_ID/resourcegroups/$KEYVAULT_RESOURCE_GROUP/providers/microsoft.keyvault/vaults/$KEYVAULT_NAME"
CopyReplace <your-email-address> with your actual value, which is your sign-in name.
Create a secret in the Keyvault
az keyvault secret set \ --vault-name ${KEYVAULT_NAME} \ --name secret1 --value "Hello"
CopyCreate a Service Principal for the key Vault
Note: If this gives you an error, you may need upgrade your Azure CLI to the latest version.
export SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac \ --name http://$KEYVAULT_NAME --query 'password' -otsv)" export SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list \ --display-name http://$KEYVAULT_NAME --query '[0].appId' -otsv)"
CopyGive the Service Principal permissions to use secrets in Key Vault
az role assignment create --role "Key Vault Secrets User" \ --assignee ${SERVICE_PRINCIPAL_CLIENT_ID} \ --scope “/subscriptions/$SUB_ID/resourcegroups/$KEYVAULT_RESOURCE_GROUP/providers/microsoft.keyvault/vaults/$KEYVAULT_NAME"
CopyCreate and label a secret for Kubernetes to use to access the Key Vault
oc create secret generic secrets-store-creds \ -n my-application \ --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} \ --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET} oc -n my-application label secret \ secrets-store-creds secrets-store.csi.k8s.io/used=true
Copy
Deploy an Application that uses the CSI
Create a Secret Provider Class to give access to this secret
cat <<EOF | oc apply -f - apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: azure-kvname namespace: my-application spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "false" userAssignedIdentityID: "" keyvaultName: "${KEYVAULT_NAME}" objects: | array: - | objectName: secret1 objectType: secret objectVersion: "" tenantId: "${AZ_TENANT_ID}" EOF
CopyCreate a Pod that uses the above Secret Provider Class
cat <<EOF | oc apply -f - kind: Pod apiVersion: v1 metadata: name: busybox-secrets-store-inline namespace: my-application spec: containers: - name: busybox image: k8s.gcr.io/e2e-test-images/busybox:1.29 command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-kvname" nodePublishSecretRef: name: secrets-store-creds EOF
CopyCheck the Secret is mounted
oc exec busybox-secrets-store-inline -- ls /mnt/secrets-store/
CopyOutput should match:
secret1
CopyPrint the Secret
oc exec busybox-secrets-store-inline \ -- cat /mnt/secrets-store/secret1
CopyOutput should match:
Hello
Copy
Cleanup
Uninstall Helm
helm uninstall -n k8s-secrets-store-csi azure-csi-provider
CopyDelete the app
oc delete project my-application
CopyDelete the Azure Key Vault
az keyvault delete -n ${KEYVAULT_NAME}
CopyDelete the Service Principal
az ad sp delete --id ${SERVICE_PRINCIPAL_CLIENT_ID}
Copy
Uninstalling the Kubernetes Secret Store CSI
Delete the secrets store csi driver
helm delete -n k8s-secrets-store-csi csi-secrets-store
CopyDelete the SecurityContextConstraints
oc adm policy remove-scc-from-user privileged \ system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
Copy