IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Azure Key Vault CSI on Azure Red Hat OpenShift

Authors: Paul Czarkowski, Diana Sari
Last Editor: Dustin Scott
Published Date: 16 August 2021
Modified Date: 25 May 2023

This document is adapted from the Azure Key Vault CSI Walkthrough specifically to run with Azure Red Hat OpenShift (ARO).

Prerequisites

  1. An ARO cluster
  2. The AZ CLI (logged in)
  3. The OC CLI (logged in)
  4. Helm 3.x CLI

Environment Variables

  1. Run this command to set some environment variables to use throughout

    Note if you created the cluster from the instructions linked above these will re-use the same environment variables, or default them to openshift and eastus.

    export KEYVAULT_RESOURCE_GROUP=${AZR_RESOURCE_GROUP:-"openshift"}
export KEYVAULT_LOCATION=${AZR_RESOURCE_LOCATION:-"eastus"}
export KEYVAULT_NAME=secret-store-$(cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 10 | head -n 1)
export AZ_TENANT_ID=$(az account show -o tsv --query tenantId)

Installing the Kubernetes Secret Store CSI

  1. Create an OpenShift Project to deploy the CSI into

    oc new-project k8s-secrets-store-csi

  2. Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

    oc adm policy add-scc-to-user privileged \
  system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver

  3. Add the Secrets Store CSI Driver to your Helm Repositories

    helm repo add secrets-store-csi-driver \
  https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

  4. Update your Helm Repositories

    helm repo update

  5. Install the secrets store csi driver

    helm install -n k8s-secrets-store-csi csi-secrets-store \
  secrets-store-csi-driver/secrets-store-csi-driver \
  --version v1.3.2 \
  --set "linux.providersDir=/var/run/secrets-store-csi-providers"

  6. Check that the Daemonsets is running

    oc -n k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"

    You should see the following

    NAME                                               READY   STATUS    RESTARTS   AGE
csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s

Deploy Azure Key Store CSI

  1. Add the Azure Helm Repository

    helm repo add csi-secrets-store-provider-azure \
  https://azure.github.io/secrets-store-csi-driver-provider-azure/charts

  2. Update your local Helm Repositories

    helm repo update

  3. Install the Azure Key Vault CSI provider

    helm install -n k8s-secrets-store-csi azure-csi-provider \
  csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
  --set linux.privileged=true --set secrets-store-csi-driver.install=false \
  --set "linux.providersDir=/var/run/secrets-store-csi-providers" \
  --version=v1.4.1

  4. Set SecurityContextConstraints to allow the CSI driver to run

    oc adm policy add-scc-to-user privileged \
  system:serviceaccount:k8s-secrets-store-csi:csi-secrets-store-provider-azure

Create Keyvault and a Secret

  1. Create a namespace for your application

    oc new-project my-application

  2. Create an Azure Keyvault in your Resource Group that contains ARO

    az keyvault create -n ${KEYVAULT_NAME} \
  -g ${KEYVAULT_RESOURCE_GROUP} \
  --location ${KEYVAULT_LOCATION}

  3. Create a secret in the Keyvault

    az keyvault secret set \
  --vault-name ${KEYVAULT_NAME} \
  --name secret1 --value "Hello"

  4. Create a Service Principal for the keyvault

    Note: If this gives you an error, you may need upgrade your Azure CLI to the latest version.

    export SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac \
  --name http://$KEYVAULT_NAME --query 'password' -otsv)"

export SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list \
  --display-name http://$KEYVAULT_NAME --query '[0].appId' -otsv)"

  5. Set an Access Policy for the Service Principal

    az keyvault set-policy -n ${KEYVAULT_NAME} \
  --secret-permissions get \
  --spn ${SERVICE_PRINCIPAL_CLIENT_ID}

  6. Create and label a secret for Kubernetes to use to access the Key Vault

    oc create secret generic secrets-store-creds \
  -n my-application \
  --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} \
  --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET}

oc -n my-application label secret \
  secrets-store-creds secrets-store.csi.k8s.io/used=true

Deploy an Application that uses the CSI

  1. Create a Secret Provider Class to give access to this secret

    cat <<EOF | oc apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
  namespace: my-application
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: "${KEYVAULT_NAME}"
    objects: |
      array:
        - |
          objectName: secret1
          objectType: secret
          objectVersion: ""
    tenantId: "${AZ_TENANT_ID}"
EOF

  2. Create a Pod that uses the above Secret Provider Class

    cat <<EOF | oc apply -f -
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline
  namespace: my-application
spec:
  containers:
  - name: busybox
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
      - "/bin/sleep"
      - "10000"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname"
        nodePublishSecretRef:
          name: secrets-store-creds
EOF

  3. Check the Secret is mounted

    oc exec busybox-secrets-store-inline -- ls /mnt/secrets-store/

    Output should match:

    secret1

  4. Print the Secret

    oc exec busybox-secrets-store-inline \
  -- cat /mnt/secrets-store/secret1

    Output should match:

    Hello

Cleanup

  1. Uninstall Helm

    helm uninstall -n k8s-secrets-store-csi azure-csi-provider

  2. Delete the app

    oc delete project my-application

  3. Delete the Azure Key Vault

    az keyvault delete -n ${KEYVAULT_NAME}

  4. Delete the Service Principal

    az ad sp delete --id ${SERVICE_PRINCIPAL_CLIENT_ID}

Uninstalling the Kubernetes Secret Store CSI

  1. Delete the secrets store csi driver

    helm delete -n k8s-secrets-store-csi csi-secrets-store

  2. Delete the SecurityContextConstraints

    oc adm policy remove-scc-from-user privileged \
  system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver