Home GitHub

Disclaimer: Mobb.ninja is not official Red Hat documentation - These guides may be experimental, proof of concept or early adoption. Officially supported documentation is available at https://docs.openshift.com.

Azure Key Vault CSI on Azure Red Hat OpenShift

Author: Paul Czarkowski Modified: 08/16/2021

This document is adapted from the Azure Key Vault CSI Walkthrough specifically to run with Azure Red Hat OpenShift (ARO).

Prerequisites

  1. An ARO cluster
  2. The AZ CLI (logged in)
  3. Helm 3.x CLI

Environment Variables

  1. Run this command to set some environment variables to use throughout

    Note if you created the cluster from the instructions linked above these will re-use the same environment variables, or default them to openshift and eastus.

     export KEYVAULT_RESOURCE_GROUP=${AZR_RESOURCE_GROUP:-"openshift"}
     export KEYVAULT_LOCATION=${AZR_RESOURCE_LOCATION:-"eastus"}
     export KEYVAULT_NAME=secret-store-$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 10 | head -n 1)
     export AZ_TENANT_ID=$(az account show -o tsv --query tenantId)
    

Installing the Kubernetes Secret Store CSI

  1. Create an OpenShift Project to deploy the CSI into

     oc new-project k8s-secrets-store-csi
    
  2. Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

     oc adm policy add-scc-to-user privileged \
       system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
    
  3. Add the Secrets Store CSI Driver to your Helm Repositories

     helm repo add secrets-store-csi-driver \
       https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts
    
  4. Update your Helm Repositories

     helm repo update
    
  5. Install the secrets store csi driver

     helm install -n k8s-secrets-store-csi csi-secrets-store \
       secrets-store-csi-driver/secrets-store-csi-driver
    
  6. Check that the Daemonsets is running

     kubectl --namespace=k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"
    

    You should see the following

     NAME                                               READY   STATUS    RESTARTS   AGE
     csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
     csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s
    

Deploy Azure Key Store CSI

  1. Add the Azure Helm Repository

     helm repo add csi-secrets-store-provider-azure \
       https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
    
  2. Update your local Helm Repositories

     helm repo update
    
  3. Install the Azure Key Vault CSI provider

     helm install -n k8s-secrets-store-csi azure-csi-provider \
       csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
       --set linux.privileged=true --set secrets-store-csi-driver.install=false
    
  4. Set SecurityContextConstraints to allow the CSI driver to run

     oc adm policy add-scc-to-user privileged \
       system:serviceaccount:k8s-secrets-store-csi:csi-secrets-store-provider-azure
    

Create Keyvault and a Secret

  1. Create a namespace for your application

    This service principal will be used by your application

     oc new-project my-application
    
  2. Create an Azure Keyvault in your Resource Group that contains ARO

     az keyvault create -n ${KEYVAULT_NAME} \
       -g ${KEYVAULT_RESOURCE_GROUP} \
       --location ${KEYVAULT_LOCATION}
    
  3. Create a secret in the Keyvault

     az keyvault secret set \
       --vault-name ${KEYVAULT_NAME} \
       --name secret1 --value "Hello"
    
  4. Create a Service Principal for the keyvault

     export SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --skip-assignment --name http://secrets-store-test --query 'password' -otsv)"
     export SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp show --id http://secrets-store-test --query 'appId' -otsv)"
    
  5. Set an Access Policy for the Service Principal

     az keyvault set-policy -n ${KEYVAULT_NAME} \
       --secret-permissions get \
       --spn ${SERVICE_PRINCIPAL_CLIENT_ID}
    
  6. Create and label a secret for Kubernetes to use to access the Key Vault

     kubectl create secret generic secrets-store-creds \
       -n my-application \
       --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} \
       --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET}
     kubectl -n my-application label secret \
       secrets-store-creds secrets-store.csi.k8s.io/used=true
    

Deploy an Application that uses the CSI

  1. Create a Secret Provider Class to give access to this secret

    cat <<EOF | kubectl apply -f -
    apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
    kind: SecretProviderClass
    metadata:
      name: azure-kvname
      namespace: my-application
    spec:
      provider: azure
      parameters:
     usePodIdentity: "false"
     useVMManagedIdentity: "false"
     userAssignedIdentityID: ""
     keyvaultName: "${KEYVAULT_NAME}"
     objects: |
       array:
         - |
           objectName: secret1
           objectType: secret
           objectVersion: ""
     tenantId: "${AZ_TENANT_ID}"
    EOF
    
  2. Create a Pod that uses the above Secret Provider Class

    cat <<EOF | kubectl apply -f -
    kind: Pod
    apiVersion: v1
    metadata:
      name: busybox-secrets-store-inline
      namespace: my-application
    spec:
      containers:
             - name: busybox
     image: k8s.gcr.io/e2e-test-images/busybox:1.29
     command:
       - "/bin/sleep"
       - "10000"
     volumeMounts:
     - name: secrets-store-inline
       mountPath: "/mnt/secrets-store"
       readOnly: true
      volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "azure-kvname"
         nodePublishSecretRef:
           name: secrets-store-creds
    EOF
    
  3. Check the Secret is mounted

     kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/
    

    Output should match:

     secret1
    
  4. Print the Secret

     kubectl exec busybox-secrets-store-inline \
       -- cat /mnt/secrets-store/secret1
    

    Output should match:

     Hello
    

Cleanup

  1. Uninstall Helm

     helm uninstall -n k8s-secrets-store-csi azure-csi-provider
    
  2. Delete the app

     oc delete project my-application
    
  3. Delete the Azure Key Vault

     az keyvault delete -n ${KEYVAULT_NAME}
    
  4. Delete the Service Principal

     az ad sp delete --id ${SERVICE_PRINCIPAL_CLIENT_ID}
    

Uninstalling the Kubernetes Secret Store CSI

  1. Delete the secrets store csi driver

     helm delete -n k8s-secrets-store-csi csi-secrets-store
    
  2. Delete the SecurityContextConstraints

     oc adm policy remove-scc-from-user privileged \
       system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver