Using AWS Secrets Manager CSI on Red Hat OpenShift on AWS with STS

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on ROSA or OSD.

This is made even easier and more secure through the use of AWS STS and Kubernetes PodIdentity.

Prerequisites

• A ROSA cluster deployed with STS
• Helm 3
• aws CLI
• oc CLI
• jq

Preparing Environment

1. Validate that your cluster has STS

   oc get authentication.config.openshift.io cluster -o json \
     | jq .spec.serviceAccountIssuer
   

   Copy

   You should see something like the following, if not you should not proceed, instead look to the Red Hat documentation on creating an STS cluster .

   "https://xxxxx.cloudfront.net/xxxxx"
   

   Copy

2. Set SecurityContextConstraints to allow the CSI driver to run

   oc new-project csi-secrets-store
   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
   

   Copy

3. Create some environment variables to refer to later

   export REGION=us-east-2
   export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \
     -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
   export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   export AWS_PAGER=""
   

   Copy

Deploy the AWS Secrets and Configuration Provider

1. Use Helm to register the secrets store csi driver

   helm repo add secrets-store-csi-driver \
     https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   

   Copy

2. Update your Helm Repositories

   helm repo update
   

   Copy

3. Install the secrets store csi driver

   helm upgrade --install -n csi-secrets-store \
     csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver
   

   Copy

4. Deploy the AWS provider

   oc -n csi-secrets-store apply -f \
     https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
   

   Copy

5. Check that both Daemonsets are running

   oc -n csi-secrets-store get ds \
     csi-secrets-store-provider-aws \
     csi-secrets-store-driver-secrets-store-csi-driver
   

   Copy

6. Add pod security profile label for CSI Driver

   This is required starting in OpenShift v4.13

   oc label csidriver/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
   

   Copy

Creating a Secret and IAM Access Policies

1. Create a secret in Secrets Manager

   SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \
     --name MySecret --secret-string \
     '{"username":"shadowman", "password":"hunter2"}' \
     --query ARN --output text)
   
   echo $SECRET_ARN
   

   Copy

2. Create IAM Access Policy document

   cat << EOF > policy.json
   {
     "Version": "2012-10-17",
     "Statement": [{
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue",
           "secretsmanager:DescribeSecret"
         ],
         "Resource": ["$SECRET_ARN"]
         }]
   }
   EOF
   

   Copy

3. Create an IAM Access Policy

   POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
     --output text iam create-policy \
     --policy-name openshift-access-to-mysecret-policy \
     --policy-document file://policy.json)
   echo $POLICY_ARN
   

   Copy

4. Create IAM Role trust policy document

   Note the trust policy is locked down to the default service account of a namespace you will create later.

   cat <<EOF > trust-policy.json
   {
     "Version": "2012-10-17",
     "Statement": [
     {
     "Effect": "Allow",
     "Condition": {
       "StringEquals" : {
         "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"]
       }
     },
     "Principal": {
       "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity"
     }
     ]
   }
   EOF
   

   Copy

5. Create IAM Role

   ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
     --assume-role-policy-document file://trust-policy.json \
     --query Role.Arn --output text)
   echo $ROLE_ARN
   

   Copy

6. Attach Role to the Policy

   aws iam attach-role-policy --role-name openshift-access-to-mysecret \
     --policy-arn $POLICY_ARN
   

   Copy

Create an Application to use this secret

1. Create an OpenShift project

   oc new-project my-application
   

   Copy

2. Annotate the default service account to use the STS Role

   oc annotate -n my-application serviceaccount default \
     eks.amazonaws.com/role-arn=$ROLE_ARN
   

   Copy

3. Create a secret provider class to access our secret

   cat << EOF | oc apply -f -
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: my-application-aws-secrets
   spec:
     provider: aws
     parameters:
       objects: |
           - objectName: "MySecret"
             objectType: "secretsmanager"
   EOF
   

   Copy

4. Create a Deployment using our secret

   cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: my-application
     labels:
       app: my-application
   spec:
     volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "my-application-aws-secrets"
     containers:
     - name: my-application-deployment
       image: k8s.gcr.io/e2e-test-images/busybox:1.29
       command:
         - "/bin/sleep"
         - "10000"
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
   EOF
   

   Copy

5. Verify the Pod has the secret mounted

   oc exec -it my-application -- cat /mnt/secrets-store/MySecret
   

   Copy

Cleanup

1. Delete application

   oc delete project my-application
   

   Copy

2. Delete the secrets store csi driver

   helm delete -n csi-secrets-store csi-secrets-store-driver
   

   Copy

3. Delete Security Context Constraints

   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
   

   Copy

4. Delete the AWS provider

   oc -n csi-secrets-store delete -f \
     https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
   

   Copy

5. Delete AWS Roles and Policies

   aws iam detach-role-policy --role-name openshift-access-to-mysecret \
     --policy-arn $POLICY_ARN
   aws iam delete-role --role-name openshift-access-to-mysecret
   aws iam delete-policy --policy-arn $POLICY_ARN
   

   Copy

6. Delete the Secrets Manager secret

   aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN
   

   Copy