Using AWS Secrets Manager CSI on Red Hat OpenShift on AWS with STS

Author Paul Czarkowski

last modified 2021-08-17

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on ROSA or OSD.

This is made even easier / more secure through the use of AWS STS and Kubernetes PodIdentity.

Prerequisites

Preparing Environment

Validate that your cluster has STS

 oc get authentication.config.openshift.io cluster -o json \
 | jq .spec.serviceAccountIssuer

You should see something like the following, if not, you could try adding support yourself like this, but YMMV.

 "https://rh-oidc.s3.us-east-1.amazonaws.com/xxxxxx"

Set SecurityContextConstraints to allow the CSI driver to run

 oc adm policy add-scc-to-user privileged \
   system:serviceaccount:kube-system:secrets-store-csi-driver
 oc adm policy add-scc-to-user privileged \
   system:serviceaccount:kube-system:csi-secrets-store-provider-aws

Create some environment variables to refer to later

 export ROSA_CLUSTER_NAME=my-cluster
 export ROSA_CLUSTER_ID=$(rosa describe cluster -c $ROSA_CLUSTER_NAME --output json | jq -r .id)
 export REGION=us-east-2 \
 export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq .spec.serviceAccountIssuer)
 export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
 export AWS_PAGER=""

Deploy the AWS Secrets and Configuration Provider

Use Helm to register the secrets store csi driver

 helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts

Update your Helm Repositories
```
 helm repo update
```

Install the secrets store csi driver

 helm install -n kube-system csi-secrets-store \
   secrets-store-csi-driver/secrets-store-csi-driver

Deploy the AWS provider

 kubectl -n kube-system apply -f \
   https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Check that both Daemonsets are running

 kubectl -n kube-system get ds \
   csi-secrets-store-provider-aws \
   csi-secrets-store-secrets-store-csi-driver

Creating a Secret and IAM Access Policies

Create a secret in Secrets Manager

 SECRET_ARN=$(aws --region "$REGION" secretsmanager  create-secret \
   --name MySecret --secret-string \
   '{"username":"shadowman", "password":"hunter2"}' \
   --query ARN --output text)

 echo $SECRET_ARN

Create IAM Access Policy document

 cat << EOF > policy.json
 {
   "Version": "2012-10-17",
   "Statement": [{
       "Effect": "Allow",
       "Action": [
         "secretsmanager:GetSecretValue",
         "secretsmanager:DescribeSecret"
       ],
       "Resource": ["$SECRET_ARN"]
       }]
 }
 EOF

Create an IAM Access Policy

 POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
   --output text iam create-policy \
   --policy-name openshift-access-to-mysecret-policy \
   --policy-document file://policy.json)
 echo $POLICY_ARN

Create IAM Role trust policy document

Note you can use Conditions to lock down to a specific namespace or service account here. But for simplicity we’re keeping it open.

 cat <<EOF > trust-policy.json
 {
 "Version": "2012-10-17",
 "Statement": [
   {
   "Effect": "Allow",
   "Principal": {
     "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/$ROSA_CLUSTER_ID"
   },
   "Action": "sts:AssumeRoleWithWebIdentity"
   }
 ]
 }
 EOF

Create IAM Role

 ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
   --assume-role-policy-document file://trust-policy.json \
   --query Role.Arn --output text)
 echo $ROLE_ARN

Attach Role to the Policy

 aws iam attach-role-policy --role-name openshift-access-to-mysecret --policy-arn $POLICY_ARN

Create an Application to use this secret

Create an OpenShift project
```
 oc new-project my-application
```

Annotate the default service account to use the STS Role

 oc annotate -n my-application serviceaccount default \
   eks.amazonaws.com/role-arn=$ROLE_ARN

Create a secret provider class to access our secret

 cat << EOF | kubectl apply -f -
 apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
 kind: SecretProviderClass
 metadata:
   name: my-application-aws-secrets
 spec:
   provider: aws
   parameters:
     objects: |
         - objectName: "MySecret"
           objectType: "secretsmanager"
 EOF

Create a Deployment using our secret

 cat << EOF | kubectl apply -f -
 apiVersion: v1
 kind: Pod
 metadata:
   name: my-application
   labels:
     app: my-application
 spec:
   volumes:
   - name: secrets-store-inline
     csi:
       driver: secrets-store.csi.k8s.io
       readOnly: true
       volumeAttributes:
         secretProviderClass: "my-application-aws-secrets"
   containers:
   - name: my-application-deployment
     image: k8s.gcr.io/e2e-test-images/busybox:1.29
     command:
       - "/bin/sleep"
       - "10000"
     volumeMounts:
     - name: secrets-store-inline
       mountPath: "/mnt/secrets-store"
       readOnly: true
 EOF

Verify the Pod has the secret mounted

 kubectl exec -it my-application -- cat /mnt/secrets-store/MySecret

Cleanup

Delete application
```
 oc delete project my-application
```

Delete the secrets store csi driver

 helm delete -n kube-system csi-secrets-store

Delete the AWS provider

 kubectl -n kube-system delete -f \
   https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Delete Security Context Constraints

 oc adm policy remove-scc-from-user privileged \
   system:serviceaccount:kube-system:secrets-store-csi-driver
 oc adm policy remove-scc-from-user privileged \
   system:serviceaccount:kube-system:csi-secrets-store-provider-aws

Delete AWS Roles and Policies

 aws iam detach-role-policy --role-name openshift-access-to-mysecret --policy-arn $POLICY_ARN
 aws iam delete-role --role-name openshift-access-to-mysecret
 aws iam delete-policy --policy-arn $POLICY_ARN