Chapter 11. Tutorial: Using AWS Secrets Manager CSI on ROSA with STS
The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on Red Hat OpenShift Service on AWS (ROSA).
11.1. Prerequisites Copy link
Ensure that you have the following resources and tools before starting this process:
- A ROSA cluster deployed with STS
- Helm 3
-
aws
CLI -
oc
CLI -
jq
CLI
Additional environment requirements
Log in to your ROSA cluster by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc login --token=<your-token> --server=<your-server-url>
$ oc login --token=<your-token> --server=<your-server-url>
You can find your login token by accessing your cluster in pull secret from Red Hat OpenShift Cluster Manager.
Validate that your cluster has STS by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get authentication.config.openshift.io cluster -o json \ | jq .spec.serviceAccountIssuer
$ oc get authentication.config.openshift.io cluster -o json \ | jq .spec.serviceAccountIssuer
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow "https://xxxxx.cloudfront.net/xxxxx"
"https://xxxxx.cloudfront.net/xxxxx"
If your output is different, do not proceed. See Red Hat documentation on creating an STS cluster before continuing this process.
Set the
SecurityContextConstraints
permission to allow the CSI driver to run by running the following command:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc new-project csi-secrets-store oc adm policy add-scc-to-user privileged \ system:serviceaccount:csi-secrets-store:secrets-store-csi-driver oc adm policy add-scc-to-user privileged \ system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
$ oc new-project csi-secrets-store $ oc adm policy add-scc-to-user privileged \ system:serviceaccount:csi-secrets-store:secrets-store-csi-driver $ oc adm policy add-scc-to-user privileged \ system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
Create environment variables to use later in this process by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \ -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text` export AWS_PAGER=""
$ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \ -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text` $ export AWS_PAGER=""
Use Helm to register the secrets store CSI driver by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow helm repo add secrets-store-csi-driver \ https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
$ helm repo add secrets-store-csi-driver \ https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
Update your Helm repositories by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow helm repo update
$ helm repo update
Install the secrets store CSI driver by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow helm upgrade --install -n csi-secrets-store \ csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver
$ helm upgrade --install -n csi-secrets-store \ csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver
Deploy the AWS provider by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n csi-secrets-store apply -f \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
$ oc -n csi-secrets-store apply -f \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
Check that both Daemonsets are running by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n csi-secrets-store get ds \ csi-secrets-store-provider-aws \ csi-secrets-store-driver-secrets-store-csi-driver
$ oc -n csi-secrets-store get ds \ csi-secrets-store-provider-aws \ csi-secrets-store-driver-secrets-store-csi-driver
Label the Secrets Store CSI Driver to allow use with the restricted pod security profile by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
$ oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
Create a secret in Secrets Manager by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \ --name MySecret --secret-string \ '{"username":"shadowman", "password":"hunter2"}' \ --query ARN --output text); echo $SECRET_ARN
$ SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \ --name MySecret --secret-string \ '{"username":"shadowman", "password":"hunter2"}' \ --query ARN --output text); echo $SECRET_ARN
Create an IAM Access Policy document by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat << EOF > policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": ["$SECRET_ARN"] }] } EOF
$ cat << EOF > policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": ["$SECRET_ARN"] }] } EOF
Create an IAM Access Policy by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \ --output text iam create-policy \ --policy-name openshift-access-to-mysecret-policy \ --policy-document file://policy.json); echo $POLICY_ARN
$ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \ --output text iam create-policy \ --policy-name openshift-access-to-mysecret-policy \ --policy-document file://policy.json); echo $POLICY_ARN
Create an IAM Role trust policy document by running the following command:
NoteThe trust policy is locked down to the default service account of a namespace you create later in this process.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat <<EOF > trust-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"] } }, "Principal": { "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF
$ cat <<EOF > trust-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"] } }, "Principal": { "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF
Create an IAM role by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \ --assume-role-policy-document file://trust-policy.json \ --query Role.Arn --output text); echo $ROLE_ARN
$ ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \ --assume-role-policy-document file://trust-policy.json \ --query Role.Arn --output text); echo $ROLE_ARN
Attach the role to the policy by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow aws iam attach-role-policy --role-name openshift-access-to-mysecret \ --policy-arn $POLICY_ARN
$ aws iam attach-role-policy --role-name openshift-access-to-mysecret \ --policy-arn $POLICY_ARN
Create an OpenShift project by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc new-project my-application
$ oc new-project my-application
Annotate the default service account to use the STS Role by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc annotate -n my-application serviceaccount default \ eks.amazonaws.com/role-arn=$ROLE_ARN
$ oc annotate -n my-application serviceaccount default \ eks.amazonaws.com/role-arn=$ROLE_ARN
Create a secret provider class to access our secret by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat << EOF | oc apply -f - apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: my-application-aws-secrets spec: provider: aws parameters: objects: | - objectName: "MySecret" objectType: "secretsmanager" EOF
$ cat << EOF | oc apply -f - apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: my-application-aws-secrets spec: provider: aws parameters: objects: | - objectName: "MySecret" objectType: "secretsmanager" EOF
Create a deployment by using our secret in the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat << EOF | oc apply -f - apiVersion: v1 kind: Pod metadata: name: my-application labels: app: my-application spec: volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "my-application-aws-secrets" containers: - name: my-application-deployment image: k8s.gcr.io/e2e-test-images/busybox:1.29 command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true EOF
$ cat << EOF | oc apply -f - apiVersion: v1 kind: Pod metadata: name: my-application labels: app: my-application spec: volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "my-application-aws-secrets" containers: - name: my-application-deployment image: k8s.gcr.io/e2e-test-images/busybox:1.29 command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true EOF
Verify the pod has the secret mounted by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc exec -it my-application -- cat /mnt/secrets-store/MySecret
$ oc exec -it my-application -- cat /mnt/secrets-store/MySecret
11.5. Clean up Copy link
Delete the application by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete project my-application
$ oc delete project my-application
Delete the secrets store csi driver by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow helm delete -n csi-secrets-store csi-secrets-store-driver
$ helm delete -n csi-secrets-store csi-secrets-store-driver
Delete the security context constraints by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc adm policy remove-scc-from-user privileged \ system:serviceaccount:csi-secrets-store:secrets-store-csi-driver; oc adm policy remove-scc-from-user privileged \ system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
$ oc adm policy remove-scc-from-user privileged \ system:serviceaccount:csi-secrets-store:secrets-store-csi-driver; oc adm policy remove-scc-from-user privileged \ system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
Delete the AWS provider by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n csi-secrets-store delete -f \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
$ oc -n csi-secrets-store delete -f \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
Delete AWS Roles and Policies by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow aws iam detach-role-policy --role-name openshift-access-to-mysecret \ --policy-arn $POLICY_ARN; aws iam delete-role --role-name openshift-access-to-mysecret; aws iam delete-policy --policy-arn $POLICY_ARN
$ aws iam detach-role-policy --role-name openshift-access-to-mysecret \ --policy-arn $POLICY_ARN; aws iam delete-role --role-name openshift-access-to-mysecret; aws iam delete-policy --policy-arn $POLICY_ARN
Delete the Secrets Manager secret by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN
$ aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN