IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Using AWS Secrets Manager CSI on Red Hat OpenShift on AWS with STS

Authors: Paul Czarkowski, Chris Kang
Last Editor: Andy Repton
Published Date: 25 May 2023
Modified Date: 25 May 2023

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on ROSA or OSD.

This is made even easier and more secure through the use of AWS STS and Kubernetes PodIdentity.

Prerequisites

A ROSA cluster deployed with STS
Helm 3
aws CLI
oc CLI
jq

Preparing Environment

Validate that your cluster has STS
```
oc get authentication.config.openshift.io cluster -o json \
  | jq .spec.serviceAccountIssuer
```
You should see something like the following, if not you should not proceed, instead look to the Red Hat documentation on creating an STS cluster .
```
"https://xxxxx.cloudfront.net/xxxxx"
```

Set SecurityContextConstraints to allow the CSI driver to run

oc new-project csi-secrets-store
oc adm policy add-scc-to-user privileged \
  system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
oc adm policy add-scc-to-user privileged \
  system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

Create some environment variables to refer to later

export REGION=us-east-2
export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \
  -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
export AWS_PAGER=""

Deploy the AWS Secrets and Configuration Provider

Use Helm to register the secrets store csi driver

helm repo add secrets-store-csi-driver \
  https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

Update your Helm Repositories
```
helm repo update
```

Install the secrets store csi driver

helm upgrade --install -n csi-secrets-store \
  csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver

Deploy the AWS provider

oc -n csi-secrets-store apply -f \
  https://raw.githubusercontent.com/rh-mobb/documentation/main/content/docs/misc/secrets-store-csi/aws-provider-installer.yaml

Check that both Daemonsets are running

oc -n csi-secrets-store get ds \
  csi-secrets-store-provider-aws \
  csi-secrets-store-driver-secrets-store-csi-driver

Creating a Secret and IAM Access Policies

Create a secret in Secrets Manager

SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \
  --name MySecret --secret-string \
  '{"username":"shadowman", "password":"hunter2"}' \
  --query ARN --output text)

echo $SECRET_ARN

Create IAM Access Policy document

cat << EOF > policy.json
{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": ["$SECRET_ARN"]
      }]
}
EOF

Create an IAM Access Policy

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
  --output text iam create-policy \
  --policy-name openshift-access-to-mysecret-policy \
  --policy-document file://policy.json)
echo $POLICY_ARN

Create IAM Role trust policy document

Note the trust policy is locked down to the default service account of a namespace you will create later.

cat <<EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
  {
  "Effect": "Allow",
  "Condition": {
    "StringEquals" : {
      "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"]
    }
  },
  "Principal": {
    "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
  },
  "Action": "sts:AssumeRoleWithWebIdentity"
  }
  ]
}
EOF

Create IAM Role

ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
  --assume-role-policy-document file://trust-policy.json \
  --query Role.Arn --output text)
echo $ROLE_ARN

Attach Role to the Policy

aws iam attach-role-policy --role-name openshift-access-to-mysecret \
  --policy-arn $POLICY_ARN

Create an Application to use this secret

Create an OpenShift project
```
oc new-project my-application
```

Annotate the default service account to use the STS Role

oc annotate -n my-application serviceaccount default \
  eks.amazonaws.com/role-arn=$ROLE_ARN

Create a secret provider class to access our secret

cat << EOF | oc apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-application-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "MySecret"
          objectType: "secretsmanager"
EOF

Create a Deployment using our secret

cat << EOF | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: my-application
  labels:
    app: my-application
spec:
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-application-aws-secrets"
  containers:
  - name: my-application-deployment
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
      - "/bin/sleep"
      - "10000"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
EOF

Verify the Pod has the secret mounted

oc exec -it my-application -- cat /mnt/secrets-store/MySecret

Cleanup

Delete application
```
oc delete project my-application
```

Delete the secrets store csi driver

helm delete -n csi-secrets-store csi-secrets-store-driver

Delete Security Context Constraints

oc adm policy remove-scc-from-user privileged \
  system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
oc adm policy remove-scc-from-user privileged \
  system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

Delete the AWS provider

oc -n csi-secrets-store delete -f \
  https://raw.githubusercontent.com/rh-mobb/documentation/main/content/docs/misc/secrets-store-csi/aws-provider-installer.yaml

Delete AWS Roles and Policies

aws iam detach-role-policy --role-name openshift-access-to-mysecret \
  --policy-arn $POLICY_ARN
aws iam delete-role --role-name openshift-access-to-mysecret
aws iam delete-policy --policy-arn $POLICY_ARN

Delete the Secrets Manager secret

aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN