Home GitHub

IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Enabling the AWS EFS CSI Driver Operator on ROSA

Author: Paul Czarkowski Modified: 03/23/2022

The Amazon Web Services Elastic File System (AWS EFS) is a Network File System (NFS) that can be provisioned on Red Hat OpenShift Service on AWS clusters. With the release of OpenShift 4.10 the EFS CSI Driver is now GA and available.

This is a guide to quickly enable the EFS Operator on ROSA to a Red Hat OpenShift on AWS (ROSA) cluster with STS enabled.

See here for the official ROSA documentation.

Prerequisites

Set up environment

  1. export some environment variables

     export CLUSTER_NAME="sts-cluster"
     export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json \
       | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
     export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
     export SCRATCH_DIR=/tmp/scratch
     export AWS_PAGER=""
     mkdir -p $SCRATCH_DIR
    

Prepare AWS Account

  1. Create an IAM Policy

    cat << EOF > $SCRATCH_DIR/efs-policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
     {
       "Effect": "Allow",
       "Action": [
         "elasticfilesystem:DescribeAccessPoints",
         "elasticfilesystem:DescribeFileSystems"
       ],
       "Resource": "*"
     },
     {
       "Effect": "Allow",
       "Action": [
         "elasticfilesystem:CreateAccessPoint"
       ],
       "Resource": "*",
       "Condition": {
         "StringLike": {
           "aws:RequestTag/efs.csi.aws.com/cluster": "true"
         }
       }
     },
     {
       "Effect": "Allow",
       "Action": "elasticfilesystem:DeleteAccessPoint",
       "Resource": "*",
       "Condition": {
         "StringEquals": {
           "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
         }
       }
     }
      ]
    }
    EOF
    
  2. Create the Policy

     POLICY=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-rosa-efs-csi" \
       --policy-document file://$SCRATCH_DIR/efs-policy.json \
       --query 'Policy.Arn' --output text) || \
       POLICY=$(aws iam list-policies \
       --query 'Policies[?PolicyName==`rosa-efs-csi`].Arn' \
       --output text)
     echo $POLICY
    
  3. Create a Trust Policy

    cat <<EOF > $SCRATCH_DIR/TrustPolicy.json
    {
      "Version": "2012-10-17",
      "Statement": [
     {
       "Effect": "Allow",
       "Principal": {
         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity",
       "Condition": {
         "StringEquals": {
           "${OIDC_PROVIDER}:sub": [
             "system:serviceaccount:openshift-cluster-csi-drivers:aws-efs-csi-driver-operator",
             "system:serviceaccount:openshift-cluster-csi-drivers:aws-efs-csi-driver-controller-sa"
           ]
         }
       }
     }
      ]
    }
    EOF
    
  4. Create Role for ALB Controller

     ROLE=$(aws iam create-role \
       --role-name "${CLUSTER_NAME}-aws-efs-csi-operator" \
       --assume-role-policy-document file://$SCRATCH_DIR/TrustPolicy.json \
       --query "Role.Arn" --output text)
     echo $ROLE
    
  5. Attach the Policies to the Role

     aws iam attach-role-policy \
       --role-name "${CLUSTER_NAME}-aws-efs-csi-operator" \
       --policy-arn $POLICY
    

Deploy and test the AWS EFS Operator

  1. Create a Secret to store the Access Keys

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
     name: aws-efs-cloud-credentials
     namespace: openshift-cluster-csi-drivers
    stringData:
      credentials: |-
     [default]
     role_arn = $ROLE
     web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
    EOF
    
  2. Install the EFS Operator

    cat <<EOF | oc create -f -
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      generateName: openshift-cluster-csi-drivers-
      namespace: openshift-cluster-csi-drivers
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      labels:
     operators.coreos.com/aws-efs-csi-driver-operator.openshift-cluster-csi-drivers: ""
      name: aws-efs-csi-driver-operator
      namespace: openshift-cluster-csi-drivers
    spec:
      channel: stable
      installPlanApproval: Automatic
      name: aws-efs-csi-driver-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    EOF
    
  3. Wait until the Operator is running

    watch oc get deployment aws-efs-csi-driver-operator -n openshift-cluster-csi-drivers
    
  4. Install the AWS EFS CSI Driver

    cat <<EOF | oc apply -f -
    apiVersion: operator.openshift.io/v1
    kind: ClusterCSIDriver
    metadata:
     name: efs.csi.aws.com
    spec:
      managementState: Managed
    EOF
    
  5. Wait until the CSI driver is running

    watch oc get daemonset aws-efs-csi-driver-node -n openshift-cluster-csi-drivers
    
  6. Create a storage class

    cat <<EOF | oc apply -f -
    allowVolumeExpansion: true
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: efs-csi
    parameters:
    provisioner: efs.csi.aws.com
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer
    EOF
    

Prepare an AWS EFS Volume

  1. Run this set of commands to update the VPC to allow EFS access

    NODE=$(oc get nodes --selector=node-role.kubernetes.io/worker \
      -o jsonpath='{.items[0].metadata.name}')
    VPC=$(aws ec2 describe-instances \
      --filters "Name=private-dns-name,Values=$NODE" \
      --query 'Reservations[*].Instances[*].{VpcId:VpcId}' \
      | jq -r '.[0][0].VpcId')
    SUBNET=$(aws ec2 describe-subnets \
      --filters Name=vpc-id,Values=$VPC Name=tag:Name,Values='*-private*' \
      --query 'Subnets[*].{SubnetId:SubnetId}' \
      | jq -r '.[0].SubnetId')
    CIDR=$(aws ec2 describe-vpcs \
      --filters "Name=vpc-id,Values=$VPC" \
      --query 'Vpcs[*].CidrBlock' \
      | jq -r '.[0]')
    SG=$(aws ec2 describe-instances --filters \
      "Name=private-dns-name,Values=$NODE" \
      --query 'Reservations[*].Instances[*].{SecurityGroups:SecurityGroups}' \
      | jq -r '.[0][0].SecurityGroups[0].GroupId')
    echo "CIDR - $CIDR,  SG - $SG"
    
  2. Assuming the CIDR and SG are correct, update the security group

    aws ec2 authorize-security-group-ingress \
      --group-id $SG \
      --protocol tcp \
      --port 2049 \
      --cidr $CIDR | jq .
    
  3. Create EFS File System

    EFS=$(aws efs create-file-system --creation-token efs-token-1 \
      --encrypted | jq -r '.FileSystemId')
    echo $EFS
    
  4. Configure Mount Target for EFS

    MOUNT_TARGET=$(aws efs create-mount-target --file-system-id $EFS \
      --subnet-id $SUBNET --security-groups $SG \
      | jq -r '.MountTargetId')
    echo $MOUNT_TARGET
    
  5. Create a Storage Class for the EFS volume

    cat <<EOF | oc apply -f -
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: efs-sc
    provisioner: efs.csi.aws.com
    parameters:
      provisioningMode: efs-ap
      fileSystemId: $EFS
      directoryPerms: "700"
      gidRangeStart: "1000"
      gidRangeEnd: "2000"
      basePath: "/dynamic_provisioning"
    EOF
    

Test

  1. Create a namespace

    oc new-project efs-demo
    
  2. Create a PVC

    cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: pvc-efs-volume
    spec:
      storageClassName: efs-sc
      accessModes:
     - ReadWriteMany
      resources:
     requests:
       storage: 5Gi
    EOF
    
  3. Create a Pod to write to the EFS Volume

    cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
     name: test-efs
    spec:
     volumes:
    - name: efs-storage-vol
      persistentVolumeClaim:
        claimName: pvc-efs-volume
     containers:
    - name: test-efs
      image: centos:latest
      command: [ "/bin/bash", "-c", "--" ]
      args: [ "while true; do echo 'hello efs' | tee -a /mnt/efs-data/verify-efs && sleep 5; done;" ]
      volumeMounts:
        - mountPath: "/mnt/efs-data"
          name: efs-storage-vol
    EOF
    

    It may take a few minutes for the pod to be ready. If you see errors such as Output: Failed to resolve "fs-XXXX.efs.us-east-2.amazonaws.com" it likely means its still setting up the EFS volume, just wait longer.

  4. Wait for the Pod to be ready

    watch oc get pod test-efs
    
  5. Create a Pod to read from the EFS Volume

    cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
     name: test-efs-read
    spec:
     volumes:
    - name: efs-storage-vol
      persistentVolumeClaim:
        claimName: pvc-efs-volume
     containers:
    - name: test-efs-read
      image: centos:latest
      command: [ "/bin/bash", "-c", "--" ]
      args: [ "tail -f /mnt/efs-data/verify-efs" ]
      volumeMounts:
        - mountPath: "/mnt/efs-data"
          name: efs-storage-vol
    EOF
    
  6. Verify the second POD can read the EFS Volume

    oc logs test-efs-read
    

    You should see a stream of “hello efs”

     hello efs
     hello efs
     hello efs
     hello efs
     hello efs
     hello efs
     hello efs
     hello efs
     hello efs
     hello efs
    

Cleanup

  1. Delete the Pods

     oc delete pod -n efs-demo test-efs test-efs-read
    
  2. Delete the Volume

     oc delete -n efs-demo pvc pvc-efs-volume
    
  3. Delete the Namespace

     oc delete project efs-demo
    
  4. Delete the storage class

     oc delete storageclass efs-sc
    
  5. Delete the EFS Shared Volume via AWS

    aws efs delete-mount-target --mount-target-id $MOUNT_TARGET
    aws efs delete-file-system --file-system-id $EFS
    

    Note if you receive the error An error occurred (FileSystemInUse) wait a few minutes and try again.

  6. Detach the Policies to the Role

     aws iam detach-role-policy \
       --role-name "${CLUSTER_NAME}-aws-efs-csi-operator" \
       --policy-arn $POLICY
    
  7. Delete the Role

     aws iam delete-role --role-name \
       ${CLUSTER_NAME}-aws-efs-csi-operator
    
  8. Delete the Policy

     aws iam delete-policy --policy-arn \
       $POLICY