AWS Load Balancer Operator On ROSA

AWS Load Balancer Controller is a controller to help manage Elastic Load Balancers for a Kubernetes cluster.

• It satisfies Kubernetes Ingress resources by provisioning Application Load Balancers .
• It satisfies Kubernetes Service resources by provisioning Network Load Balancers .

Compared with default AWS In Tree Provider, this controller is actively developed with advanced annotations for both ALB and NLB . Some advanced usecases are:

• Using native kubernetes ingress with ALB
• Integrate ALB with WAF
• Specify NLB source IP ranges
• Specify NLB internal IP address

AWS Load Balancer Operator is used to used to install, manage and configure an instance of aws-load-balancer-controller in a OpenShift cluster.

Prerequisites

• A multi AZ ROSA cluster deployed with STS
• AWS CLI
• OC CLI

Environment

1. Prepare the environment variables

   export AWS_PAGER=""
   export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
   export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/alb-operator"
   mkdir -p ${SCRATCH}
   echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"
   

AWS VPC / Subnets

Note: This section only applies to BYO VPC clusters, if you let ROSA create your VPCs you can skip to the following Installation section.

1. Set Variables describing your VPC and Subnets:

   export VPC_ID=<vpc-id>
   export PUBLIC_SUBNET_IDS=<public-subnets>
   export PRIVATE_SUBNET_IDS=<private-subnets>
   export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}")
   

2. Tag VPC with the cluster name

   aws ec2 create-tags --resources ${VPC_ID} --tags Key=kubernetes.io/cluster/${CLUSTER_NAME},Value=owned --region ${REGION}
   

3. Add tags to Public Subnets

   aws ec2 create-tags \
     --resources ${PUBLIC_SUBNET_IDS} \
     --tags Key=kubernetes.io/role/elb,Value='' \
     --region ${REGION}
   

4. Add tags to Private Subnets

   aws ec2 create-tags \
     --resources "${PRIVATE_SUBNET_IDS}" \
     --tags Key=kubernetes.io/role/internal-elb,Value='' \
     --region ${REGION}
   

Installation

1. Create Policy for the aws load balancer controller

   Note: Policy is from AWS Load Balancer Controller Policy plus subnet create tags permission (required by the operator)

   oc new-project aws-load-balancer-operator
   POLICY_ARN=$(aws iam list-policies --query \
     "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \
     --output text)
   if [[ -z "${POLICY_ARN}" ]]; then
     wget -O "${SCRATCH}/load-balancer-operator-policy.json" \
       https://raw.githubusercontent.com/rh-mobb/documentation/main/content/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json
     POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
     --output text iam create-policy \
     --policy-name aws-load-balancer-operator-policy \
     --policy-document "file://${SCRATCH}/load-balancer-operator-policy.json")
   fi
   echo $POLICY_ARN
   

2. Create trust policy for ALB Operator

   cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
     "Version": "2012-10-17",
     "Statement": [
     {
     "Effect": "Allow",
     "Condition": {
       "StringEquals" : {
         "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"]
       }
     },
     "Principal": {
       "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity"
     }
     ]
   }
   EOF
   

3. Create Role for ALB Operator

   ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
   --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
   --query Role.Arn --output text)
   echo $ROLE_ARN
   
   aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
     --policy-arn $POLICY_ARN
   

4. Create secret for ALB Operator

   cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   stringData:
     credentials: |
       [default]
       role_arn = $ROLE_ARN
       web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF
   

5. Install Red Hat AWS Load Balancer Operator

   cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     channel: stable-v1.0
     installPlanApproval: Automatic
     name: aws-load-balancer-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: aws-load-balancer-operator.v1.0.0
   EOF
   

6. Install Red Hat AWS Load Balancer Controller

   Note: If you get an error here wait a minute and try again, it likely means the Operator hasn’t completed installing yet.

   cat << EOF | oc apply -f -
   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     credentials:
       name: aws-load-balancer-operator
   EOF
   

7. Check the Operator and Controller pods are both running

   oc -n aws-load-balancer-operator get pods
   

   You should see the following, if not wait a moment and retry.

   NAME                                                             READY   STATUS    RESTARTS   AGE
   aws-load-balancer-controller-cluster-6ddf658785-pdp5d            1/1     Running   0          99s
   aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn   2/2     Running   0          2m4s
   

Validate the deployment with Echo Server application

1. Deploy Echo Server Ingress with ALB

   oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-namespace.yaml
   oc adm policy add-scc-to-user anyuid system:serviceaccount:echoserver:default
   oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-deployment.yaml
   oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-service.yaml
   oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-ingress.yaml
   

2. Curl the ALB ingress endpoint to verify the echoserver pod is accessible

   INGRESS=$(oc -n echoserver get ingress echoserver \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   curl -sH "Host: echoserver.example.com" \
     "http://${INGRESS}" | grep Hostname
   

   Hostname: echoserver-7757d5ff4d-ftvf2
   

3. Deploy Echo Server NLB Load Balancer

   cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: echoserver-nlb
     namespace: echoserver
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-type: external
       service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
       service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
   spec:
     ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
     type: LoadBalancer
     selector:
       app: echoserver
   EOF
   

4. Test the NLB endpoint

   NLB=$(oc -n echoserver get service echoserver-nlb \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   curl -s "http://${NLB}" | grep Hostname
   

   Hostname: echoserver-7757d5ff4d-ftvf2
   

Clean Up

1. Delete the Operator and the AWS Roles

   oc delete subscription aws-load-balancer-operator -n aws-load-balancer-operator
   aws iam detach-role-policy \
     --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
     --policy-arn $POLICY_ARN
   aws iam delete-role \
     --role-name "${ROSA_CLUSTER_NAME}-alb-operator"
   

2. If you wish to delete the policy you can run

   aws iam delete-policy --policy-arn $POLICY_ARN