IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

AWS Load Balancer Operator On ROSA

Author Shaozhen Ding, Paul Czarkowski

last edited: 01/05/2023

AWS Load Balancer Controller is a controller to help manage Elastic Load Balancers for a Kubernetes cluster.

Compared with default AWS In Tree Provider, this controller is actively developed with advanced annotations for both ALB and NLB. Some advanced usecases are:

  • Using native kubernetes ingress with ALB
  • Integrate ALB with WAF
  • Specify NLB source IP ranges
  • Specify NLB internal IP address

AWS Load Balancer Operator is used to used to install, manage and configure an instance of aws-load-balancer-controller in a OpenShift cluster.

Prerequisites

Environment

  1. Prepare the environment variables

    export AWS_PAGER=""
export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/alb-operator"
mkdir -p ${SCRATCH}
echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

AWS VPC / Subnets

Note: This section only applies to BYO VPC clusters, if you let ROSA create your VPCs you can skip to the following Installation section.

  1. Set Variables describing your VPC and Subnets:

    export VPC=<vpc-id>
export PUBLIC_SUBNET_IDS=<public-subnets>
export PRIVATE_SUBNET_IDS=<private-subnets>
export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}")

  2. Tag VPC with the cluster name

    aws ec2 create-tags --resources VPC_ID --tags Key=kubernetes.io/cluster/${CLUSTER_NAME},Value=owned

  3. Add tags to Public Subnets

    aws ec2 create-tags \
  --resources "${PUBLIC_SUBNET_IDS}" \
  --tags Key=kubernetes.io/role/elb,Value=''

  4. Add tags to Private Subnets

    aws ec2 create-tags \
  --resources "${PRIVATE_SUBNET_IDS}" \
  --tags Key=kubernetes.io/role/internal-elb,Value=''

Installation

  1. Create Policy for the aws load balancer controller

    Note: Policy is from AWS Load Balancer Controller Policy plus subnet create tags permission (required by the operator)

    oc new-project aws-load-balancer-operator
POLICY_ARN=$(aws iam list-policies --query \
  "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \
  --output text)
if [[ -z "${POLICY_ARN}" ]]; then
  wget -O "${SCRATCH}/load-balancer-operator-policy.json" \
    https://raw.githubusercontent.com/rh-mobb/documentation/main/docs/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json
  POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
  --output text iam create-policy \
  --policy-name aws-load-balancer-operator-policy \
  --policy-document "file://${SCRATCH}/load-balancer-operator-policy.json")
fi
echo $POLICY_ARN

  2. Create trust policy for ALB Operator

    cat <<EOF > "${SCRATCH}/trust-policy.json"
{
  "Version": "2012-10-17",
  "Statement": [
  {
  "Effect": "Allow",
  "Condition": {
    "StringEquals" : {
      "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"]
    }
  },
  "Principal": {
    "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
  },
  "Action": "sts:AssumeRoleWithWebIdentity"
  }
  ]
}
EOF

  3. Create Role for ALB Operator

    ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
--assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
--query Role.Arn --output text)
echo $ROLE_ARN

aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
  --policy-arn $POLICY_ARN

  4. Create secret for ALB Operator

    cat << EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: aws-load-balancer-operator
  namespace: aws-load-balancer-operator
stringData:
  credentials: |
    [default]
    role_arn = $ROLE_ARN
    web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
EOF

  5. Install Red Hat AWS Load Balancer Operator

    cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: aws-load-balancer-operator
  namespace: aws-load-balancer-operator
spec:
  targetNamespaces:
    - aws-load-balancer-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: aws-load-balancer-operator
  namespace: aws-load-balancer-operator
spec:
  channel: stable-v0
  installPlanApproval: Automatic
  name: aws-load-balancer-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: aws-load-balancer-operator.v0.2.0
EOF

  6. Install Red Hat AWS Load Balancer Controller

    Note: If you get an error here wait a minute and try again, it likely means the Operator hasn’t completed installing yet.

    cat << EOF | oc apply -f -
apiVersion: networking.olm.openshift.io/v1alpha1
kind: AWSLoadBalancerController
metadata:
  name: cluster
spec:
  credentials:
    name: aws-load-balancer-operator
EOF

  7. Check the Operator and Controller pods are both running

    oc -n aws-load-balancer-operator get pods

    You should see the following, if not wait a moment and retry.

    NAME                                                             READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-cluster-6ddf658785-pdp5d            1/1     Running   0          99s
aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn   2/2     Running   0          2m4s

Validate the deployment with Echo Server application

  1. Deploy Echo Server Ingress with ALB

    oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-namespace.yaml
oc adm policy add-scc-to-user anyuid system:serviceaccount:echoserver:default
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-deployment.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-service.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/echoservice/echoserver-ingress.yaml

  2. Curl the ALB ingress endpoint to verify the echoserver pod is accessible

    INGRESS=$(oc -n echoserver get ingress echoserver \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -sH "Host: echoserver.example.com" \
  "http://${INGRESS}" | grep Hostname

    Hostname: echoserver-7757d5ff4d-ftvf2

  3. Deploy Echo Server NLB Load Balancer

    cat << EOF | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  name: echoserver-nlb
  namespace: echoserver
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  selector:
    app: echoserver
EOF

  4. Test the NLB endpoint

    NLB=$(oc -n echoserver get service echoserver-nlb \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -s "http://${NLB}" | grep Hostname

    Hostname: echoserver-7757d5ff4d-ftvf2

Clean Up

  1. Delete the Operator and the AWS Roles

    oc delete subscription aws-load-balancer-operator -n aws-load-balancer-operator
aws iam detach-role-policy \
  --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
  --policy-arn $POLICY_ARN
aws iam delete-role \
  --role-name "${ROSA_CLUSTER_NAME}-alb-operator"

  2. If you wish to delete the policy you can run

    aws iam delete-policy --policy-arn $POLICY_ARN