AWS ALB

Authors: Connor Wooley
Last Editor: Dustin Scott
Published Date: 17 June 2021
Modified Date: 25 May 2023

Note: It is recommended that you use the Cloud Front based guide unless you absolutely must use an ALB based solution.

Here ’s a good overview of AWS LB types and what they support

Problem Statement

1. Operator requires WAF (Web Application Firewall) in front of their workloads running on OpenShift (ROSA)

2. Operator does not want WAF running on OpenShift to ensure that OCP resources do not experience Denial of Service through handling the WAF

Proposed Solution

Loosely based off EKS instructions here - https://aws.amazon.com/premiumsupport/knowledge-center/eks-alb-ingress-aws-waf/

1. Deploy secondary Ingress solution (+TLS +DNS) that uses an AWS ALB

   • Todo Configure TLS + DNS for that Ingress (Lets Encrypt + WildCard DNS)

Pre Requisites

• A ROSA / OSD on AWS cluster
• Helm 3 cli
• oc / kubectl
• AWS cli

1. Disable AWS cli output paging

   export AWS_PAGER=""
   

2. Set the ALB Controller version

   export ALB_VERSION="v2.2.0"
   

3. Set the name of your cluster for lookup

   export CLUSTER_NAME="waf-demo"
   

Deployment

Create a new public ROSA cluster called waf-demo and make sure to set it to be multi-AZ enabled, or replace the cluster name variable with your own cluster name.

AWS Load Balancer Controller

AWS Load Balancer controller manages the following AWS resources

Application Load Balancers to satisfy Kubernetes ingress objects Network Load Balancers in IP mode to satisfy Kubernetes service objects of type LoadBalancer with NLB IP mode annotation

1. Create AWS Policy and Service Account

   curl -so iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/${ALB_VERSION}/docs/install/iam_policy.json
   
   POLICY_ARN=$(aws iam create-policy --policy-name "AWSLoadBalancerControllerIAMPolicy" --policy-document file://iam-policy.json --query Policy.Arn --output text)
   
   echo $POLICY_ARN
   

2. Create service account

   aws iam create-user --user-name aws-lb-controller  \
     --query User.Arn --output text
   

3. Attach policy to user

   aws iam attach-user-policy --user-name aws-lb-controller \
     --policy-arn ${POLICY_ARN}
   

4. Create access key and save the output (Paste the AccessKeyId and SecretAccessKey into values.yaml)

   aws iam create-access-key --user-name aws-lb-controller
   

   export AWS_ID=<from above>
   export AWS_KEY=<from above>
   

5. Modify the VPC ID and cluster name in the values.yaml with the output from (replace poc-waf with your cluster name):

   VPC_ID=$(aws ec2 describe-vpcs --output json --filters \
     Name=tag-value,Values="${CLUSTER_NAME}*" \
     --query "Vpcs[].VpcId" --output text)
   echo ${VPC_ID}
   

6. Modify the subnet list in ingress.yaml with the output from: (replace poc-waf with your cluster name)

   SUBNET_IDS=$(aws ec2 describe-subnets --output json \
     --filters Name=tag-value,Values="${CLUSTER_NAME}-*public*" \
     --query "Subnets[].SubnetId" --output text | sed 's/\t/ /g')
   echo ${SUBNET_IDS}
   

7. Add tags to those subnets (change the subnet ids in the resources line)

   aws ec2 create-tags \
     --resources $(echo ${SUBNET_IDS}) \
     --tags Key=kubernetes.io/role/elb,Value= Key=kubernetes.io/cluster/${CLUSTER_NAME},Value=shared
   

8. Create a namespace for the controller

   kubectl create ns aws-load-balancer-controller
   

9. Apply CRDs

   kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
   

10. Add the helm repo and install the controller (install helm3 if not already)

    helm repo add eks https://aws.github.io/eks-charts
    helm install -n aws-load-balancer-controller \
      aws-load-balancer-controller eks/aws-load-balancer-controller \
      --set "env.AWS_ACCESS_KEY_ID=${AWS_ID}" \
      --set "env.AWS_SECRET_ACCESS_KEY=${AWS_KEY}" \
      --set "vpcID=${VPC_ID}" \
      --set "clusterName=${CLUSTER_NAME}" \
      --set "image.tag=${ALB_VERSION}" \
      --create-namespace
    

Deploy Sample Application

1. Create a new application in OpenShift

   oc new-project demo
   oc new-app https://github.com/sclorg/django-ex.git
   kubectl -n demo patch service django-ex -p '{"spec":{"type":"NodePort"}}'
   

2. Create an Ingress to trigger an ALB

   cat << EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: django-ex
     namespace: demo
     annotations:
       kubernetes.io/ingress.class: alb
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: instance
       # alb.ingress.kubernetes.io/subnets: subnet-0982bb73ca67d61de,subnet-0aa9967e8767d792f,subnet-0fd57669a80eb7596
       alb.ingress.kubernetes.io/shield-advanced-protection: "true"
       # wafv2 arn to use
       # alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-2:660250927410:regional/webacl/waf-demo/6565d2a1-6d26-4b6b-b56f-1e996c7e9e8f
     labels:
       app: django-ex
   spec:
     rules:
       - host: foo.bar
         http:
           paths:
             - pathType: Prefix
               path: /*
               backend:
                 service:
                   name: django-ex
                   port:
                     number: 8080
   

3. Check the logs of the ALB controller

   kubectl logs -f deployment/aws-load-balancer-controller
   

4. use the second address from the ingress to browse to the app

   kubectl -n demo get ingress
   

   curl -s --header "Host: foo.bar" k8s-demo-djangoex-49f31c1921-782305710.us-east-2.elb.amazonaws.com | head
   

WAF time

1. Create a WAF rule here https://console.aws.amazon.com/wafv2/homev2/web-acls/new and use the Core and SQL Injection rules. (make sure region matches us-east-2)

2. View your WAF

   aws wafv2 list-web-acls --scope REGIONAL --region us-east-2 | jq .
   

3. set the waf annotation to match the ARN provided above (and uncomment it) then re-apply the ingress

   kubectl apply -f ingress.yaml
   

4. test the app still works

   curl -s --header "Host: foo.bar" --location "k8s-demo-djangoex-49f31c1921-782305710.us-east-2.elb.amazonaws.com"
   

5. test the WAF denies a bad request

   You should get a 403 Forbidden error

   curl -X POST http://k8s-demo-djangoex-49f31c1921-782305710.us-east-2.elb.amazonaws.com -F "user='<script><alert>Hello></alert></script>'"