Home GitHub

Disclaimer: Mobb.ninja is not official Red Hat documentation - These guides may be experimental, proof of concept or early adoption. Officially supported documentation is available at https://docs.openshift.com.

This is a POC of ROSA with a AWS WAF service

Note: It is recommended that you use the Cloud Front based guide unless you absolutely must use an ALB based solution.

Here’s a good overview of AWS LB types and what they support

Problem Statement

  1. Operator requires WAF (Web Application Firewall) in front of their workloads running on OpenShift (ROSA)

  2. Operator does not want WAF running on OpenShift to ensure that OCP resources do not experience Denial of Service through handling the WAF

Proposed Solution

Loosely based off EKS instructions here - https://aws.amazon.com/premiumsupport/knowledge-center/eks-alb-ingress-aws-waf/

  1. Deploy secondary Ingress solution (+TLS +DNS) that uses an AWS ALB

    • Todo Configure TLS + DNS for that Ingress (Lets Encrypt + WildCard DNS)

Pre Requisites

  1. Disable AWS cli output paging

     export AWS_PAGER=""
    
  2. Set the ALB Controller version

     export ALB_VERSION="v2.2.0"
    
  3. Set the name of your cluster for lookup

     export CLUSTER_NAME="waf-demo"
    

Deployment

Create a new public ROSA cluster called waf-demo and make sure to set it to be multi-AZ enabled, or replace the cluster name variable with your own cluster name.

AWS Load Balancer Controller

AWS Load Balancer controller manages the following AWS resources

Application Load Balancers to satisfy Kubernetes ingress objects Network Load Balancers in IP mode to satisfy Kubernetes service objects of type LoadBalancer with NLB IP mode annotation

  1. Create AWS Policy and Service Account

     curl -so iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/${ALB_VERSION}/docs/install/iam_policy.json
    
     POLICY_ARN=$(aws iam create-policy --policy-name "AWSLoadBalancerControllerIAMPolicy" --policy-document file://iam-policy.json --query Policy.Arn --output text)
    
     echo $POLICY_ARN
    
  2. Create service account

     aws iam create-user --user-name aws-lb-controller  \
       --query User.Arn --output text
    
  3. Attach policy to user

     aws iam attach-user-policy --user-name aws-lb-controller \
       --policy-arn ${POLICY_ARN}
    
  4. Create access key and save the output (Paste the AccessKeyId and SecretAccessKey into values.yaml)

     aws iam create-access-key --user-name aws-lb-controller
    
     export AWS_ID=<from above>
     export AWS_KEY=<from above>
    
  5. Modify the VPC ID and cluster name in the values.yaml with the output from (replace poc-waf with your cluster name):

     VPC_ID=$(aws ec2 describe-vpcs --output json --filters \
       Name=tag-value,Values="${CLUSTER_NAME}*" \
       --query "Vpcs[].VpcId" --output text)
     echo ${VPC_ID}
    
  6. Modify the subnet list in ingress.yaml with the output from: (replace poc-waf with your cluster name)

     SUBNET_IDS=$(aws ec2 describe-subnets --output json \
       --filters Name=tag-value,Values="${CLUSTER_NAME}-*public*" \
       --query "Subnets[].SubnetId" --output text | sed 's/\t/ /g')
     echo ${SUBNET_IDS}
    
  7. Add tags to those subnets (change the subnet ids in the resources line)

     aws ec2 create-tags \
       --resources $(echo ${SUBNET_IDS}) \
       --tags Key=kubernetes.io/role/elb,Value= Key=kubernetes.io/cluster/${CLUSTER_NAME},Value=shared
    
  8. Create a namespace for the controller

     kubectl create ns aws-load-balancer-controller
    
  9. Apply CRDs

     kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
    
  10. Add the helm repo and install the controller (install helm3 if not already)

     helm repo add eks https://aws.github.io/eks-charts
     helm install -n aws-load-balancer-controller \
       aws-load-balancer-controller eks/aws-load-balancer-controller \
       --set "env.AWS_ACCESS_KEY_ID=${AWS_ID}" \
       --set "env.AWS_SECRET_ACCESS_KEY=${AWS_KEY}" \
       --set "vpcID=${VPC_ID}" \
       --set "clusterName=${CLUSTER_NAME}" \
       --set "image.tag=${ALB_VERSION}" \
       --create-namespace
    

Deploy Sample Application

  1. Create a new application in OpenShift

     oc new-project demo
     oc new-app https://github.com/sclorg/django-ex.git
     kubectl -n demo patch service django-ex -p '{"spec":{"type":"NodePort"}}'
    
  2. Create an Ingress to trigger an ALB

     cat << EOF | kubectl apply -f -
     apiVersion: networking.k8s.io/v1
     kind: Ingress
     metadata:
       name: django-ex
       namespace: demo
       annotations:
         kubernetes.io/ingress.class: alb
         alb.ingress.kubernetes.io/scheme: internet-facing
         alb.ingress.kubernetes.io/target-type: instance
         # alb.ingress.kubernetes.io/subnets: subnet-0982bb73ca67d61de,subnet-0aa9967e8767d792f,subnet-0fd57669a80eb7596
         alb.ingress.kubernetes.io/shield-advanced-protection: "true"
         # wafv2 arn to use
         # alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-2:660250927410:regional/webacl/waf-demo/6565d2a1-6d26-4b6b-b56f-1e996c7e9e8f
       labels:
         app: django-ex
     spec:
       rules:
         - host: foo.bar
           http:
             paths:
               - pathType: Prefix
                 path: /*
                 backend:
                   service:
                     name: django-ex
                     port:
                       number: 8080
    
  3. Check the logs of the ALB controller

     kubectl logs -f deployment/aws-load-balancer-controller
    
  4. use the second address from the ingress to browse to the app

     kubectl -n demo get ingress
    
     curl -s --header "Host: foo.bar" k8s-demo-djangoex-49f31c1921-782305710.us-east-2.elb.amazonaws.com | head
    

WAF time

  1. Create a WAF rule here https://console.aws.amazon.com/wafv2/homev2/web-acls/new and use the Core and SQL Injection rules. (make sure region matches us-east-2)

  2. View your WAF

     aws wafv2 list-web-acls --scope REGIONAL --region us-east-2 | jq .
    
  3. set the waf annotation to match the ARN provided above (and uncomment it) then re-apply the ingress

     kubectl apply -f ingress.yaml
    
  4. test the app still works

     curl -s --header "Host: foo.bar" --location "k8s-demo-djangoex-49f31c1921-782305710.us-east-2.elb.amazonaws.com"
    
  5. test the WAF denies a bad request

    You should get a 403 Forbidden error

     curl -X POST http://k8s-demo-djangoex-49f31c1921-782305710.us-east-2.elb.amazonaws.com -F "user='<script><alert>Hello></alert></script>'"