Chapter 6. Tutorial: Using AWS WAF and Amazon CloudFront to protect ROSA workloads
AWS WAF is a web application firewall that lets you monitor the HTTP and HTTPS requests that are forwarded to your protected web application resources.
You can use an Amazon CloudFront to add a Web Application Firewall (WAF) to your Red Hat OpenShift Service on AWS (ROSA) workloads. Using an external solution protects ROSA resources from experiencing denial of service due to handling the WAF.
6.1. Prerequisites
- A ROSA (HCP or Classic) cluster.
-
You have access to the OpenShift CLI (
oc
). -
You have access to the AWS CLI (
aws
).
6.1.1. Environment setup
Prepare the environment variables:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow export DOMAIN=apps.example.com export AWS_PAGER="" export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}$//') export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export SCRATCH="/tmp/${CLUSTER}/cloudfront-waf" mkdir -p ${SCRATCH} echo "Cluster: ${CLUSTER}, Region: ${REGION}, AWS Account ID: ${AWS_ACCOUNT_ID}"
$ export DOMAIN=apps.example.com
1 $ export AWS_PAGER="" $ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}$//') $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) $ export SCRATCH="/tmp/${CLUSTER}/cloudfront-waf" $ mkdir -p ${SCRATCH} $ echo "Cluster: ${CLUSTER}, Region: ${REGION}, AWS Account ID: ${AWS_ACCOUNT_ID}"
- 1
- Replace with the custom domain you want to use for the
IngressController
.
NoteThe "Cluster" output from the previous command might be the name of your cluster, the internal ID of your cluster, or the cluster’s domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow export CLUSTER=my-custom-value
$ export CLUSTER=my-custom-value
6.2. Setting up the secondary ingress controller
It is necessary to configure a secondary ingress controller to segment your external WAF-protected traffic from your standard (and default) cluster ingress controller.
Prerequisites
A publicly trusted SAN or wildcard certificate for your custom domain, such as
CN=*.apps.example.com
ImportantAmazon CloudFront uses HTTPS to communicate with your cluster’s secondary ingress controller. As explained in the Amazon CloudFront documentation, you cannot use a self-signed certificate for HTTPS communication between CloudFront and your cluster. Amazon CloudFront verifies that the certificate was issued by a trusted certificate authority.
Procedure
Create a new TLS secret from a private key and a public certificate, where
fullchain.pem
is your full wildcard certificate chain (including any intermediaries) andprivkey.pem
is your wildcard certificate’s private key.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n openshift-ingress create secret tls waf-tls --cert=fullchain.pem --key=privkey.pem
$ oc -n openshift-ingress create secret tls waf-tls --cert=fullchain.pem --key=privkey.pem
Create a new
IngressController
resource:Example
waf-ingress-controller.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: cloudfront-waf namespace: openshift-ingress-operator spec: domain: apps.example.com defaultCertificate: name: waf-tls endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Unmanaged providerParameters: aws: type: NLB type: AWS scope: External type: LoadBalancerService routeSelector: matchLabels: route: waf
apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: cloudfront-waf namespace: openshift-ingress-operator spec: domain: apps.example.com
1 defaultCertificate: name: waf-tls endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Unmanaged providerParameters: aws: type: NLB type: AWS scope: External type: LoadBalancerService routeSelector:
2 matchLabels: route: waf
Apply the
IngressController
:Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc apply -f waf-ingress-controller.yaml
$ oc apply -f waf-ingress-controller.yaml
Verify that your IngressController has successfully created an external load balancer:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n openshift-ingress get service/router-cloudfront-waf
$ oc -n openshift-ingress get service/router-cloudfront-waf
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE router-cloudfront-waf LoadBalancer 172.30.16.141 a68a838a7f26440bf8647809b61c4bc8-4225395f488830bd.elb.us-east-1.amazonaws.com 80:30606/TCP,443:31065/TCP 2m19s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE router-cloudfront-waf LoadBalancer 172.30.16.141 a68a838a7f26440bf8647809b61c4bc8-4225395f488830bd.elb.us-east-1.amazonaws.com 80:30606/TCP,443:31065/TCP 2m19s
6.2.1. Configure the AWS WAF
The AWS WAF service is a web application firewall that lets you monitor, protect, and control the HTTP and HTTPS requests that are forwarded to your protected web application resources, like ROSA.
Create a AWS WAF rules file to apply to our web ACL:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat << EOF > ${SCRATCH}/waf-rules.json [ { "Name": "AWS-AWSManagedRulesCommonRuleSet", "Priority": 0, "Statement": { "ManagedRuleGroupStatement": { "VendorName": "AWS", "Name": "AWSManagedRulesCommonRuleSet" } }, "OverrideAction": { "None": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "AWS-AWSManagedRulesCommonRuleSet" } }, { "Name": "AWS-AWSManagedRulesSQLiRuleSet", "Priority": 1, "Statement": { "ManagedRuleGroupStatement": { "VendorName": "AWS", "Name": "AWSManagedRulesSQLiRuleSet" } }, "OverrideAction": { "None": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "AWS-AWSManagedRulesSQLiRuleSet" } } ] EOF
$ cat << EOF > ${SCRATCH}/waf-rules.json [ { "Name": "AWS-AWSManagedRulesCommonRuleSet", "Priority": 0, "Statement": { "ManagedRuleGroupStatement": { "VendorName": "AWS", "Name": "AWSManagedRulesCommonRuleSet" } }, "OverrideAction": { "None": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "AWS-AWSManagedRulesCommonRuleSet" } }, { "Name": "AWS-AWSManagedRulesSQLiRuleSet", "Priority": 1, "Statement": { "ManagedRuleGroupStatement": { "VendorName": "AWS", "Name": "AWSManagedRulesSQLiRuleSet" } }, "OverrideAction": { "None": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "AWS-AWSManagedRulesSQLiRuleSet" } } ] EOF
This will enable the Core (Common) and SQL AWS Managed Rule Sets.
Create an AWS WAF Web ACL using the rules we specified above:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow WAF_WACL=$(aws wafv2 create-web-acl \ --name cloudfront-waf \ --region ${REGION} \ --default-action Allow={} \ --scope CLOUDFRONT \ --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=${CLUSTER}-waf-metrics \ --rules file://${SCRATCH}/waf-rules.json \ --query 'Summary.Name' \ --output text)
$ WAF_WACL=$(aws wafv2 create-web-acl \ --name cloudfront-waf \ --region ${REGION} \ --default-action Allow={} \ --scope CLOUDFRONT \ --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=${CLUSTER}-waf-metrics \ --rules file://${SCRATCH}/waf-rules.json \ --query 'Summary.Name' \ --output text)
6.3. Configure Amazon CloudFront
Retrieve the newly created custom ingress controller’s NLB hostname:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NLB=$(oc -n openshift-ingress get service router-cloudfront-waf \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
$ NLB=$(oc -n openshift-ingress get service router-cloudfront-waf \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
Import your certificate into Amazon Certificate Manager, where
cert.pem
is your wildcard certificate,fullchain.pem
is your wildcard certificate’s chain andprivkey.pem
is your wildcard certificate’s private key.NoteRegardless of what region your cluster is deployed, you must import this certificate to
us-east-1
as Amazon CloudFront is a global AWS service.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow aws acm import-certificate --certificate file://cert.pem \ --certificate-chain file://fullchain.pem \ --private-key file://privkey.pem \ --region us-east-1
$ aws acm import-certificate --certificate file://cert.pem \ --certificate-chain file://fullchain.pem \ --private-key file://privkey.pem \ --region us-east-1
- Log into the AWS console to create a CloudFront distribution.
Configure the CloudFront distribution by using the following information:
NoteIf an option is not specified in the table below, leave them the default (which may be blank).
Option Value Origin domain
Output from the previous command [1]
Name
rosa-waf-ingress [2]
Viewer protocol policy
Redirect HTTP to HTTPS
Allowed HTTP methods
GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
Cache policy
CachingDisabled
Origin request policy
AllViewer
Web Application Firewall (WAF)
Enable security protections
Use existing WAF configuration
true
Choose a web ACL
cloudfront-waf
Alternate domain name (CNAME)
*.apps.example.com [3]
Custom SSL certificate
Select the certificate you imported from the step above [4]
-
Run
echo ${NLB}
to get the origin domain. - If you have multiple clusters, ensure the origin name is unique.
- This should match the wildcard domain you used to create the custom ingress controller.
- This should match the alternate domain name entered above.
-
Run
Retrieve the Amazon CloudFront Distribution endpoint:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow aws cloudfront list-distributions --query "DistributionList.Items[?Origins.Items[?DomainName=='${NLB}']].DomainName" --output text
$ aws cloudfront list-distributions --query "DistributionList.Items[?Origins.Items[?DomainName=='${NLB}']].DomainName" --output text
Update the DNS of your custom wildcard domain with a CNAME to the Amazon CloudFront Distribution endpoint from the step above.
Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow *.apps.example.com CNAME d1b2c3d4e5f6g7.cloudfront.net
*.apps.example.com CNAME d1b2c3d4e5f6g7.cloudfront.net
6.4. Deploy a sample application
Create a new project for your sample application by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc new-project hello-world
$ oc new-project hello-world
Deploy a hello world application:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n hello-world new-app --image=docker.io/openshift/hello-openshift
$ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift
Create a route for the application specifying your custom domain name:
Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls \ --hostname hello-openshift.${DOMAIN}
$ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls \ --hostname hello-openshift.${DOMAIN}
Label the route to admit it to your custom ingress controller:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n hello-world label route.route.openshift.io/hello-openshift-tls route=waf
$ oc -n hello-world label route.route.openshift.io/hello-openshift-tls route=waf
6.5. Test the WAF
Test that the app is accessible behind Amazon CloudFront:
Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow curl "https://hello-openshift.${DOMAIN}"
$ curl "https://hello-openshift.${DOMAIN}"
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Hello OpenShift!
Hello OpenShift!
Test that the WAF denies a bad request:
Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow curl -X POST "https://hello-openshift.${DOMAIN}" \ -F "user='<script><alert>Hello></alert></script>'"
$ curl -X POST "https://hello-openshift.${DOMAIN}" \ -F "user='<script><alert>Hello></alert></script>'"
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <TITLE>ERROR: The request could not be satisfied</TITLE> </HEAD><BODY> <H1>403 ERROR</H1> <H2>The request could not be satisfied.</H2> <HR noshade size="1px"> Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. <BR clear="all"> If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation. <BR clear="all"> <HR noshade size="1px"> <PRE> Generated by cloudfront (CloudFront) Request ID: nFk9q2yB8jddI6FZOTjdliexzx-FwZtr8xUQUNT75HThPlrALDxbag== </PRE> <ADDRESS> </ADDRESS> </BODY></HTML>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <TITLE>ERROR: The request could not be satisfied</TITLE> </HEAD><BODY> <H1>403 ERROR</H1> <H2>The request could not be satisfied.</H2> <HR noshade size="1px"> Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. <BR clear="all"> If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation. <BR clear="all"> <HR noshade size="1px"> <PRE> Generated by cloudfront (CloudFront) Request ID: nFk9q2yB8jddI6FZOTjdliexzx-FwZtr8xUQUNT75HThPlrALDxbag== </PRE> <ADDRESS> </ADDRESS> </BODY></HTML>
The expected result is a
403 ERROR
, which means the AWS WAF is protecting your application.