AWS

Deploying ROSA in STS mode

Tip The official documentation for installing a ROSA cluster in STS mode can be found here . Quick Introduction by Ryan Niksch (AWS) and Shaozen Ding (Red Hat) on YouTubeexternal link (opens in new tab) STS allows us to deploy ROSA without needing a ROSA admin account, instead it uses roles and policies with Amazon STS (secure token service) to gain access to the AWS resources needed to install and operate the cluster.

Migrating ROSA Ingress Controllers from a CLB to NLB

This guide will show you how to migrate the default Red Hat OpenShift Service on AWS (ROSA) IngressController from an AWS Classic Load Balancer to an AWS Network Load Balancer. In version 4.14 of ROSA, Red Hat introduced changes to IngressControllers to give customers more control over their workloads and configuration. The operation below requires a cluster running version 4.14 or higher. To request early access to this additional functionality in version 4.

Configuring AWS CLB Access Logging

This guide will show you how to enable access logging on the default Classic Load Balancer ingress controller used in Red Hat OpenShift Service on AWS (ROSA) version 4.13 and earlier. Prerequisites A ROSA Cluster (Version 4.13 or earlier) A logged in oc CLI A logged in aws CLI S3 Bucket Creation Run the following command, making sure to update the name of the S3 bucket you wish to create and the account number of the Elastic Load Balancing root account (this is not your AWS account):

Add an Ingress Controller to a ROSA Cluster and optionally with a custom domain.

Starting with OpenShift 4.14, ROSA supports adding additional Ingress Controllers which can use used to configure a custom domain on a ROSA cluster without having to use the now deprecated Custom Domain Operator. This guide shows how to add an additional Ingress Controller ( public or private ) to a ROSA cluster and optionally also configuring a custom domain. Prerequisites A Red Hat OpenShift on AWS (ROSA) cluster The oc CLI #logged in.

Cross-account Access using Custom OIDC Provider

Access AWS Cross Account resources using OIDC When employing ROSA, a common enterprise pattern involves establishing a cluster in a centralized AWS account while enabling development teams to manage services in their respective AWS accounts. This necessitates granting the ROSA cluster access to services residing in AWS accounts different from its own. Various approaches exist to address this challenge, but one straightforward method is to establish a secondary OIDC provider in the AWS account of the development team, enabling direct access for pods.

Customizing the console URL in ROSA

Starting with ROSA 4.14.X, it is possible to modify the hostname and TLS certificate of component Routes post-install. These are the OAuth, Console, and Downloads routes. For example, the default ROSA console uses the built-in domain https://console-openshift-console.apps.<cluster_name>.<random>.p1.openshiftapps.com. You can now specify a custom domain, for example test.example.com, and the ROSA console will be available at a URL such as https://console-openshift-console.test.example.com. This guide will walk you through how to customize the console url for a ROSA Classic cluster (not tested on ROSA HCP yet).

Setup a VPN Connection into a PrivateLink ROSA Cluster with OpenVPN

When you configure a Red Hat OpenShift on AWS (ROSA) cluster with a private link configuration, you will need connectivity to this private network in order to access your cluster. This guide will show you how to configute an AWS Client VPN connection so you won’t need to setup and configure Jump Boxes. Prerequisites a private link ROSA Cluster - follow this guide to create a private ROSA Cluster jq Set Envrionment Variables Start by setting environment variables that we will use to setup the VPN connection

Connect to RDS database with STS from ROSA

The Amazon Web Services Relational Database Service (AWS RDS) can be consumed from Red Hat OpenShift Service on AWS (ROSA) and authenticate to DB with Security Token Service (STS). This is a guide to quickly connect to RDS Database (Postgres engine) from ROSA. Amazon Web Services Relational Database Service Amazon Web Services Relational Database Service (AWS RDS) is a distributed relational database service by Amazon Web Services. It is designed to simplify setup, operation, and scaling of a relational database for use in applications.

Creating ROSA Components with GitOps

Many organizations want to use GitOps methodologies as a main part of their operational practices. Often times, this includes infrastructure as well. The advantage to this practice is that anything controlled in this manner can exist as infrastructure-as-code, by way of Kubernetes YAML definitions, in a centralized repository backed by Git. Additionally, all processes and procedures become a part of the Git workflow with a standardized Continuous Deployment pipeline controlling the outcome.

Using AWS Secrets Manager CSI on Red Hat OpenShift on AWS with STS

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on ROSA or OSD. This is made even easier and more secure through the use of AWS STS and Kubernetes PodIdentity. Prerequisites A ROSA cluster deployed with STS Helm 3 aws CLI oc CLI jq Preparing Environment Validate that your cluster has STS

Enabling the AWS EFS CSI Driver Operator on ROSA

The Amazon Web Services Elastic File System (AWS EFS) is a Network File System (NFS) that can be provisioned on Red Hat OpenShift Service on AWS clusters. With the release of OpenShift 4.10 the EFS CSI Driver is now GA and available. This is a guide to quickly enable the EFS Operator on ROSA to a Red Hat OpenShift on AWS (ROSA) cluster with STS enabled. Note: The official supported installation instructions for the EFS CSI Driver on ROSA are available here .

ROSA with Nvidia GPU Workloads

ROSA guide to running Nvidia GPU workloads. Prerequisites ROSA Cluster (4.10+) rosa cli #logged-in oc cli #logged-in-cluster-admin jq If you need to install a ROSA cluster, please read our ROSA Quickstart Guide . Please be sure you are installing or using an existing ROSA cluster that it is 4.10.x or higher. As of OpenShift 4.10, it is no longer necessary to set up entitlements to use the nVidia Operator. This has greatly simplified the setup of the cluster for GPU workloads.

External DNS for ROSA Custom Domain

Configuring the Custom Domain Operator requires a wildcard CNAME DNS record in your Route53 Hosted Zone. If you do not wish to use a wildcard record, you can use the External DNS Operator to create individual entries for routes. This document will guide you through deploying and configuring the External DNS Operator with a Custom Domain in ROSA. Important Note: The ExternalDNS Operator does not support STS yet and uses long lived IAM credentials.

AWS Load Balancer Operator On ROSA

AWS Load Balancer Controllerexternal link (opens in new tab) is a controller to help manage Elastic Load Balancers for a Kubernetes cluster. It satisfies Kubernetes Ingress resourcesexternal link (opens in new tab) by provisioning Application Load Balancersexternal link (opens in new tab) . It satisfies Kubernetes Service resourcesexternal link (opens in new tab) by provisioning Network Load Balancersexternal link (opens in new tab) . Compared with default AWS In Tree Provider, this controller is actively developed with advanced annotations for both ALBexternal link (opens in new tab) and NLBexternal link (opens in new tab) .

Dynamic Certificates for ROSA Custom Domain

There may be situations when you prefer not to use wild-card certificates. This ROSA guide talks about certificate management with cert-manager and letsencrypt, to dynamically issue certificates to routes created on a custom domain that’s hosted on AWS Route53. Prerequisites Set up environment Prepare AWS Account Set up cert-manager Create the Issuer and the Certficiate Configure Certificate Requestor Create the Certificate, which will later be used by the Custom Domain. Create the Custom Domain, which will be used to access your applications.

Configure a load balancer service to use a static public IP

This guide demonstrates how to create and assign a static public IP address to an OpenShift service in Azure Red Hat OpenShift (ARO). By default, the public IP address assigned to an OpenShift service with a type of LoadBalancer created by an ARO cluster is only valid for the lifespan of that resource. If you delete the OpenShift service, the associated load balancer and IP address are also deleted. If you want to assign a specific IP address or retain an IP address for redeployed OpenShift services, you can create and use a static public IP address.

Verify Permissions for ROSA STS Deployment

To proceed with the deployment of a ROSA cluster, an account must support the required roles and permissions. AWS Service Control Policies (SCPs) cannot block the API calls made by the installer or operator roles. Details about the IAM resources required for an STS-enabled installation of ROSA can be found here: https://docs.openshift.com/rosa/rosa_architecture/rosa-sts-about-iam-resources.html This guide is validated for ROSA v4.11.X. Prerequisites AWS CLIexternal link (opens in new tab) ROSA CLIexternal link (opens in new tab) v1.

STS OIDC in ROSA : How it works!

If you prefer a more visual medium, you can watch this video on YouTubeexternal link (opens in new tab) . This short video talks about how the STSexternal link (opens in new tab) OIDC flow work in ROSA (Red Hat OpenShift Service on AWS).

Security Reference Architecture for ROSA

The Security Reference Architecture for ROSA is a set of guidelines for deploying Red Hat OpenShift on AWS (ROSA) clusters to support high-security production workloads that align with Red Hat and AWS best practices. This overall architectural guidance compliments detailed, specific recommendations for AWS services and Red Hat OpenShift Container Platform. The Security Reference Architecture (SRA) for ROSA is a living document and is updated periodically based on new feature releases, customer feedback and evolving security best practices.

Deploying OpenShift API for Data Protection on a ROSA cluster

Prerequisites An STS enabled ROSA cluster Getting Started Create the following environment variables Change the cluster name to match your ROSA cluster and ensure you’re logged into the cluster as an Administrator. Ensure all fields are outputted correctly before moving on. export CLUSTER_NAME=my-cluster export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id) export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id) export OIDC_ENDPOINT=$(oc get authentication.

Custom AlertManager in ROSA 4.9.x

This page is deprecated. In order to get the best experience for custom alerting in ROSA, please upgrade your cluster to to 4.12 and follow the newer documentation. ROSA 4.9.x introduces a new way to provide custom AlertManager configuration to receive alerts from User Workload Management. The OpenShift Administrator can use the Prometheus Operator to create a custom AlertManager resource and then use the AlertManagerConfig resource to configure User Workload Monitoring to use the custom AlertManager.

Configure ROSA/OSD to use custom TLS ciphers on the ingress controllers

This guide demonstrates how to properly patch the cluster ingress controllers, as well as ingress controllers created by the Custom Domain Operator. This functionality allows customers to modify the tlsSecurityProfile value on cluster ingress controllers. This guide will demonstrate how to apply a custom tlsSecurityProfile, a scoped service account (with the associated role and role binding), and a CronJob that the cipher changes are reapplied with 60 minutes (in the event that an ingress controller is recreated or modified).

Configuring the Cluster Log Forwarder for CloudWatch Logs and STS

This guide shows how to deploy the Cluster Log Forwarder operator and configure it to use STS authentication to forward logs to CloudWatch. Prerequisites A ROSA cluster (configured with STS) The jq cli command The aws cli command Environment Setup Configure the following environment variables Change the cluster name to match your ROSA cluster and ensure you’re logged into the cluster as an Administrator. Ensure all fields are outputted correctly before moving on.

Using AWS Controllers for Kubernetes (ACK) on ROSA

AWS Controllers for Kubernetesexternal link (opens in new tab) (ACK) lets you define and use AWS service resources directly from Kubernetes. With ACK, you can take advantage of AWS-managed services for your Kubernetes applications without needing to define resources outside of the cluster or run services that provide supporting capabilities like databases or message queues within the cluster. ROSA clusters have a set of the ACK controllers in Operator Hub which makes it relatively easy to get started and use it.

Create IAM user and Policy

Notes: These are sample commands. Please fill in your own resource parameters E.g. ARN Create the policy cat <<EOF > /tmp/iam_policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken" ], "Resource": "*" } ] } EOF aws iam create-policy \ --policy-name ECRLoginPolicy \ --policy-document file:///tmp/iam_policy.json Create a user and access key and attach the policy aws iam create-user --user-name ecr-bot aws iam create-access-key --user-name ecr-bot aws iam attach-user-policy --policy-arn arn:aws:iam::[ACCOUNT_ID]:policy/ECRLoginPolicy --user-name ecr-bot Notes: Save access key id and key for later usage

Create STS Assume Role

About AWS STS and Assume Roleexternal link (opens in new tab) Notes: These are sample commands. Please fill in your own resource parameters E.g. ARN Prequisites An STS Openshift Cluster Setup Environment Variables export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json \ | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///") export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export REPOSITORY_NAME=test Create the policy cat <<EOF > /tmp/iam_policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken" ], "Resource": "*" } ] } EOF aws iam create-policy \ --policy-name ECRLoginPolicy \ --policy-document file:///tmp/iam_policy.

ECR Secret Operator

Amazon Elastic Container Registry Private Registry Authenticationexternal link (opens in new tab) provides a temporary authorization token valid only for 12 hours. This operator refreshes automatically the Amazon ECR authorization token before it expires, reducing the overhead in managing the authentication flow. This operator contains two Custom Resources which direct the operator to generate/refresh Amazon ECR authorization token in a timely manner: Image Pull Secret APIexternal link (opens in new tab) Argo CD Repo Helm Chart Secretexternal link (opens in new tab) How to use this operator Prerequisites Create an ECR private repositoryexternal link (opens in new tab) Provide AWS Authentication to the operator.

Adding a Public Ingress endpoint to a ROSA PrivateLink Cluster

This is an example guide for creating a public ingress endpoint for a ROSA Private-Link cluster. Be aware of the security implications of creating a public subnet in your ROSA VPC this way. Refer to the blog “How to add public Ingress to a PrivateLink ROSA cluster” , to expose applications to the internet by deploying in a PrivateLink Red Hat OpenShift Service on AWS (ROSA) cluster within a truly private Virtual Private Cloud (VPC) that doesn’t have an internet gateway attached to it.

Configuring a ROSA cluster to pull images from AWS Elastic Container Registry (ECR)

Prerequisites AWS CLIexternal link (opens in new tab) Openshift CLI 4.11+ Podman Desktopexternal link (opens in new tab) ROSA Clusterexternal link (opens in new tab) Note your ROSA cluster must be a classic STS cluster Background Quick Introduction by Ryan Niksch & Charlotte Fung on YouTubeexternal link (opens in new tab) . There are two options to use to authenticate wth Amazon ECR to pull images. The traditional method is to create a pull secret for ecr.

Creating a ROSA cluster in STS mode with custom KMS key

Tip Official Documentation ROSA STS with custom KMS key This guide will walk you through installing ROSA (Red Hat OpenShift Service on AWS) with a customer-provided KMS key that will be used to encrypt both the root volumes of nodes as well as persistent volumes for mounted EBS claims. Prerequisites AWS CLIexternal link (opens in new tab) ROSA CLIexternal link (opens in new tab) v1.1.11 or higher OpenShift CLI - rosa download openshift-client Prepare AWS Account for ROSA Configure the AWS CLI by running the following command

ROSA - Federating Metrics to AWS Prometheus

Federating Metrics from ROSA/OSD is a bit tricky as the cluster metrics require pulling from its /federated endpoint while the user workload metrics require using the prometheus remoteWrite configuration. This guide will walk you through using the MOBB Helm Chart to deploy the necessary agents to federate the metrics into AWS Prometheus and then use Grafana to visualize those metrics. As a bonus it will set up a CloudWatch datasource to view any metrics or logs you have in Cloud Watch.

Configuring IDP for ROSA, OSD and ARO

Red Hat OpenShift on AWS (ROSA) and OpenShift Dedicated (OSD) provide a simple way for the cluster administrator to configure one or more identity providers for their cluster[s] via the OpenShift Cluster Manager (OCM) , while Azure Red Hat OpenShift relies on the internal cluster authentication operatorexternal link (opens in new tab) . The identity providers available for use are: GitHub GitLab Google LDAP OpenID HTPasswd Configuring Specific Identity Providers ARO GitLab Azure AD Azure AD with Group Claims Azure AD via CLI Azure AD with Red Hat SSO ROSA/OSD GitLab Azure AD Azure AD with Group Claims (ROSA Only) Configuring Group Synchronization Using Group Sync Operator with Azure Active Directory and ROSA/OSD Using Group Sync Operator with Okta and ROSA/OSD

Federating Metrics to a centralized Prometheus Cluster

This document has been removed as it was written for older ROSA clusters which did not allow for custom Alert Manager configs as a way to provide a second Prometheus with a configurable Alert Manager. If you want to configure custom Alerts, you can upgrade your cluster and follow the steps found at Custom Alerts in ROSA 4.11.x . If you want to federate your metrics to a central location we recommend using one of the following:

Custom Alerts in ROSA 4.11.x

Starting with OpenShift 4.11 it is possible to manage alerting rules for user-defined projects . Similarly, in ROSA clusters the OpenShift Administrator can enable a second AlertManager instance in the user workload monitoring namespace which can be used to create such alerts. Note: Currently this is not a managed feature of ROSA. Such an implementation may get overwritten if the User Workload Monitoring functionality is toggled off and on using the OpenShift Cluster Manager (OCM).

Extending ROSA STS to include authentication with AWS Services

In this example we will deploy the Amazon Ingress Controller that uses ALBs, and configure it to use STS authentication. Deployment Configure STS Make sure your cluster has the pod identity webhook kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io pod-identity-webhook Download the IAM Policy for the AWS Load Balancer Hooks wget https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json Create AWS Role with inline policy aws iam create-role \ --role-name AWSLoadBalancerController --query Policy.Arn --output text Create AWS Policy and Service Account

Integrating with AWS resources using Pod Identity

Prerequisites ROSA CLI AWS CLI ROSA Cluster with STS

Using the AWS Cloud Watch agent to publish metrics to CloudWatch in ROSA

This document shows how you can use the AWS Cloud Watch agent to scrape Prometheus endpoints and publish metrics to CloudWatch in a Red Hat OpenShift Container Platform (ROSA) cluster. It pulls from The AWS documentation for installing the CloudWatch agent to Kubernetes and collections and publishes metrics for the Kubernetes API Server and provides a simple Dashboard to view the results. Currently the AWS Cloud Watch Agent does not supportexternal link (opens in new tab) pulling all metrics from the Prometheus federated endpoint, but the hope is that when it does we can ship all Cluster and User Workload metrics to CloudWatch.

Creating a ROSA cluster with PrivateLink enabled (custom VPC) and STS

This is a combination of the private-link and sts setup documents to show the full picture Prerequisites AWS CLIexternal link (opens in new tab) Rosa CLIexternal link (opens in new tab) v1.1.7 jqexternal link (opens in new tab) AWS Preparation If this is a brand new AWS account that has never had a AWS Load Balancer installed in it, you should run the following aws iam create-service-linked-role --aws-service-name \ "elasticloadbalancing.

AWS ALB

Note: It is recommended that you use the Cloud Front based guide unless you absolutely must use an ALB based solution. Hereexternal link (opens in new tab) ’s a good overview of AWS LB types and what they support Problem Statement Operator requires WAF (Web Application Firewall) in front of their workloads running on OpenShift (ROSA) Operator does not want WAF running on OpenShift to ensure that OCP resources do not experience Denial of Service through handling the WAF

Examples of using a WAF in front of ROSA / OSD on AWS / OCP on AWS

Problem Statement Operator requires WAF (Web Application Firewall) in front of their workloads running on OpenShift (ROSA) Operator does not want WAF running on OpenShift to ensure that OCP resources do not experience Denial of Service through handling the WAF Quick Introduction by Paul Czarkowskiexternal link (opens in new tab) & Ryan Niksch on YouTubeexternal link (opens in new tab) Solutions Cloud Front -> WAF -> CustomDomain -> $APP This is the preferred method and can also work with most third party WAF systems that act as a reverse proxy

Using CloudFront + WAF

Problem Statement Operator requires WAF (Web Application Firewall) in front of their workloads running on OpenShift (ROSA) Operator does not want WAF running on OpenShift to ensure that OCP resources do not experience Denial of Service through handling the WAF Proposed Solution Add a CustomDomain resource to the cluster using a wildcard DNS and TLS certificate. Set the Wildcard DNS CNAME’s to CloudFront and enable the CloudFront + WAF services to reverse proxy and inspect the traffic before sending it to the cluster.

Creating a ROSA cluster with PrivateLink enabled

Prerequisites AWS CLIexternal link (opens in new tab) Rosa CLIexternal link (opens in new tab) v1.0.8 jqexternal link (opens in new tab) Create VPC and Subnets The following instructions use the AWS CLI to create the necessary networking to deploy a PrivateLink ROSA cluster into a Single AZ and are intended to be a guide. Ideally you would use an Automation tool like Ansible or Terraform to manage your VPCs.

Federating System and User metrics to S3 in Red Hat OpenShift for AWS

This guide walks through setting up federating Prometheus metrics to S3 storage. ToDo - Add Authorization in front of Thanos APIs Prerequisites A ROSA cluster deployed with STS aws CLI Set up environment Create environment variables export CLUSTER_NAME=my-cluster export S3_BUCKET=my-thanos-bucket export REGION=us-east-2 export NAMESPACE=federated-metrics export SA=aws-prometheus-proxy export SCRATCH_DIR=/tmp/scratch export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///") export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export AWS_PAGER="" rm -rf $SCRATCH_DIR mkdir -p $SCRATCH_DIR Create namespace

AWS

Interested in contributing to these docs?

Products

Tools

Try, buy & sell

Communicate

About Red Hat

Subscribe to our newsletter, Red Hat Shares

Red Hat legal and privacy links

Red Hat legal and privacy links