Cloud Experts Documentation

Red Hat OpenShift Service on AWS

Red Hat OpenShift Service on AWS (ROSA) is a fully-managed turnkey application platform that allows you to focus on what matters most, delivering value to your customers by building and deploying applications. Red Hat and AWS site reliability engineering (SRE) experts manage the underlying platform so you don’t have to worry about the complexity of infrastructure management.

Deploying ROSA in STS mode

Tip The official documentation for installing a ROSA cluster in STS mode can be found here . Quick Introduction by Ryan Niksch (AWS) and Shaozen Ding (Red Hat) on YouTubeexternal link (opens in new tab) STS allows us to deploy ROSA without needing a ROSA admin account, instead it uses roles and policies with Amazon STS (secure token service) to gain access to the AWS resources needed to install and operate the cluster.

Migrating ROSA Ingress Controllers from a CLB to NLB

This guide will show you how to migrate the default Red Hat OpenShift Service on AWS (ROSA) IngressController from an AWS Classic Load Balancer to an AWS Network Load Balancer. In version 4.14 of ROSA, Red Hat introduced changes to IngressControllers to give customers more control over their workloads and configuration. The operation below requires a cluster running version 4.14 or higher. To request early access to this additional functionality in version 4.

Configuring AWS CLB Access Logging

This guide will show you how to enable access logging on the default Classic Load Balancer ingress controller used in Red Hat OpenShift Service on AWS (ROSA) version 4.13 and earlier. Prerequisites A ROSA Cluster (Version 4.13 or earlier) A logged in oc CLI A logged in aws CLI S3 Bucket Creation Run the following command, making sure to update the name of the S3 bucket you wish to create and the account number of the Elastic Load Balancing root account (this is not your AWS account):

Add an Ingress Controller to a ROSA Cluster and optionally with a custom domain.

Starting with OpenShift 4.14, ROSA supports adding additional Ingress Controllers which can use used to configure a custom domain on a ROSA cluster without having to use the now deprecated Custom Domain Operator. This guide shows how to add an additional Ingress Controller ( public or private ) to a ROSA cluster and optionally also configuring a custom domain. Prerequisites A Red Hat OpenShift on AWS (ROSA) cluster The oc CLI #logged in.

Cross-account Access using Custom OIDC Provider

Access AWS Cross Account resources using OIDC When employing ROSA, a common enterprise pattern involves establishing a cluster in a centralized AWS account while enabling development teams to manage services in their respective AWS accounts. This necessitates granting the ROSA cluster access to services residing in AWS accounts different from its own. Various approaches exist to address this challenge, but one straightforward method is to establish a secondary OIDC provider in the AWS account of the development team, enabling direct access for pods.

Customizing the console URL in ROSA

Starting with ROSA 4.14.X, it is possible to modify the hostname and TLS certificate of component Routes post-install. These are the OAuth, Console, and Downloads routes. For example, the default ROSA console uses the built-in domain https://console-openshift-console.apps.<cluster_name>.<random>.p1.openshiftapps.com. You can now specify a custom domain, for example test.example.com, and the ROSA console will be available at a URL such as https://console-openshift-console.test.example.com. This guide will walk you through how to customize the console url for a ROSA Classic cluster (not tested on ROSA HCP yet).

ROSA Break Glass Troubleshooting

Background WARNING: this procedure should only be initiated by a member of the Black Belt team or someone incredibly familiar with ROSA as a whole. THIS IS NOT COMMON!!! This guide shows how to access ROSA instances in the situation that a break glass scenario is required in the account where ROSA is deployed. This procedure should only be performed under unusual circumstances like a failed provision in order to collect logs.

Setup a VPN Connection into a PrivateLink ROSA Cluster with OpenVPN

When you configure a Red Hat OpenShift on AWS (ROSA) cluster with a private link configuration, you will need connectivity to this private network in order to access your cluster. This guide will show you how to configute an AWS Client VPN connection so you won’t need to setup and configure Jump Boxes. Prerequisites a private link ROSA Cluster - follow this guide to create a private ROSA Cluster jq Set Envrionment Variables Start by setting environment variables that we will use to setup the VPN connection

Prerequisites Checklist to Deploy ROSA Cluster with STS

Background This is a quick checklist of prerequisites needed to spin up a classic Red Hat OpenShift Service on AWS (ROSA) cluster with STSexternal link (opens in new tab) . Note that this is a high level checklist and your implementation may vary. Before running the installation process, make sure that you deploy this from a machine that has access to: The API services for the cloud to which you provision.

Connect to RDS database with STS from ROSA

The Amazon Web Services Relational Database Service (AWS RDS) can be consumed from Red Hat OpenShift Service on AWS (ROSA) and authenticate to DB with Security Token Service (STS). This is a guide to quickly connect to RDS Database (Postgres engine) from ROSA. Amazon Web Services Relational Database Service Amazon Web Services Relational Database Service (AWS RDS) is a distributed relational database service by Amazon Web Services. It is designed to simplify setup, operation, and scaling of a relational database for use in applications.

Deploying ROSA PrivateLink Cluster with Ansible

Background This guide shows an example of how to deploy a classic Red Hat OpenShift Services on AWS (ROSA) cluster with PrivateLinkexternal link (opens in new tab) with STSexternal link (opens in new tab) enabled using Ansibleexternal link (opens in new tab) playbook from our MOBB GitHub repositoryexternal link (opens in new tab) and makefilesexternal link (opens in new tab) to compile them. Note that this is an unofficial Red Hat guide and your implementation may vary.

Creating ROSA Components with GitOps

Many organizations want to use GitOps methodologies as a main part of their operational practices. Often times, this includes infrastructure as well. The advantage to this practice is that anything controlled in this manner can exist as infrastructure-as-code, by way of Kubernetes YAML definitions, in a centralized repository backed by Git. Additionally, all processes and procedures become a part of the Git workflow with a standardized Continuous Deployment pipeline controlling the outcome.

Using AWS Secrets Manager CSI on Red Hat OpenShift on AWS with STS

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on ROSA or OSD. This is made even easier and more secure through the use of AWS STS and Kubernetes PodIdentity. Prerequisites A ROSA cluster deployed with STS Helm 3 aws CLI oc CLI jq Preparing Environment Validate that your cluster has STS

Enabling the AWS EFS CSI Driver Operator on ROSA

The Amazon Web Services Elastic File System (AWS EFS) is a Network File System (NFS) that can be provisioned on Red Hat OpenShift Service on AWS clusters. With the release of OpenShift 4.10 the EFS CSI Driver is now GA and available. This is a guide to quickly enable the EFS Operator on ROSA to a Red Hat OpenShift on AWS (ROSA) cluster with STS enabled. Note: The official supported installation instructions for the EFS CSI Driver on ROSA are available here .

Assign Consistent Egress IP for External Traffic

It may be desirable to assign a consistent IP address for traffic that leaves the cluster when configuring items such as security groups or other sorts of security controls which require an IP-based configuration. By default, Kubernetes via the OVN-Kubernetes CNI will assign random IP addresses from a pool which will make configuring security lockdowns unpredictable or unnecessarily open. This guide shows you how to configure a set of predictable IP addresses for egress cluster traffic to meet common security standards and guidance and other potential use cases.

ROSA with Nvidia GPU Workloads

ROSA guide to running Nvidia GPU workloads. Prerequisites ROSA Cluster (4.10+) rosa cli #logged-in oc cli #logged-in-cluster-admin jq If you need to install a ROSA cluster, please read our ROSA Quickstart Guide . Please be sure you are installing or using an existing ROSA cluster that it is 4.10.x or higher. As of OpenShift 4.10, it is no longer necessary to set up entitlements to use the nVidia Operator. This has greatly simplified the setup of the cluster for GPU workloads.

External DNS for ROSA Custom Domain

Configuring the Custom Domain Operator requires a wildcard CNAME DNS record in your Route53 Hosted Zone. If you do not wish to use a wildcard record, you can use the External DNS Operator to create individual entries for routes. This document will guide you through deploying and configuring the External DNS Operator with a Custom Domain in ROSA. Important Note: The ExternalDNS Operator does not support STS yet and uses long lived IAM credentials.

AWS Load Balancer Operator On ROSA

AWS Load Balancer Controllerexternal link (opens in new tab) is a controller to help manage Elastic Load Balancers for a Kubernetes cluster. It satisfies Kubernetes Ingress resourcesexternal link (opens in new tab) by provisioning Application Load Balancersexternal link (opens in new tab) . It satisfies Kubernetes Service resourcesexternal link (opens in new tab) by provisioning Network Load Balancersexternal link (opens in new tab) . Compared with default AWS In Tree Provider, this controller is actively developed with advanced annotations for both ALBexternal link (opens in new tab) and NLBexternal link (opens in new tab) .

Dynamic Certificates for ROSA Custom Domain

There may be situations when you prefer not to use wild-card certificates. This ROSA guide talks about certificate management with cert-manager and letsencrypt, to dynamically issue certificates to routes created on a custom domain that’s hosted on AWS Route53. Prerequisites Set up environment Prepare AWS Account Set up cert-manager Create the Issuer and the Certficiate Configure Certificate Requestor Create the Certificate, which will later be used by the Custom Domain. Create the Custom Domain, which will be used to access your applications.

Verify Permissions for ROSA STS Deployment

To proceed with the deployment of a ROSA cluster, an account must support the required roles and permissions. AWS Service Control Policies (SCPs) cannot block the API calls made by the installer or operator roles. Details about the IAM resources required for an STS-enabled installation of ROSA can be found here: https://docs.openshift.com/rosa/rosa_architecture/rosa-sts-about-iam-resources.html This guide is validated for ROSA v4.11.X. Prerequisites AWS CLIexternal link (opens in new tab) ROSA CLIexternal link (opens in new tab) v1.

STS OIDC in ROSA : How it works!

If you prefer a more visual medium, you can watch this video on YouTubeexternal link (opens in new tab) . This short video talks about how the STSexternal link (opens in new tab) OIDC flow work in ROSA (Red Hat OpenShift Service on AWS).

Security Reference Architecture for ROSA

The Security Reference Architecture for ROSA is a set of guidelines for deploying Red Hat OpenShift on AWS (ROSA) clusters to support high-security production workloads that align with Red Hat and AWS best practices. This overall architectural guidance compliments detailed, specific recommendations for AWS services and Red Hat OpenShift Container Platform. The Security Reference Architecture (SRA) for ROSA is a living document and is updated periodically based on new feature releases, customer feedback and evolving security best practices.

Custom AlertManager in ROSA 4.9.x

This page is deprecated. In order to get the best experience for custom alerting in ROSA, please upgrade your cluster to to 4.12 and follow the newer documentation. ROSA 4.9.x introduces a new way to provide custom AlertManager configuration to receive alerts from User Workload Management. The OpenShift Administrator can use the Prometheus Operator to create a custom AlertManager resource and then use the AlertManagerConfig resource to configure User Workload Monitoring to use the custom AlertManager.

Configuring the Cluster Log Forwarder for CloudWatch Logs and STS

This guide shows how to deploy the Cluster Log Forwarder operator and configure it to use STS authentication to forward logs to CloudWatch. Prerequisites A ROSA cluster (configured with STS) The jq cli command The aws cli command Environment Setup Configure the following environment variables Change the cluster name to match your ROSA cluster and ensure you’re logged into the cluster as an Administrator. Ensure all fields are outputted correctly before moving on.

Using AWS Controllers for Kubernetes (ACK) on ROSA

AWS Controllers for Kubernetesexternal link (opens in new tab) (ACK) lets you define and use AWS service resources directly from Kubernetes. With ACK, you can take advantage of AWS-managed services for your Kubernetes applications without needing to define resources outside of the cluster or run services that provide supporting capabilities like databases or message queues within the cluster. ROSA clusters have a set of the ACK controllers in Operator Hub which makes it relatively easy to get started and use it.

ECR Secret Operator

Amazon Elastic Container Registry Private Registry Authenticationexternal link (opens in new tab) provides a temporary authorization token valid only for 12 hours. This operator refreshes automatically the Amazon ECR authorization token before it expires, reducing the overhead in managing the authentication flow. This operator contains two Custom Resources which direct the operator to generate/refresh Amazon ECR authorization token in a timely manner: Image Pull Secret APIexternal link (opens in new tab) Argo CD Repo Helm Chart Secretexternal link (opens in new tab) How to use this operator Prerequisites Create an ECR private repositoryexternal link (opens in new tab) Provide AWS Authentication to the operator.

Configuring a ROSA cluster to pull images from AWS Elastic Container Registry (ECR)

Prerequisites AWS CLIexternal link (opens in new tab) Openshift CLI 4.11+ Podman Desktopexternal link (opens in new tab) ROSA Clusterexternal link (opens in new tab) Note your ROSA cluster must be a classic STS cluster Background Quick Introduction by Ryan Niksch & Charlotte Fung on YouTubeexternal link (opens in new tab) . There are two options to use to authenticate wth Amazon ECR to pull images. The traditional method is to create a pull secret for ecr.

Creating a ROSA cluster in STS mode with custom KMS key

Tip Official Documentation ROSA STS with custom KMS key This guide will walk you through installing ROSA (Red Hat OpenShift Service on AWS) with a customer-provided KMS key that will be used to encrypt both the root volumes of nodes as well as persistent volumes for mounted EBS claims. Prerequisites AWS CLIexternal link (opens in new tab) ROSA CLIexternal link (opens in new tab) v1.1.11 or higher OpenShift CLI - rosa download openshift-client Prepare AWS Account for ROSA Configure the AWS CLI by running the following command

ROSA - Federating Metrics to AWS Prometheus

Federating Metrics from ROSA/OSD is a bit tricky as the cluster metrics require pulling from its /federated endpoint while the user workload metrics require using the prometheus remoteWrite configuration. This guide will walk you through using the MOBB Helm Chart to deploy the necessary agents to federate the metrics into AWS Prometheus and then use Grafana to visualize those metrics. As a bonus it will set up a CloudWatch datasource to view any metrics or logs you have in Cloud Watch.

Federating Metrics to a centralized Prometheus Cluster

This document has been removed as it was written for older ROSA clusters which did not allow for custom Alert Manager configs as a way to provide a second Prometheus with a configurable Alert Manager. If you want to configure custom Alerts, you can upgrade your cluster and follow the steps found at Custom Alerts in ROSA 4.11.x . If you want to federate your metrics to a central location we recommend using one of the following:

Custom Alerts in ROSA 4.11.x

Starting with OpenShift 4.11 it is possible to manage alerting rules for user-defined projects . Similarly, in ROSA clusters the OpenShift Administrator can enable a second AlertManager instance in the user workload monitoring namespace which can be used to create such alerts. Note: Currently this is not a managed feature of ROSA. Such an implementation may get overwritten if the User Workload Monitoring functionality is toggled off and on using the OpenShift Cluster Manager (OCM).

Extending ROSA STS to include authentication with AWS Services

In this example we will deploy the Amazon Ingress Controller that uses ALBs, and configure it to use STS authentication. Deployment Configure STS Make sure your cluster has the pod identity webhook kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io pod-identity-webhook Download the IAM Policy for the AWS Load Balancer Hooks wget https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json Create AWS Role with inline policy aws iam create-role \ --role-name AWSLoadBalancerController --query Policy.Arn --output text Create AWS Policy and Service Account

Integrating with AWS resources using Pod Identity

Prerequisites ROSA CLI AWS CLI ROSA Cluster with STS

Using the AWS Cloud Watch agent to publish metrics to CloudWatch in ROSA

This document shows how you can use the AWS Cloud Watch agent to scrape Prometheus endpoints and publish metrics to CloudWatch in a Red Hat OpenShift Container Platform (ROSA) cluster. It pulls from The AWS documentation for installing the CloudWatch agent to Kubernetes and collections and publishes metrics for the Kubernetes API Server and provides a simple Dashboard to view the results. Currently the AWS Cloud Watch Agent does not supportexternal link (opens in new tab) pulling all metrics from the Prometheus federated endpoint, but the hope is that when it does we can ship all Cluster and User Workload metrics to CloudWatch.

Creating a ROSA cluster with PrivateLink enabled (custom VPC) and STS

This is a combination of the private-link and sts setup documents to show the full picture Prerequisites AWS CLIexternal link (opens in new tab) Rosa CLIexternal link (opens in new tab) v1.1.7 jqexternal link (opens in new tab) AWS Preparation If this is a brand new AWS account that has never had a AWS Load Balancer installed in it, you should run the following aws iam create-service-linked-role --aws-service-name \ "elasticloadbalancing.

Examples of using a WAF in front of ROSA / OSD on AWS / OCP on AWS

Problem Statement Operator requires WAF (Web Application Firewall) in front of their workloads running on OpenShift (ROSA) Operator does not want WAF running on OpenShift to ensure that OCP resources do not experience Denial of Service through handling the WAF Quick Introduction by Paul Czarkowskiexternal link (opens in new tab) & Ryan Niksch on YouTubeexternal link (opens in new tab) Solutions Cloud Front -> WAF -> CustomDomain -> $APP This is the preferred method and can also work with most third party WAF systems that act as a reverse proxy

Creating a ROSA cluster with PrivateLink enabled

Prerequisites AWS CLIexternal link (opens in new tab) Rosa CLIexternal link (opens in new tab) v1.0.8 jqexternal link (opens in new tab) Create VPC and Subnets The following instructions use the AWS CLI to create the necessary networking to deploy a PrivateLink ROSA cluster into a Single AZ and are intended to be a guide. Ideally you would use an Automation tool like Ansible or Terraform to manage your VPCs.

Federating System and User metrics to S3 in Red Hat OpenShift for AWS

This guide walks through setting up federating Prometheus metrics to S3 storage. ToDo - Add Authorization in front of Thanos APIs Prerequisites A ROSA cluster deployed with STS aws CLI Set up environment Create environment variables export CLUSTER_NAME=my-cluster export S3_BUCKET=my-thanos-bucket export REGION=us-east-2 export NAMESPACE=federated-metrics export SA=aws-prometheus-proxy export SCRATCH_DIR=/tmp/scratch export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///") export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export AWS_PAGER="" rm -rf $SCRATCH_DIR mkdir -p $SCRATCH_DIR Create namespace

Interested in contributing to these docs?

Collaboration drives progress. Help improve our documentation The Red Hat Way.

GitHub Edit this page
Red Hat logo LinkedIn YouTube Facebook Twitter

Products

Tools

Try, buy & sell

Communicate

About Red Hat

We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Subscribe to our newsletter, Red Hat Shares

Sign up now © 2023 Red Hat, Inc.