Private ARO Cluster with access via JumpHost
A Quickstart guide to deploying a Private Azure Red Hat OpenShift cluster.
Once the cluster is running you will need a way to access the private network that ARO is deployed into.
Author: Paul Czarkowski
Prerequisites
Azure CLI
Obviously you’ll need to have an Azure account to configure the CLI against.
MacOS
See Azure Docs for alternative install options.
Install Azure CLI using homebrew
brew update && brew install azure-cli
Linux
See Azure Docs for alternative install options.
Import the Microsoft Keys
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
Add the Microsoft Yum Repository
cat << EOF | sudo tee /etc/yum.repos.d/azure-cli.repo [azure-cli] name=Azure CLI baseurl=https://packages.microsoft.com/yumrepos/azure-cli enabled=1 gpgcheck=1 gpgkey=https://packages.microsoft.com/keys/microsoft.asc EOF
Install Azure CLI
sudo dnf install -y azure-cli
Prepare Azure Account for Azure OpenShift
Log into the Azure CLI by running the following and then authorizing through your Web Browser
az login
Make sure you have enough Quota (change the location if you’re not using
East US
)az vm list-usage --location "East US" -o table
see Addendum - Adding Quota to ARO account if you have less than
36
Quota left forTotal Regional vCPUs
.Register resource providers
az provider register -n Microsoft.RedHatOpenShift --wait az provider register -n Microsoft.Compute --wait az provider register -n Microsoft.Storage --wait az provider register -n Microsoft.Authorization --wait
Get Red Hat pull secret
Log into cloud.redhat.com
Browse to https://cloud.redhat.com/openshift/install/azure/aro-provisioned
click the Download pull secret button and remember where you saved it, you’ll reference it later.
Deploy Azure OpenShift
Variables and Resource Group
Set some environment variables to use later, and create an Azure Resource Group.
Set the following environment variables
Change the values to suit your environment, but these defaults should work.
AZR_RESOURCE_LOCATION=eastus AZR_RESOURCE_GROUP=openshift-private AZR_CLUSTER=private-cluster AZR_PULL_SECRET=~/Downloads/pull-secret.txt NETWORK_SUBNET=10.0.0.0/20 CONTROL_SUBNET=10.0.0.0/24 MACHINE_SUBNET=10.0.1.0/24 FIREWALL_SUBNET=10.0.2.0/24 JUMPHOST_SUBNET=10.0.3.0/24
Create an Azure resource group
az group create \ --name $AZR_RESOURCE_GROUP \ --location $AZR_RESOURCE_LOCATION
Networking
Create a virtual network with two empty subnets
Create virtual network
az network vnet create \ --address-prefixes $NETWORK_SUBNET \ --name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ --resource-group $AZR_RESOURCE_GROUP
Create control plane subnet
az network vnet subnet create \ --resource-group $AZR_RESOURCE_GROUP \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ --name "$AZR_CLUSTER-aro-control-subnet-$AZR_RESOURCE_LOCATION" \ --address-prefixes $CONTROL_SUBNET \ --service-endpoints Microsoft.ContainerRegistry
Create machine subnet
az network vnet subnet create \ --resource-group $AZR_RESOURCE_GROUP \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ --name "$AZR_CLUSTER-aro-machine-subnet-$AZR_RESOURCE_LOCATION" \ --address-prefixes $MACHINE_SUBNET \ --service-endpoints Microsoft.ContainerRegistry
Disable network policies for Private Link Service on the control plane subnet
This is required for the service to be able to connect to and manage the cluster.
az network vnet subnet update \ --name "$AZR_CLUSTER-aro-control-subnet-$AZR_RESOURCE_LOCATION" \ --resource-group $AZR_RESOURCE_GROUP \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ --disable-private-link-service-network-policies true
Firewall + Internet Egress
This replaces the routes for the cluster to go through the Firewall for egress vs the LoadBalancer which we can later remove. It does come with extra Azure costs of course.
You can skip this step if you don’t need to restrict egress.
Make sure you have the AZ CLI firewall extensions
az extension add -n azure-firewall az extension update -n azure-firewall
Create a firewall network, IP, and firewall
az network vnet subnet create \ -g $AZR_RESOURCE_GROUP \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ -n "AzureFirewallSubnet" \ --address-prefixes $FIREWALL_SUBNET az network public-ip create -g $AZR_RESOURCE_GROUP -n fw-ip \ --sku "Standard" --location $AZR_RESOURCE_LOCATION az network firewall create -g $AZR_RESOURCE_GROUP \ -n aro-private -l $AZR_RESOURCE_LOCATION
Configure the firewall and configure IP Config (this may take 15 minutes)
az network firewall ip-config create -g $AZR_RESOURCE_GROUP \ -f aro-private -n fw-config --public-ip-address fw-ip \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" FWPUBLIC_IP=$(az network public-ip show -g $AZR_RESOURCE_GROUP -n fw-ip --query "ipAddress" -o tsv) FWPRIVATE_IP=$(az network firewall show -g $AZR_RESOURCE_GROUP -n aro-private --query "ipConfigurations[0].privateIpAddress" -o tsv) echo $FWPUBLIC_IP echo $FWPRIVATE_IP
Create and configure a route table
az network route-table create -g $AZR_RESOURCE_GROUP --name aro-udr sleep 10 az network route-table route create -g $AZR_RESOURCE_GROUP --name aro-udr \ --route-table-name aro-udr --address-prefix 0.0.0.0/0 \ --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP az network route-table route create -g $AZR_RESOURCE_GROUP --name aro-vnet \ --route-table-name aro-udr --address-prefix 10.0.0.0/16 --name local-route \ --next-hop-type VirtualNetworkGateway
Create firewall rules for ARO resources
Note: ARO clusters do not need access to the internet, however your own workloads running on them may. You can skip this step if you don’t need any egress at all.
Create a Network Rule to allow all http/https egress traffic (not recommended)
az network firewall network-rule create -g $AZR_RESOURCE_GROUP -f aro-private \ --collection-name 'allow-https' --name allow-all \ --action allow --priority 100 \ --source-addresses '*' --dest-addr '*' \ --protocols 'Any' --destination-ports 1-65535
Create Application Rules to allow to a restricted set of destinations
replace the target-fqdns with your desired destinations
az network firewall application-rule create -g $AZR_RESOURCE_GROUP -f aro-private \ --collection-name 'Allow_Egress' \ --action allow \ --priority 100 \ -n 'required' \ --source-addresses '*' \ --protocols 'http=80' 'https=443' \ --target-fqdns '*.google.com' '*.bing.com' az network firewall application-rule create -g $AZR_RESOURCE_GROUP -f aro-private \ --collection-name 'Docker' \ --action allow \ --priority 200 \ -n 'docker' \ --source-addresses '*' \ --protocols 'http=80' 'https=443' \ --target-fqdns '*cloudflare.docker.com' '*registry-1.docker.io' 'apt.dockerproject.org' 'auth.docker.io'
Update the subnets to use the Firewall
Once the cluster is deployed successfully you can update the subnets to use the firewall instead of the default outbound loadbalancer rule.
az network vnet subnet update -g $AZR_RESOURCE_GROUP \ --vnet-name $AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION \ --name "$AZR_CLUSTER-aro-control-subnet-$AZR_RESOURCE_LOCATION" \ --route-table aro-udr az network vnet subnet update -g $AZR_RESOURCE_GROUP \ --vnet-name $AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION \ --name "$AZR_CLUSTER-aro-machine-subnet-$AZR_RESOURCE_LOCATION" \ --route-table aro-udr
Create the cluster
This will take between 30 and 45 minutes.
az aro create \ --resource-group $AZR_RESOURCE_GROUP \ --name $AZR_CLUSTER \ --vnet "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ --master-subnet "$AZR_CLUSTER-aro-control-subnet-$AZR_RESOURCE_LOCATION" \ --worker-subnet "$AZR_CLUSTER-aro-machine-subnet-$AZR_RESOURCE_LOCATION" \ --apiserver-visibility Private \ --ingress-visibility Private \ --pull-secret @$AZR_PULL_SECRET
Jump Host
With the cluster in a private network, we can create a Jump host in order to connect to it. You can do this while the cluster is being created.
Create jump subnet
az network vnet subnet create \ --resource-group $AZR_RESOURCE_GROUP \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION" \ --name JumpSubnet \ --address-prefixes $JUMPHOST_SUBNET \ --service-endpoints Microsoft.ContainerRegistry
Create a jump host
az vm create --name jumphost \ --resource-group $AZR_RESOURCE_GROUP \ --ssh-key-values $HOME/.ssh/id_rsa.pub \ --admin-username aro \ --image "RedHat:RHEL:8.2:8.2.2021040911" \ --subnet JumpSubnet \ --public-ip-address jumphost-ip \ --vnet-name "$AZR_CLUSTER-aro-vnet-$AZR_RESOURCE_LOCATION"
Save the jump host public IP address
JUMP_IP=$(az vm list-ip-addresses -g $AZR_RESOURCE_GROUP -n jumphost -o tsv \ --query '[].virtualMachine.network.publicIpAddresses[0].ipAddress') echo $JUMP_IP
ssh to jump host forwarding port 1337 as a socks proxy.
replace the IP with the IP of the jump box from the previous step.
ssh -D 1337 -C -i $HOME/.ssh/id_rsa aro@$JUMP_IP
test the socks proxy
curl --socks5-hostname localhost:1337 http://www.google.com/
Install tools
sudo yum install -y gcc libffi-devel python3-devel openssl-devel jq sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc cat << EOF | sudo tee /etc/yum.repos.d/azure-cli.repo [azure-cli] name=Azure CLI baseurl=https://packages.microsoft.com/yumrepos/azure-cli enabled=1 gpgcheck=1 gpgkey=https://packages.microsoft.com/keys/microsoft.asc EOF sudo yum install -y azure-cli wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz mkdir openshift tar -zxvf openshift-client-linux.tar.gz -C openshift sudo install openshift/oc /usr/local/bin/oc sudo install openshift/kubectl /usr/local/bin/kubectl
Wait until the ARO cluster is fully provisioned.
Login to Azure
az login
Get OpenShift console URL
set these variables to match the ones you set at the start.
AZR_RESOURCE_GROUP=openshift-private AZR_CLUSTER=private-cluster APISERVER=$(az aro show \ --name $AZR_CLUSTER \ --resource-group $AZR_RESOURCE_GROUP \ -o tsv --query apiserverProfile.url) echo $APISERVER
Get OpenShift credentials
ADMINPW=$(az aro list-credentials \ --name $AZR_CLUSTER \ --resource-group $AZR_RESOURCE_GROUP \ --query kubeadminPassword \ -o tsv)
Test Access
Test Access to the cluster via the socks proxy
CONSOLE=$(az aro show \ --name $AZR_CLUSTER \ --resource-group $AZR_RESOURCE_GROUP \ -o tsv --query consoleProfile) echo $CONSOLE curl --socks5-hostname localhost:1337 $CONSOLE
Unfortunately you can’t [easily] use the socks proxy with the
oc
command, but at least you can access the console via the socks proxy.
- Set localhost:1337 as a socks proxy in your browser and verify you can access the cluster by browsing to the
$CONSOLE
url.
Delete Cluster
Once you’re done its a good idea to delete the cluster to ensure that you don’t get a surprise bill.
Delete the cluster
az aro delete -y \ --resource-group $AZR_RESOURCE_GROUP \ --name $AZR_CLUSTER
Delete the Azure resource group
Only do this if there’s nothing else in the resource group.
az group delete -y \ --name $AZR_RESOURCE_GROUP
Addendum
Adding Quota to ARO account
Set Issue Type to “Service and subscription limits (quotas)”
Set Quota Type to “Compute-VM (cores-vCPUs) subscription limit increases”
Click Next Solutions »
Click Enter details
Set Deployment Model to “Resource Manager
Set Locations to “(US) East US”
Set Types to “Standard”
Under Standard check “DSv3” and “DSv4”
Set New vCPU Limit for each (example “60”)
Click Save and continue
Click Review + create »
Wait until quota is increased.