Using the Egressip Ipam Operator with a Private ARO Cluster

Last Editor: Dustin Scott
Published Date: 29 June 2021
Modified Date: 25 May 2023

Prerequisites

• A private ARO cluster

Deploy the Egressip Ipam Operator

Via GUI

1. Log into the ARO cluster’s Console

2. Switch to the Administrator view

3. Click on Operators -> Operator Hub

4. Search for “Egressip Ipam Operator”

5. Install it with the default settings

or

Via CLI

1. Deploy the egress-ipam-operator

   cat << EOF | kubectl apply -f -
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: egressip-ipam-operator
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: egressip-ipam-operator
     namespace: openshift-operators
     labels:
       operators.coreos.com/egressip-ipam-operator.egressip-ipam-operator: ''
   spec:
     channel: alpha
     installPlanApproval: Automatic
     name: egressip-ipam-operator
     source: community-operators
     sourceNamespace: openshift-marketplace
     startingCSV: egressip-ipam-operator.v1.2.2
   EOF
   

Configure EgressIP

1. Create an EgressIPAM resource for your cluster. Update the CIDR to reflect the worker node subnet.

   cat << EOF | kubectl apply -f -
   apiVersion: redhatcop.redhat.io/v1alpha1
   kind: EgressIPAM
   metadata:
     name: egressipam-azure
     annotations:
       egressip-ipam-operator.redhat-cop.io/azure-egress-load-balancer: none
   spec:
     cidrAssignments:
       - labelValue: ""
         CIDR: 10.0.1.0/24
         reservedIPs: []
     topologyLabel: "node-role.kubernetes.io/worker"
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker: ""
   EOF
   

2. Create test namespaces

   cat << EOF | kubectl apply -f -
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: egressipam-azure-test
     annotations:
       egressip-ipam-operator.redhat-cop.io/egressipam: egressipam-azure
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: egressipam-azure-test-1
     annotations:
       egressip-ipam-operator.redhat-cop.io/egressipam: egressipam-azure
   EOF
   

3. Check the namespaces have IPs assigned

   kubectl get namespace egressipam-azure-test \
     egressipam-azure-test-1 -o yaml | grep egressips
   

   The output should look like:

   egressip-ipam-operator.redhat-cop.io/egressips: 10.0.1.8
   egressip-ipam-operator.redhat-cop.io/egressips: 10.0.1.7
   

4. Check they’re actually set as Egress IPs

   oc get netnamespaces | egrep 'NAME|egress'
   

   The output should look like:

   NAME                                               NETID      EGRESS IPS
   egressip-ipam-operator                             6374875
   egressipam-azure-test                              6917470    ["10.0.1.8"]
   egressipam-azure-test-1                            16320378   ["10.0.1.7"]
   

5. Finally check the Host Subnets for Egress IPS

   oc get hostsubnets
   

   The output should look like:

   NAME                                         HOST                                         HOST IP    SUBNET          EGRESS CIDRS   EGRESS IPS
   private-cluster-bj275-master-0               private-cluster-bj275-master-0               10.0.0.8   10.129.0.0/23
   private-cluster-bj275-master-1               private-cluster-bj275-master-1               10.0.0.7   10.128.0.0/23
   private-cluster-bj275-master-2               private-cluster-bj275-master-2               10.0.0.9   10.130.0.0/23
   private-cluster-bj275-worker-eastus1-zt59t   private-cluster-bj275-worker-eastus1-zt59t   10.0.1.4   10.128.2.0/23                  ["10.0.1.8"]
   private-cluster-bj275-worker-eastus2-bfrwt   private-cluster-bj275-worker-eastus2-bfrwt   10.0.1.5   10.129.2.0/23                  ["10.0.1.7"]
   private-cluster-bj275-worker-eastus3-fgjzk   private-cluster-bj275-worker-eastus3-fgjzk   10.0.1.6   10.131.0.0/23
   

Test Egress

1. Log into your jumpbox and allow http into firewall

   sudo firewall-cmd --zone=public --add-service=http
   

2. Install and start apache httpd

   sudo yum -y install httpd
   sudo systemctl start httpd
   

3. Create a index.html

   echo HELLO | sudo tee /var/www/html/index.html
   

4. tail apache logs

   sudo tail -f /var/log/httpd/access_log
   

5. Start an interactive pod in one of your new namespaces

   kubectl run -n egressipam-azure-test -i \
   --tty --rm debug --image=alpine \
   --restart=Never -- wget -O - 10.0.3.4
   

   The output should look the following (the IP should match the egress IP of your namespace):

   10.0.1.7 - - [03/Feb/2022:19:33:54 +0000] "GET / HTTP/1.1" 200 6 "-" "Wget"