Home GitHub

IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Using the Egressip Ipam Operator with a Private ARO Cluster

Prerequisites

Deploy the Egressip Ipam Operator

Via GUI

  1. Log into the ARO cluster’s Console

  2. Switch to the Administrator view

  3. Click on Operators -> Operator Hub

  4. Search for “Egressip Ipam Operator”

  5. Install it with the default settings

or

Via CLI

  1. Deploy the egress-ipam-operator

    cat << EOF | kubectl apply -f -
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: egressip-ipam-operator
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: egressip-ipam-operator
      namespace: openshift-operators
      labels:
        operators.coreos.com/egressip-ipam-operator.egressip-ipam-operator: ''
    spec:
      channel: alpha
      installPlanApproval: Automatic
      name: egressip-ipam-operator
      source: community-operators
      sourceNamespace: openshift-marketplace
      startingCSV: egressip-ipam-operator.v1.2.2
    EOF
    

Configure EgressIP

  1. Create an EgressIPAM resource for your cluster. Update the CIDR to reflect the worker node subnet.

    cat << EOF | kubectl apply -f -
    apiVersion: redhatcop.redhat.io/v1alpha1
    kind: EgressIPAM
    metadata:
      name: egressipam-azure
      annotations:
        egressip-ipam-operator.redhat-cop.io/azure-egress-load-balancer: none
    spec:
      cidrAssignments:
        - labelValue: ""
          CIDR: 10.0.1.0/24
          reservedIPs: []
      topologyLabel: "node-role.kubernetes.io/worker"
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/worker: ""
    EOF
    
  2. Create test namespaces

    cat << EOF | kubectl apply -f -
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: egressipam-azure-test
      annotations:
        egressip-ipam-operator.redhat-cop.io/egressipam: egressipam-azure
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: egressipam-azure-test-1
      annotations:
        egressip-ipam-operator.redhat-cop.io/egressipam: egressipam-azure
    EOF
    
  3. Check the namespaces have IPs assigned

    kubectl get namespace egressipam-azure-test \
      egressipam-azure-test-1 -o yaml | grep egressips
    

    The output should look like:

    egressip-ipam-operator.redhat-cop.io/egressips: 10.0.1.8
    egressip-ipam-operator.redhat-cop.io/egressips: 10.0.1.7
    
  4. Check they’re actually set as Egress IPs

    oc get netnamespaces | egrep 'NAME|egress'
    

    The output should look like:

    NAME                                               NETID      EGRESS IPS
    egressip-ipam-operator                             6374875
    egressipam-azure-test                              6917470    ["10.0.1.8"]
    egressipam-azure-test-1                            16320378   ["10.0.1.7"]
    
  5. Finally check the Host Subnets for Egress IPS

    oc get hostsubnets
    

    The output should look like:

    NAME                                         HOST                                         HOST IP    SUBNET          EGRESS CIDRS   EGRESS IPS
    private-cluster-bj275-master-0               private-cluster-bj275-master-0               10.0.0.8   10.129.0.0/23
    private-cluster-bj275-master-1               private-cluster-bj275-master-1               10.0.0.7   10.128.0.0/23
    private-cluster-bj275-master-2               private-cluster-bj275-master-2               10.0.0.9   10.130.0.0/23
    private-cluster-bj275-worker-eastus1-zt59t   private-cluster-bj275-worker-eastus1-zt59t   10.0.1.4   10.128.2.0/23                  ["10.0.1.8"]
    private-cluster-bj275-worker-eastus2-bfrwt   private-cluster-bj275-worker-eastus2-bfrwt   10.0.1.5   10.129.2.0/23                  ["10.0.1.7"]
    private-cluster-bj275-worker-eastus3-fgjzk   private-cluster-bj275-worker-eastus3-fgjzk   10.0.1.6   10.131.0.0/23
    

Test Egress

  1. Log into your jumpbox and allow http into firewall

    sudo firewall-cmd --zone=public --add-service=http
    
  2. Install and start apache httpd

    sudo yum -y install httpd
    sudo systemctl start httpd
    
  3. Create a index.html

    echo HELLO | sudo tee /var/www/html/index.html
    
  4. tail apache logs

    sudo tail -f /var/log/httpd/access_log
    
  5. Start an interactive pod in one of your new namespaces

    kubectl run -n egressipam-azure-test -i \
    --tty --rm debug --image=alpine \
    --restart=Never -- wget -O - 10.0.3.4
    

    The output should look the following (the IP should match the egress IP of your namespace):

    10.0.1.7 - - [03/Feb/2022:19:33:54 +0000] "GET / HTTP/1.1" 200 6 "-" "Wget"