IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Adding an additional ingress controller to an ARO cluster

Paul Czarkowski, Stuart Kirk, Anton Nesterov

03/30/2022 (updated on 12/09/2022)

Prerequisites

  • an Azure Red Hat OpenShift cluster
  • a DNS zone that you can easily modify

Get Started

  1. Create some environment variables

    DOMAIN=custom.azure.mobb.ninja
EMAIL=example@email.com
SCRATCH_DIR=/tmp/aro

  2. Create a certificate for the ingress controller

    certbot certonly --manual \
  --preferred-challenges=dns \
  --email $EMAIL \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --agree-tos \
  --manual-public-ip-logging-ok \
  -d "*.$DOMAIN" \
  --config-dir "$SCRATCH_DIR/config" \
  --work-dir "$SCRATCH_DIR/work" \
  --logs-dir "$SCRATCH_DIR/logs"

  3. Create a secret for the certificate

    oc create secret tls custom-tls \
  -n openshift-ingress \
  --cert=$SCRATCH_DIR/config/live/$DOMAIN/fullchain.pem \
  --key=$SCRATCH_DIR/config/live/$DOMAIN/privkey.pem

  4. Create an ingress controller

    cat <<EOF | oc apply -f -
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: custom
  namespace: openshift-ingress-operator
spec:
  domain: $DOMAIN
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/worker: ""
  routeSelector:
    matchLabels:
      type: custom
  defaultCertificate:
    name: custom-tls
  httpEmptyRequestsPolicy: Respond
  httpErrorCodePages:
    name: ""
  replicas: 3
EOF

    NOTE: By default the ingress controller is created with external scope. This means that the corresponding Azure Load Balancer will have a public frontend IP. If you wish to deploy a privately visible ingress controller add the following lines to the spec:

    spec:
  ...
  endpointPublishingStrategy:
    loadBalancer:
      scope: Internal
    type: LoadBalancerService
  ...

  5. Wait a few moments then get the EXTERNAL-IP of the new ingress controller

    oc get -n openshift-ingress svc router-custom

    In case of an Externally (publicly) scoped ingress controller the output should look like:

     NAME            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
 router-custom   LoadBalancer   172.30.90.84   20.120.48.78   80:32160/TCP,443:32511/TCP   49s

    In case of an Internal (private) one:

    NAME            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
router-custom   LoadBalancer   172.30.55.36     10.0.2.4     80:30475/TCP,443:30249/TCP   10s

  6. Optionally verify in the Azure portal or using CLI that the Load Balancer Service has gotten the new Frontend IP and two Load Balancing Rules - one for port 80 and another one for port 443. In case of an Internally scoped Ingress Controller the changes are to be observed within the Load Balancer that has the -internal suffix.

  7. Create a wildcard DNS record pointing at the EXTERNAL-IP

  8. Test that the Ingress is working

    NOTE: For the Internal ingress controller, make sure that the test host has the necessary reachability to the VPC/subnet as well as the DNS resolver.

    curl -s https://test.$DOMAIN | head

     <html>
   <head>
     <meta name="viewport" content="width=device-width, initial-scale=1">

  9. Create a new project to deploy an application to

    oc new-project demo

  10. Create a new application

    oc new-app --docker-image=docker.io/openshift/hello-openshift

  11. Expose

    cat << EOF | oc apply -f -
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: hello-openshift
    app.kubernetes.io/component: hello-openshift
    app.kubernetes.io/instance: hello-openshift
    type: custom
  name: hello-openshift-tls
spec:
  host: hello.$DOMAIN
  port:
    targetPort: 8080-tcp
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  to:
    kind: Service
    name: hello-openshift
EOF

  12. Verify it works

    curl https://hello.custom.azure.mobb.ninja

     Hello OpenShift!