IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Enable the Managed Upgrade Operator in ARO and schedule Upgrades


Authors: Paul Czarkowski, Connor Wooley
Last Editor: Dustin Scott
Published Date: 12 April 2022
Modified Date: 25 May 2023


Prerequisites

  • an Azure Red Hat OpenShift cluster

Get Started

  1. Run this oc command to enable the Managed Upgrade Operator (MUO)

    oc patch cluster.aro.openshift.io cluster --patch \
     '{"spec":{"operatorflags":{"rh.srep.muo.enabled": "true","rh.srep.muo.managed": "true","rh.srep.muo.deploy.pullspec":"arosvc.azurecr.io/managed-upgrade-operator@sha256:f57615aa690580a12c1e5031ad7ea674ce249c3d0f54e6dc4d070e42a9c9a274"}}}' \
     --type=merge
    
  2. Wait a few moments to ensure the Management Upgrade Operator is ready

    oc -n openshift-managed-upgrade-operator \
      get deployment managed-upgrade-operator
    
    NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
    managed-upgrade-operator   1/1     1            1           2m2s
    
  3. Configure the Managed Upgrade Operator

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: managed-upgrade-operator-config
      namespace:  openshift-managed-upgrade-operator
    data:
      config.yaml: |
        configManager:
          source: LOCAL
          localConfigName: managed-upgrade-config
          watchInterval: 1
        maintenance:
          controlPlaneTime: 90
          ignoredAlerts:
            controlPlaneCriticals:
            - ClusterOperatorDown
            - ClusterOperatorDegraded
        upgradeWindow:
          delayTrigger: 30
          timeOut: 120
        nodeDrain:
          timeOut: 45
          expectedNodeDrainTime: 8
        scale:
          timeOut: 30
        healthCheck:
          ignoredCriticals:
          - PrometheusRuleFailures
          - CannotRetrieveUpdates
          - FluentdNodeDown
          ignoredNamespaces:
          - openshift-logging
          - openshift-redhat-marketplace
          - openshift-operators
          - openshift-user-workload-monitoring
          - openshift-pipelines
    EOF
    
  4. Restart the Managed Upgrade Operator

    oc -n openshift-managed-upgrade-operator \
      scale deployment managed-upgrade-operator --replicas=0
    oc -n openshift-managed-upgrade-operator \
      scale deployment managed-upgrade-operator --replicas=1
    
  5. Look for available Upgrades

    If there output is nil there are no available upgrades and you cannot continue.

    oc get clusterversion version -o jsonpath='{.status.availableUpdates}'
    
  6. Schedule an Upgrade

    Set the Channel and Version to the desired values from the above list of available upgrades.

    cat << EOF | oc apply -f -
    apiVersion: upgrade.managed.openshift.io/v1alpha1
    kind: UpgradeConfig
    metadata:
      name: managed-upgrade-config
      namespace: openshift-managed-upgrade-operator
    spec:
      type: "ARO"
      upgradeAt: $(date -u --iso-8601=seconds --date "+5 minutes")
      PDBForceDrainTimeout: 60
      capacityReservation: false
      desired:
        channel: "stable-4.9"
        version: "4.9.27"
    EOF
    
  7. Check the status of the scheduled upgrade

    oc -n openshift-managed-upgrade-operator get \
      upgradeconfigs.upgrade.managed.openshift.io \
      managed-upgrade-config -o jsonpath='{.status}' | jq
    

    The output of this command should show upgrades in progress

    {
    "history": [
      {
        "conditions": [
          {
            "lastProbeTime": "2022-04-12T14:42:02Z",
            "lastTransitionTime": "2022-04-12T14:16:44Z",
            "message": "ControlPlaneUpgraded still in progress",
            "reason": "ControlPlaneUpgraded not done",
            "startTime": "2022-04-12T14:16:44Z",
            "status": "False",
            "type": "ControlPlaneUpgraded"
          },
    
  8. You can verify the upgrade has completed successfully via the following

    oc get clusterversion version
    
    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.9.27    True        False         161m    Cluster version is 4.9.27