IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at and

Custom Alerts in ROSA 4.11.x

Authors: Paul Czarkowski
Last Editor: Dustin Scott
Published Date: 20 October 2021
Modified Date: 25 May 2023

Starting with OpenShift 4.11 it is possible to manage alerting rules for user-defined projects . Similarly, in ROSA clusters the OpenShift Administrator can enable a second AlertManager instance in the user workload monitoring namespace which can be used to create such alerts.

Note: Currently this is not a managed feature of ROSA. Such an implementation may get overwritten if the User Workload Monitoring functionality is toggled off and on using the OpenShift Cluster Manager (OCM). We


  • A Red Hat OpenShift for AWS (ROSA) cluster 4.11.0 or higher

Create Environment Variables

Configure User Workload Monitoring to include AlertManager

  1. Edit the user workload config to include AlertManager

    Note: If you have other modifications to this config, you will need to hand edit the resource rather than brute forcing it like below.

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: ConfigMap
      name: user-workload-monitoring-config
      namespace: openshift-user-workload-monitoring
      config.yaml: |
          enabled: true
          enableAlertmanagerConfig: true
  2. Verify that a new Alert Manager instance is defined

    oc -n openshift-user-workload-monitoring get alertmanager
    NAME            VERSION   REPLICAS   AGE
    user-workload   0.24.0    2          2m
  3. If you want non-admin users to be able to define alerts in their own namespaces you can run the following.

    oc -n <namespace> adm policy add-role-to-user alert-routing-edit <user>
  4. Create a Slack webhook integration in your Slack workspace. Create environment variables for it.

  5. Update the Alert Manager Configuration file

    This will create a basic AlertManager configuration to send alerts to your slack channel. If you have an existing configuration you’ll need to edit it, not brute force paste like shown.

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: Secret
      name: alertmanager-user-workload
      namespace: openshift-user-workload-monitoring
      alertmanager.yaml: |
          slack_api_url: "${SLACK_API_URL}"
          receiver: Default
          group_by: [alertname]
          - name: Default
              - channel: ${SLACK_CHANNEL}
                send_resolved: true

Create an Example Alert

  1. Create a Namespace for your custom alert

    oc create namespace custom-alert
  2. Verify it works by creating a Prometheus Rule that will fire off an alert

    cat << EOF | oc apply -f -
    kind: PrometheusRule
      name: prometheus-example-rules
      namespace: custom-alert
      - name: example.rules
        - alert: ExampleAlert
          expr: vector(1)
  3. Log into your OpenShift Console and switch to the Developer view. Select the Observe item from the menu on the left and then click Alerts. Make your your Project is set to custom-alert

    Screenshot of Alert in OCP Console

  4. Check the Alert was sent to Slack

    Screenshot of Alert in Slack

What about Cluster Alerts ?

By default, cluster alerts are only sent to Red Hat SRE and you cannot modify the cluster alert receivers. However you can make a copy of any of the alerts that you wish to see in your own alerting namespace which will then get picked up by the user workload alert-manager.

  1. Duplicate a subset of the Cluster Monitoring Prometheus Rules

    oc -n openshift-monitoring get prometheusrule \
      cluster-monitoring-operator-prometheus-rules -o json | \
      jq '.metadata.namespace = "custom-alert"' | \
      kubectl create -f -
  2. Check the Alerts in the custom-alert Project in the OpenShift Console, and you’ll now see the Watchdog and other cluster alerts.

    Screenshot of cluster alerts in custom project

  3. You should also have received an alert for the Watchdog alert in your Slack channel.

    Screenshot of watchdog alert in slack