IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Custom AlertManager in ROSA 4.9.x


Authors: Paul Czarkowski
Last Editor: Dustin Scott
Published Date: 1 September 2022
Modified Date: 25 May 2023


This page is deprecated. In order to get the best experience for custom alerting in ROSA, please upgrade your cluster to to 4.12 and follow the newer documentation.

ROSA 4.9.x introduces a new way to provide custom AlertManager configuration to receive alerts from User Workload Management.

The OpenShift Administrator can use the Prometheus Operator to create a custom AlertManager resource and then use the AlertManagerConfig resource to configure User Workload Monitoring to use the custom AlertManager.

Prerequisites

  • AWS CLI
  • A Red Hat OpenShift for AWS (ROSA) cluster 4.9.0 or higher

Create Environment Variables

  1. Before we get started we need to set some environment variables to be used throughout the guide.

    export PROM_NAMESPACE=custom-alert-manager
    

Install Prometheus Operator

If you prefer you can do this from the Operator Hub in the cluster console itself.

  1. Create a OperatorGroup and Subscription for the Prometheus Operator

    cat << EOF | kubectl apply -f -
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: ${PROM_NAMESPACE}
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: federated-metrics
      namespace: ${PROM_NAMESPACE}
    spec:
      targetNamespaces:
      - ${PROM_NAMESPACE}
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: prometheus
      namespace: ${PROM_NAMESPACE}
    spec:
      channel: beta
      installPlanApproval: Automatic
      name: prometheus
      source: community-operators
      sourceNamespace: openshift-marketplace
    EOF
    

Deploy AlertManager

  1. Create an Alert Manager Configuration file

    This will create a basic AlertManager configuration to send alerts to a slack channel. Configuring slack is outside the scope of this document. Update the variables to suit your slack integration.

    SLACK_API_URL=https://hooks.slack.com/services/XXX/XXX/XXX
    SLACK_CHANNEL='#paultest'
    cat << EOF | kubectl apply -n ${PROM_NAMESPACE} -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: custom-alertmanager
      namespace: ${PROM_NAMESPACE}
    stringData:
      alertmanager.yaml: |
        global:
          slack_api_url: "${SLACK_API_URL}"
        route:
          receiver: slack-notifications
          group_by: [alertname]
        receivers:
        - name: slack-notifications
          slack_configs:
          - channel: ${SLACK_CHANNEL}
            send_resolved: true
    ---
    apiVersion: monitoring.coreos.com/v1
    kind: Alertmanager
    metadata:
      name: custom-alertmanager
      namespace: ${PROM_NAMESPACE}
    spec:
      securityContext: {}
      replicas: 3
      configSecret: custom-alertmanager
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: custom-alertmanager
      namespace: ${PROM_NAMESPACE}
    spec:
      type: ClusterIP
      ports:
      - name: web
        port: 9093
        protocol: TCP
        targetPort: web
      selector:
        alertmanager: custom-alertmanager
    EOF
    

Configure User Workload Monitoring to use the custom AlertManager

  1. Create an AlertManagerConfig for User Workload Monitoring

    Note: This next command assumes the existing config.yaml in the user-workload-monitoring-config config map is empty. You should verify it with kubectl get -n openshift-user-workload-monitoring cm user-workload-monitoring-config -o yaml and simply edit in the differences if its not.

    cat << EOF | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: user-workload-monitoring-config
      namespace: openshift-user-workload-monitoring
    data:
      config.yaml: |
        thanosRuler:
          additionalAlertmanagerConfigs:
          - scheme: http
            pathPrefix: /
            timeout: "30s"
            apiVersion: v1
            staticConfigs: ["custom-alertmanager.$PROM_NAMESPACE.svc.cluster.local:9093"]
    EOF
    

Create an Example Alert

  1. Verify it works by creating a Prometheus Rule that will fire off an alert

    cat << EOF | kubectl apply -n $PROM_NAMESPACE -f -
    apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      name: prometheus-example-rules
      namespace: ${PROM_NAMESPACE}
    spec:
      groups:
      - name: example.rules
        rules:
        - alert: ExampleAlert
          expr: vector(1)
    EOF
    
  2. Forward a port to the alert manager service

     kubectl port-forward -n ${PROM_NAMESPACE} svc/custom-alertmanager 9093:9093
    
  3. Browse to http://localhost:9093/#/alerts to see the alert “ExampleAlert”

    Screenshot of Alert Manager

  4. Check the Alert was sent to Slack

    Screenshot of Alert in Slack