Custom Alerts in ROSA 4.11.x
Starting with OpenShift 4.11 it is possible to manage alerting rules for user-defined projects. Similarly, in ROSA clusters the OpenShift Administrator can enable a second AlertManager instance in the user workload monitoring namespace which can be used to create such alerts.
Note: Currently this is not a managed feature of ROSA. Such an implementation may get overwritten if the User Workload Monitoring functionality is toggled using the OpenShift Cluster Manager (OCM).
Prerequisites
- AWS CLI
- A Red Hat OpenShift for AWS (ROSA) cluster 4.11.0 or higher
Create Environment Variables
Configure User Workload Monitoring to include AlertManager
- Edit the user workload config to include AlertManager
Note: If you have other modifications to this config, you will need to hand edit the resource rather than brute forcing it like below.
cat << EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: user-workload-monitoring-config
namespace: openshift-user-workload-monitoring
data:
config.yaml: |
alertmanager:
enabled: true
enableAlertmanagerConfig: true
EOF
Verify that a new Alert Manager instance is defined
oc -n openshift-user-workload-monitoring get alertmanager
NAME VERSION REPLICAS AGE user-workload 0.24.0 2 2m
If you want non-admin users to be able to define alerts in their own namespaces you can run the following.
oc -n <namespace> adm policy add-role-to-user alert-routing-edit <user>
Update the Alert Manager Configuration file
This will create a basic AlertManager configuration to send alerts to a slack channel. Configuring slack is outside the scope of this document. Update the variables to suit your slack integration.
SLACK_API_URL=https://hooks.slack.com/services/XXX/XXX/XXX SLACK_CHANNEL='#paultest' cat << EOF | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: alertmanager-user-workload namespace: openshift-user-workload-monitoring stringData: alertmanager.yaml: | global: slack_api_url: "${SLACK_API_URL}" route: receiver: Default group_by: [alertname] receivers: - name: Default slack_configs: - channel: ${SLACK_CHANNEL} send_resolved: true EOF
Create an Example Alert
Create a Namespace for your custom alert
oc create namespace custom-alert
Verify it works by creating a Prometheus Rule that will fire off an alert
cat << EOF | kubectl apply -n custom-alert -f - apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: prometheus-example-rules spec: groups: - name: example.rules rules: - alert: ExampleAlert expr: vector(1) EOF
Forward a port to the alert manager service
kubectl port-forward -n openshift-user-workload-monitoring \ svc/alertmanager-operated 9093:9093
Browse to http://localhost:9093/#/alerts to see the alert “ExampleAlert”
Check the Alert was sent to Slack