Cloud Experts Documentation

Federating System and User metrics to Azure Blob storage in Azure Red Hat OpenShift

This content is authored by Red Hat experts, but has not yet been tested on every supported configuration.

By default Azure Red Hat OpenShift (ARO) stores metrics in Ephemeral volumes, and its advised that users do not change this setting. However its not unreasonable to expect that metrics should be persisted for a set amount of time.

This guide shows how to set up Thanos to federate both System and User Workload Metrics to a Thanos gateway that stores the metrics in Azure Blob Container and makes them available via a Grafana instance (managed by the Grafana Operator).

ToDo - Add Authorization in front of Thanos APIs

Pre-Prequsites

  1. An ARO cluster

  2. Set some environment variables to use throughout to suit your environment

    Note: AZR_STORAGE_ACCOUNT_NAME must be unique

    export AZR_RESOURCE_LOCATION="eastus"
    export AZR_RESOURCE_GROUP="openshift"
    export CLUSTER_NAME="openshift"
    export UNIQUE="$(cat /dev/urandom | LC_ALL=C tr -dc 'a-z0-9' | fold -w 5 | head -n 1)"
    export AZR_STORAGE_ACCOUNT_NAME="arometrics${UNIQUE}"
    export NAMESPACE="aro-thanos-af"
    
    Copy

Azure Preperation

  1. Create an Azure storage account

    modify the arguments to suit your environment

    az storage account create \
      --name $AZR_STORAGE_ACCOUNT_NAME \
      --resource-group $AZR_RESOURCE_GROUP \
      --location $AZR_RESOURCE_LOCATION \
      --sku Standard_RAGRS \
      --kind StorageV2
    
    Copy
  2. Get the account key and update the secret in thanos-store-credentials.yaml

    AZR_STORAGE_KEY=$(az storage account keys list -g $AZR_RESOURCE_GROUP \
      -n $AZR_STORAGE_ACCOUNT_NAME --query "[0].value" -o tsv)
    
    Copy
  3. Create a namespace to use

    oc new-project $NAMESPACE
    
    Copy
  4. Add the MOBB chart repository to your Helm

    helm repo add mobb https://rh-mobb.github.io/helm-charts/
    
    Copy
  5. Update your repositories

    helm repo update
    
    Copy
  6. Use the mobb/operatorhub chart to deploy the grafana operator

    helm upgrade -n $NAMESPACE grafana-operator \
      mobb/operatorhub --install \
      --values https://raw.githubusercontent.com/rh-mobb/helm-charts/main/charts/aro-thanos-af/files/grafana-operator.yaml
    
    Copy
  7. Wait for the Operator to be ready

    oc rollout status -n $NAMESPACE \
      deployment/grafana-operator-controller-manager
    
    Copy
  8. Use Helm deploy the OpenShift Patch Operator

    helm upgrade -n patch-operator patch-operator --create-namespace \
      mobb/operatorhub --install \
      --values https://raw.githubusercontent.com/rh-mobb/helm-charts/main/charts/aro-thanos-af/files/patch-operator.yaml
    
    Copy
  9. Wait for the Operator to be ready

    oc rollout status -n patch-operator \
      deployment/patch-operator-controller-manager
    
    Copy
  10. Deploy ARO Thanos Azure Blob container Helm Chart (mobb/aro-thanos-af)

    **> Note: enableUserWorkloadMetrics=true will overwrite configs for cluster and userworkload metrics. If you have customized them already, you may need to modify patch-monitoring-configs.yaml in the Helm chart to include your changes.

    helm upgrade -n $NAMESPACE aro-thanos-af \
      --install mobb/aro-thanos-af \
      --set "aro.storageAccount=$AZR_STORAGE_ACCOUNT_NAME" \
      --set "aro.storageAccountKey=$AZR_STORAGE_KEY" \
      --set "aro.storageContainer=$CLUSTER_NAME" \
      --set "aro.clusterName=$CLUSTER_NAME" \
      --set "enableUserWorkloadMetrics=true"
    
    Copy

Validate Grafana is installed and seeing metrics from Azure Blob storage

  1. get the Route URL for Grafana (remember its https) and login using username admin and the password password.

    oc -n $NAMESPACE get route grafana-route
    
    Copy
  2. Once logged in go to Dashboards->Manage and expand the aro-thanos-af group and you should see the cluster metrics dashboards. Click on the Use Method / Cluster Dashboard and you should see metrics. \o/.

    Note: If it complains about a missing datasource run the following: oc annotate -n $NAMESPACE grafanadatasource aro-thanos-af-prometheus "retry=1"

    screenshot of grafana with federated cluster metrics

Cleanup

  1. Uninstall the aro-thanos-af chart

    helm delete -n $NAMESPACE aro-thanos-af
    
    Copy
  2. Uninstall the federated-metrics-operators chart

    helm delete -n $NAMESPACE federated-metrics-operators
    
    Copy
  3. Delete the aro-thanos-af namespace

    oc delete namespace $NAMESPACE
    
    Copy
  4. Delete the storage account

    az storage account delete \
      --name $AZR_STORAGE_ACCOUNT_NAME \
      --resource-group $AZR_RESOURCE_GROUP
    
    Copy

Interested in contributing to these docs?

Collaboration drives progress. Help improve our documentation The Red Hat Way.

Red Hat logo LinkedIn YouTube Facebook Twitter

Products

Tools

Try, buy & sell

Communicate

About Red Hat

We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Subscribe to our newsletter, Red Hat Shares

Sign up now
© 2023 Red Hat, Inc.