Home GitHub

Disclaimer: Mobb.ninja is not official Red Hat documentation - These guides may be experimental, proof of concept or early adoption. Officially supported documentation is available at https://docs.openshift.com.

Advanced Cluster Management Observability on ROSA

This document will take you through deploying ACM Observability on an STS enabled ROSA cluster. see here for the original documentation.

Note: This guide uses an unsupported (by ACM) method to utilize ROSA’s STS authentication back into AWS S3 and is not advised for production use at this time.


Set up environment

  1. Set environment variables
    export CLUSTER_NAME=my-cluster
    export S3_BUCKET=$CLUSTER_NAME-acm-observability
    export REGION=us-east-2
    export NAMESPACE=open-cluster-management-observability
    export SA=tbd
    export SCRATCH_DIR=/tmp/scratch
    export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
    export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    export AWS_PAGER=""
    rm -rf $SCRATCH_DIR
    mkdir -p $SCRATCH_DIR

Prepare AWS Account

  1. Create an S3 bucket

     aws s3 mb s3://$S3_BUCKET
  2. Create a Policy for access to S3

    cat <<EOF > $SCRATCH_DIR/s3-policy.json
     "Version": "2012-10-17",
     "Statement": [
             "Sid": "Statement",
             "Effect": "Allow",
             "Action": [
             "Resource": [
  3. Apply the Policy

     S3_POLICY=$(aws iam create-policy --policy-name $CLUSTER_NAME-acm-obs \
       --policy-document file://$SCRATCH_DIR/s3-policy.json \
       --query 'Policy.Arn' --output text)
     echo $S3_POLICY
  4. Create a Trust Policy

    cat <<EOF > $SCRATCH_DIR/TrustPolicy.json
      "Version": "2012-10-17",
      "Statement": [
       "Effect": "Allow",
       "Principal": {
         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
       "Action": "sts:AssumeRoleWithWebIdentity",
       "Condition": {
         "StringEquals": {
           "${OIDC_PROVIDER}:sub": [
  5. Create Role for AWS Prometheus and CloudWatch

     S3_ROLE=$(aws iam create-role \
       --role-name "$CLUSTER_NAME-acm-obs-s3" \
       --assume-role-policy-document file://$SCRATCH_DIR/TrustPolicy.json \
       --query "Role.Arn" --output text)
     echo $S3_ROLE
  6. Attach the Policies to the Role

     aws iam attach-role-policy \
       --role-name "$CLUSTER_NAME-acm-obs-s3" \
       --policy-arn $S3_POLICY


Log into the OpenShift cluster that is running your ACM Hub. We’ll set up Observability here

  1. Create a namespace for the observability

     oc new-project $NAMESPACE
  2. Generate a pull secret (this will check if the pull secret exists, if not, it will create it)

    DOCKER_CONFIG_JSON=`oc extract secret/multiclusterhub-operator-pull-secret -n open-cluster-management --to=-` || \
     DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-` && \
     oc create secret generic multiclusterhub-operator-pull-secret \
     -n open-cluster-management-observability \
     --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
  3. Create a Secret containing your S3 details

    cat << EOF | kubectl apply -f -
    apiVersion: v1
    kind: Secret
      name: thanos-object-storage
      namespace: open-cluster-management-observability
    type: Opaque
      thanos.yaml: |
     type: s3
       bucket: $S3_BUCKET
       endpoint: s3.$REGION.amazonaws.com
       signature_version2: false
  4. Create a CR for MulticlusterHub

    cat << EOF | kubectl apply -f -
    apiVersion: observability.open-cluster-management.io/v1beta2
    kind: MultiClusterObservability
      name: observability
      observabilityAddonSpec: {}
       name: thanos-object-storage
       key: thanos.yaml
  5. Annotate the service accounts to use STS

    oc annotate -n open-cluster-management-observability serviceaccount observability-thanos-query \
    oc annotate -n open-cluster-management-observability serviceaccount observability-thanos-store-shard \
    oc annotate -n open-cluster-management-observability serviceaccount observability-thanos-compact \
  6. Restart thanos pods to use the new credentials

    kubectl rollout restart deployment/observability-thanos-query
    kubectl rollout restart statefulset/observability-thanos-compact
    kubectl rollout restart statefulset/observability-thanos-store-shard-0
    kubectl rollout restart statefulset/observability-thanos-store-shard-1
    kubectl rollout restart statefulset/observability-thanos-store-shard-2
  7. Check for any pods that aren’t running

     kubectl get pods --field-selector status.phase!=Running

    Delete any pods that are not running in order to have them recreated with the correct credentials

     kubectl delete pods $(kubectl get pods --field-selector status.phase!=Running -o name | xargs)
  8. Log into Advanced Cluster management and access the new Grafana dashboard

ACM Grafana Dashboard