Advanced Cluster Management Observability on ROSA

Authors: Connor Wooley
Last Editor: Dustin Scott
Published Date: 11 January 2022
Modified Date: 25 May 2023

This document will take you through deploying ACM Observability on a ROSA cluster. see here for the original documentation.

Prerequisites

• An existing ROSA cluster
• An Advanced Cluster Management (ACM) deployment

Set up environment

1. Set environment variables

   export CLUSTER_NAME=my-cluster
   export S3_BUCKET=$CLUSTER_NAME-acm-observability
   export REGION=us-east-2
   export NAMESPACE=open-cluster-management-observability
   export SA=tbd
   export SCRATCH_DIR=/tmp/scratch
   export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   export AWS_PAGER=""
   rm -rf $SCRATCH_DIR
   mkdir -p $SCRATCH_DIR
   

Prepare AWS Account

1. Create an S3 bucket

   aws s3 mb s3://$S3_BUCKET
   

2. Create a Policy for access to S3

   cat <<EOF > $SCRATCH_DIR/s3-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "Statement",
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:PutObjectAcl",
                   "s3:CreateBucket",
                   "s3:DeleteBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::$S3_BUCKET/*",
                   "arn:aws:s3:::$S3_BUCKET"
               ]
           }
       ]
   }
   EOF
   

3. Apply the Policy

   S3_POLICY=$(aws iam create-policy --policy-name $CLUSTER_NAME-acm-obs \
     --policy-document file://$SCRATCH_DIR/s3-policy.json \
     --query 'Policy.Arn' --output text)
   echo $S3_POLICY
   

4. Create service account

   aws iam create-user --user-name $CLUSTER_NAME-acm-obs  \
     --query User.Arn --output text
   

5. Attach policy to user

   aws iam attach-user-policy --user-name $CLUSTER_NAME-acm-obs \
     --policy-arn ${S3_POLICY}
   

6. Create Access Keys

   read -r ACCESS_KEY_ID ACCESS_KEY < <(aws iam create-access-key \
     --user-name $CLUSTER_NAME-acm-obs \
     --query 'AccessKey.[AccessKeyId,SecretAccessKey]' --output text)
   

ACM Hub

Log into the OpenShift cluster that is running your ACM Hub. We’ll set up Observability here

1. Create a namespace for the observability

   oc new-project $NAMESPACE
   

2. Generate a pull secret (this will check if the pull secret exists, if not, it will create it)

   DOCKER_CONFIG_JSON=`oc extract secret/multiclusterhub-operator-pull-secret -n open-cluster-management --to=-` || \
     DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-` && \
     oc create secret generic multiclusterhub-operator-pull-secret \
     -n open-cluster-management-observability \
     --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
     --type=kubernetes.io/dockerconfigjson
   

3. Create a Secret containing your S3 details

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: thanos-object-storage
     namespace: open-cluster-management-observability
   type: Opaque
   stringData:
     thanos.yaml: |
       type: s3
       config:
         bucket: $S3_BUCKET
         endpoint: s3.$REGION.amazonaws.com
         signature_version2: false
         access_key: $ACCESS_KEY_ID
         secret_key: $ACCESS_KEY    
   EOF
   

4. Create a CR for MulticlusterHub

   cat << EOF | kubectl apply -f -
   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     observabilityAddonSpec: {}
     storageConfig:
       metricObjectStorage:
         name: thanos-object-storage
         key: thanos.yaml
   EOF
   

Access ACM Observability

1. Log into Advanced Cluster management and access the new Grafana dashboard