Home GitHub

Disclaimer: Mobb.ninja is not official Red Hat documentation - These guides may be experimental, proof of concept or early adoption. Officially supported documentation is available at https://docs.openshift.com.

Advanced Cluster Management Observability on ROSA

This document will take you through deploying ACM Observability on an STS enabled ROSA cluster. see here for the original documentation.

Note: This guide uses an unsupported (by ACM) method to utilize ROSA’s STS authentication back into AWS S3 and is not advised for production use at this time.

Prerequisites

Set up environment

  1. Set environment variables
    export CLUSTER_NAME=my-cluster
    export S3_BUCKET=$CLUSTER_NAME-acm-observability
    export REGION=us-east-2
    export NAMESPACE=open-cluster-management-observability
    export SA=tbd
    export SCRATCH_DIR=/tmp/scratch
    export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
    export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    export AWS_PAGER=""
    rm -rf $SCRATCH_DIR
    mkdir -p $SCRATCH_DIR

Prepare AWS Account

  1. Create an S3 bucket

     aws s3 mb s3://$S3_BUCKET
    
  2. Create a Policy for access to S3

    cat <<EOF > $SCRATCH_DIR/s3-policy.json
    {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "Statement",
             "Effect": "Allow",
             "Action": [
                 "s3:ListBucket",
                 "s3:GetObject",
                 "s3:DeleteObject",
                 "s3:PutObject",
                 "s3:PutObjectAcl",
                 "s3:CreateBucket",
                 "s3:DeleteBucket"
             ],
             "Resource": [
                 "arn:aws:s3:::$S3_BUCKET/*",
                 "arn:aws:s3:::$S3_BUCKET"
             ]
         }
     ]
    }
    EOF
    
  3. Apply the Policy

     S3_POLICY=$(aws iam create-policy --policy-name $CLUSTER_NAME-acm-obs \
       --policy-document file://$SCRATCH_DIR/s3-policy.json \
       --query 'Policy.Arn' --output text)
     echo $S3_POLICY
    
  4. Create a Trust Policy

    cat <<EOF > $SCRATCH_DIR/TrustPolicy.json
    {
      "Version": "2012-10-17",
      "Statement": [
     {
       "Effect": "Allow",
       "Principal": {
         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity",
       "Condition": {
         "StringEquals": {
           "${OIDC_PROVIDER}:sub": [
             "system:serviceaccount:${NAMESPACE}:observability-thanos-query",
             "system:serviceaccount:${NAMESPACE}:observability-thanos-store-shard",
             "system:serviceaccount:${NAMESPACE}:observability-thanos-compact"
           ]
         }
       }
     }
      ]
    }
    EOF
    
  5. Create Role for AWS Prometheus and CloudWatch

     S3_ROLE=$(aws iam create-role \
       --role-name "$CLUSTER_NAME-acm-obs-s3" \
       --assume-role-policy-document file://$SCRATCH_DIR/TrustPolicy.json \
       --query "Role.Arn" --output text)
     echo $S3_ROLE
    
  6. Attach the Policies to the Role

     aws iam attach-role-policy \
       --role-name "$CLUSTER_NAME-acm-obs-s3" \
       --policy-arn $S3_POLICY
    

ACM Hub

Log into the OpenShift cluster that is running your ACM Hub. We’ll set up Observability here

  1. Create a namespace for the observability

     oc new-project $NAMESPACE
    
  2. Generate a pull secret (this will check if the pull secret exists, if not, it will create it)

    DOCKER_CONFIG_JSON=`oc extract secret/multiclusterhub-operator-pull-secret -n open-cluster-management --to=-` || \
     DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-` && \
     oc create secret generic multiclusterhub-operator-pull-secret \
     -n open-cluster-management-observability \
     --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
     --type=kubernetes.io/dockerconfigjson
    
  3. Create a Secret containing your S3 details

    cat << EOF | kubectl apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: thanos-object-storage
      namespace: open-cluster-management-observability
    type: Opaque
    stringData:
      thanos.yaml: |
     type: s3
     config:
       bucket: $S3_BUCKET
       endpoint: s3.$REGION.amazonaws.com
       signature_version2: false
    EOF
    
  4. Create a CR for MulticlusterHub

    cat << EOF | kubectl apply -f -
    apiVersion: observability.open-cluster-management.io/v1beta2
    kind: MultiClusterObservability
    metadata:
      name: observability
    spec:
      observabilityAddonSpec: {}
      storageConfig:
     metricObjectStorage:
       name: thanos-object-storage
       key: thanos.yaml
    EOF
    
  5. Annotate the service accounts to use STS

    oc annotate -n open-cluster-management-observability serviceaccount observability-thanos-query \
       eks.amazonaws.com/role-arn=$S3_ROLE
    oc annotate -n open-cluster-management-observability serviceaccount observability-thanos-store-shard \
       eks.amazonaws.com/role-arn=$S3_ROLE
    oc annotate -n open-cluster-management-observability serviceaccount observability-thanos-compact \
       eks.amazonaws.com/role-arn=$S3_ROLE
    
  6. Restart thanos pods to use the new credentials

    kubectl rollout restart deployment/observability-thanos-query
    kubectl rollout restart statefulset/observability-thanos-compact
    kubectl rollout restart statefulset/observability-thanos-store-shard-0
    kubectl rollout restart statefulset/observability-thanos-store-shard-1
    kubectl rollout restart statefulset/observability-thanos-store-shard-2
    
  7. Check for any pods that aren’t running

     kubectl get pods --field-selector status.phase!=Running
    

    Delete any pods that are not running in order to have them recreated with the correct credentials

     kubectl delete pods $(kubectl get pods --field-selector status.phase!=Running -o name | xargs)
    
  8. Log into Advanced Cluster management and access the new Grafana dashboard

ACM Grafana Dashboard