IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

Configuring the Cluster Log Forwarder for CloudWatch Logs and STS

Authors: Thatcher Hubbard, Connor Wooley, Paul Czarkowski, Charlotte Fung
Last Editor: Paul Czarkowski
Published Date: 19 August 2022
Modified Date: 19 September 2023

This guide shows how to deploy the Cluster Log Forwarder operator and configure it to use STS authentication to forward logs to CloudWatch.

Prerequisites

  • A ROSA cluster (configured with STS)
  • The jq cli command
  • The aws cli command

Environment Setup

  1. Configure the following environment variables

    Change the cluster name to match your ROSA cluster and ensure you’re logged into the cluster as an Administrator. Ensure all fields are outputted correctly before moving on.

    export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
export AWS_PAGER=""
export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/clf-cloudwatch-sts"
mkdir -p ${SCRATCH}
echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

Prepare AWS Account

  1. Create an IAM Policy for OpenShift Log Forwarding

    POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaCloudWatch'].{ARN:Arn}" --output text)
if [[ -z "${POLICY_ARN}" ]]; then
cat << EOF > ${SCRATCH}/policy.json
{
"Version": "2012-10-17",
"Statement": [
   {
         "Effect": "Allow",
         "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:DescribeLogGroups",
            "logs:DescribeLogStreams",
            "logs:PutLogEvents",
            "logs:PutRetentionPolicy"
         ],
         "Resource": "arn:aws:logs:*:*:*"
   }
]
}
EOF
POLICY_ARN=$(aws iam create-policy --policy-name "RosaCloudWatch" \
--policy-document file:///${SCRATCH}/policy.json --query Policy.Arn --output text)
fi
echo ${POLICY_ARN}

  2. Create an IAM Role trust policy for the cluster

    cat <<EOF > ${SCRATCH}/trust-policy.json
{
   "Version": "2012-10-17",
   "Statement": [{
     "Effect": "Allow",
     "Principal": {
       "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
       "StringEquals": {
         "${OIDC_ENDPOINT}:sub": "system:serviceaccount:openshift-logging:logcollector"
       }
     }
   }]
}
EOF
ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
   --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
   --query Role.Arn --output text)
echo ${ROLE_ARN}

  3. Attach the IAM Policy to the IAM Role

    aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
--policy-arn ${POLICY_ARN}

Deploy Operators

  1. Deploy the Cluster Logging operator

    cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
   operators.coreos.com/cluster-logging.openshift-logging: ""
  name: cluster-logging
  namespace: openshift-logging
spec:
  channel: stable
  installPlanApproval: Automatic
  name: cluster-logging
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

  2. Create a secret

    cat << EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: cloudwatch-credentials
  namespace: openshift-logging
stringData:
  role_arn: $ROLE_ARN
EOF

Configure Cluster Logging

  1. Create a cluster log forwarding resource

    cat << EOF | oc apply -f -
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: cw
      type: cloudwatch
      cloudwatch:
        groupBy: namespaceName
        groupPrefix: rosa-${ROSA_CLUSTER_NAME}
        region: ${REGION}
      secret:
        name: cloudwatch-credentials
  pipelines:
     - name: to-cloudwatch
       inputRefs:
         - infrastructure
         - audit
         - application
       outputRefs:
         - cw
EOF

  2. Create a cluster logging resource

    cat << EOF | oc apply -f -
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
       type: vector
  forwarder:
  managementState: Managed
EOF

Check AWS CloudWatch for logs

  1. Use the AWS console or CLI to validate that there are log streams from the cluster

    Note: If this is a fresh cluster you may not see a log group for application logs as there are no applications running yet.

    aws logs describe-log-groups --log-group-name-prefix rosa-${ROSA_CLUSTER_NAME}

    {
   "logGroups": [
      {
            "logGroupName": "rosa-xxxx.audit",
            "creationTime": 1661286368369,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.audit:*",
            "storedBytes": 0
      },
      {
            "logGroupName": "rosa-xxxx.infrastructure",
            "creationTime": 1661286369821,
            "metricFilterCount": 0,
            "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.infrastructure:*",
            "storedBytes": 0
      }
   ]
}

Cleanup

  1. Delete the Cluster Log Forwarding resource

    oc delete -n openshift-logging clusterlogforwarder instance

  2. Delete the Cluster Logging resource

    oc delete -n openshift-logging clusterlogging instance

  3. Detach the IAM Policy to the IAM Role

    aws iam detach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
--policy-arn "${POLICY_ARN}"

  4. Delete the IAM Role

    aws iam delete-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch"

  5. Delete the IAM Policy

    Only run this command if there are no other resources using the Policy

    aws iam delete-policy --policy-arn "${POLICY_ARN}"

  6. Delete the CloudWatch Log Groups

    aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.audit"
aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.infrastructure"