Configuring log forwarding for CloudWatch logs and STS | Tutorials

Setting up your environment
Preparing your AWS account
Deploying Operators
Configuring cluster logging
Checking CloudWatch for logs
Cleaning up your resources

Use this tutorial to deploy the Red Hat OpenShift Logging Operator and configure it to use Security Token Services (STS) authentication to forward logs to CloudWatch.

Prerequisites

A Red Hat OpenShift Service on AWS (ROSA) Classic cluster
The jq command-line interface (CLI)
The Amazon Web Services (AWS) CLI (aws)

edit

Setting up your environment

Configure the following environment variables, changing the cluster name to suit your cluster:

You must be logged in as an administrator.

$ export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
$ export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
$ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
$ export AWS_PAGER=""
$ export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/clf-cloudwatch-sts"
$ mkdir -p ${SCRATCH}

Ensure all fields output correctly before moving to the next section:

$ echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

Preparing your AWS account

Create an Identity Access Management (IAM) policy for logging:

$ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaCloudWatch'].{ARN:Arn}" --output text)
$ if [[ -z "${POLICY_ARN}" ]]; then
cat << EOF > ${SCRATCH}/policy.json
{
  "Version": "2012-10-17",
  "Statement": [
     {
           "Effect": "Allow",
           "Action": [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:DescribeLogGroups",
              "logs:DescribeLogStreams",
              "logs:PutLogEvents",
              "logs:PutRetentionPolicy"
           ],
           "Resource": "arn:aws:logs:*:*:*"
     }
]
}
EOF
POLICY_ARN=$(aws iam create-policy --policy-name "RosaCloudWatch" \
--policy-document file:///${SCRATCH}/policy.json --query Policy.Arn --output text)
fi
$ echo ${POLICY_ARN}

Create an IAM role trust policy for the cluster:

$ cat <<EOF > ${SCRATCH}/trust-policy.json
{
   "Version": "2012-10-17",
   "Statement": [{
     "Effect": "Allow",
     "Principal": {
       "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
       "StringEquals": {
         "${OIDC_ENDPOINT}:sub": "system:serviceaccount:openshift-logging:logcollector"
       }
     }
   }]
}
EOF
$ ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
      --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
      --query Role.Arn --output text)
$ echo ${ROLE_ARN}

Attach the IAM policy to the IAM role:

$ aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
   --policy-arn ${POLICY_ARN}

Deploying Operators

Deploy the Red Hat OpenShift Logging Operator:

$  cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     labels:
      operators.coreos.com/cluster-logging.openshift-logging: ""
     name: cluster-logging
     namespace: openshift-logging
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: cluster-logging
     source: redhat-operators
     sourceNamespace: openshift-marketplace
EOF

Create a secret:

$  cat << EOF | oc apply -f -
  apiVersion: v1
  kind: Secret
  metadata:
    name: cloudwatch-credentials
    namespace: openshift-logging
  stringData:
    role_arn: $ROLE_ARN
EOF

Configuring cluster logging

Create a ClusterLogForwarder custom resource (CR):

$ cat << EOF | oc apply -f -
  apiVersion: "logging.openshift.io/v1"
  kind: ClusterLogForwarder
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    outputs:
      - name: cw
        type: cloudwatch
        cloudwatch:
          groupBy: namespaceName
          groupPrefix: rosa-${ROSA_CLUSTER_NAME}
          region: ${REGION}
        secret:
          name: cloudwatch-credentials
    pipelines:
      - name: to-cloudwatch
        inputRefs:
          - infrastructure
          - audit
          - application
        outputRefs:
          - cw
EOF

Create a ClusterLogging CR:

$ cat << EOF | oc apply -f -
  apiVersion: logging.openshift.io/v1
  kind: ClusterLogging
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    collection:
      logs:
         type: vector
    managementState: Managed
EOF

Checking CloudWatch for logs

Use either the AWS console or the AWS CLI to validate that there are log streams from the cluster.

To validate the logs in the AWS CLI, run the following command:

$ aws logs describe-log-groups --log-group-name-prefix rosa-${ROSA_CLUSTER_NAME}

Sample output

{
  "logGroups": [
      {
        "logGroupName": "rosa-xxxx.audit",
        "creationTime": 1661286368369,
        "metricFilterCount": 0,
        "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.audit:*",
        "storedBytes": 0
      },
      {
        "logGroupName": "rosa-xxxx.infrastructure",
        "creationTime": 1661286369821,
        "metricFilterCount": 0,
        "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.infrastructure:*",
        "storedBytes": 0
      }
  ]
}

If this is a new cluster, you might not see a log group for application logs as applications are not yet running.

Cleaning up your resources

Delete the ClusterLogForwarder CR:

$ oc delete -n openshift-logging clusterlogforwarder instance

Delete the ClusterLogging CR:

$ oc delete -n openshift-logging clusterlogging instance

Detach the IAM policy to the IAM role:

$ aws iam detach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
   --policy-arn "${POLICY_ARN}"

Delete the IAM role:

$ aws iam delete-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch"

Delete the IAM policy:

Only delete the IAM policy if there are no other resources using the policy.
```
$ aws iam delete-policy --policy-arn "${POLICY_ARN}"
```

Delete the CloudWatch log groups:

$ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.audit"
$ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.infrastructure"