IMPORTANT NOTE: This site is not official Red Hat documentation and is provided for informational purposes only. These guides may be experimental, proof of concept, or early adoption. Officially supported documentation is available at docs.openshift.com and access.redhat.com.

OpenShift Logging

A guid to shipping logs and metrics on OpenShift

Author: Aaron Aldrich

Prerequisites

  1. OpenShift CLI (oc)
  2. Rights to install operators on the cluster

Setup OpenShift Logging

This is for setup of centralized logging on OpenShift making use of Elasticsearch OSS edition. This largely follows the processes outlined in the OpenShift documentation here. Retention and storage considerations are reviewed in Red Hat’s primary source documentation.

This setup is primarily concerned with simplicity and basic log searching. Consequently it is insufficient for long-lived retention or for advanced visualization of logs. For more advanced observability setups, you’ll want to look at Forwarding Logs to Third Party Systems

  1. Create a namespace for the OpenShift Elasticsearch Operator.

    This is necessary to avoid potential conflicts with community operators that could send similarly named metrics/logs into the stack.

    oc create -f - <<EOF
    apiVersion: v1
    kind: Namespace
    metadata:
    name: openshift-operators-redhat
    annotations:
      openshift.io/node-selector: ""
    labels:
      openshift.io/cluster-monitoring: "true"
    EOF
    
  2. Create a namespace for the OpenShift Logging Operator

    oc create -f - <<EOF
    apiVersion: v1
    kind: Namespace
    metadata:
    name: openshift-logging
    annotations:
      openshift.io/node-selector: ""
    labels:
      openshift.io/cluster-monitoring: "true"
    EOF
    
  3. Install the OpenShift Elasticsearch Operator by creating the following objects:

    1. Operator Group for OpenShift Elasticsearch Operator

      oc create -f - <<EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-operators-redhat
        namespace: openshift-operators-redhat
      spec: {}
      EOF
      
    2. Subscription object to subscribe a Namespace to the OpenShift Elasticsearch Operator

      oc create -f - <<EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: "elasticsearch-operator"
        namespace: "openshift-operators-redhat"
      spec:
        channel: "stable"
        installPlanApproval: "Automatic"
        source: "redhat-operators"
        sourceNamespace: "openshift-marketplace"
        name: "elasticsearch-operator"
      EOF
      
    3. Verify Operator Installation

      oc get csv --all-namespaces
      

      Example Output

      NAMESPACE                                               NAME                                            DISPLAY                  VERSION               REPLACES   PHASE
      default                                                 elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      kube-node-lease                                         elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      kube-public                                             elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      kube-system                                             elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      openshift-apiserver-operator                            elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      openshift-apiserver                                     elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      openshift-authentication-operator                       elasticsearch-operator.5.0.0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      openshift-authentication                                elasticsearch-operator.5.0. 0-202007012112.p0    OpenShift Elasticsearch Operator   5.0.0-202007012112.p0               Succeeded
      ...
      
  4. Install the Red Hat OpenShift Logging Operator by creating the following objects:

    1. The Cluster Logging OperatorGroup

      oc create -f - <<EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: cluster-logging
        namespace: openshift-logging
      spec:
        targetNamespaces:
        - openshift-logging
      EOF
      
    2. Subscription Object to subscribe a Namespace to the Red Hat OpenShift Logging Operator

      oc create -f - <<EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: cluster-logging
        namespace: openshift-logging
      spec:
        channel: "stable"
        name: cluster-logging
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF
      
    3. Verify the Operator installation, the PHASE should be Succeeded

    oc get csv -n openshift-logging
    

    Example Output

    NAME                              DISPLAY                            VERSION    REPLACES   PHASE
    cluster-logging.5.0.5-11          Red Hat OpenShift Logging          5.0.5-11              Succeeded
    elasticsearch-operator.5.0.5-11   OpenShift Elasticsearch Operator   5.0.5-11              Succeeded
    
  5. Create an OpenShift Logging instance:

    NOTE: For the storageClassName below, you will need to adjust for the platform on which you’re running OpenShift. managed-premium as listed below is for Azure Red Hat OpenShift (ARO). You can verify your available storage classes with oc get storageClasses

    oc create -f - <<EOF
    apiVersion: "logging.openshift.io/v1"
    kind: "ClusterLogging"
    metadata:
      name: "instance"
      namespace: "openshift-logging"
    spec:
      managementState: "Managed"
      logStore:
        type: "elasticsearch"
        retentionPolicy:
          application:
            maxAge: 1d
          infra:
            maxAge: 7d
          audit:
            maxAge: 7d
        elasticsearch:
          nodeCount: 3
          storage:
            storageClassName: "managed-premium"
            size: 200G
          resources:
            requests:
              memory: "8Gi"
          proxy:
            resources:
              limits:
                memory: 256Mi
              requests:
                memory: 256Mi
          redundancyPolicy: "SingleRedundancy"
      visualization:
        type: "kibana"
        kibana:
          replicas: 1
      curation:
        type: "curator"
        curator:
          schedule: "30 3 * * *"
      collection:
        logs:
          type: "fluentd"
          fluentd: {}
    EOF
    
  6. It will take a few minutes for everything to start up. You can monitor this progress by watching the pods.

    watch oc get pods -n openshift-logging
    
  7. Your logging instances are now configured and recieving logs. To view them, you will need to log into your Kibana instance and create the appropriate index patterns. For more information on index patterns, see the Kibana documentation.

    NOTE: The following restrictions and notes apply to index patterns:

    • All users can view the app- logs for namespaces they have access to
    • Only cluster-admins can view the infra- and audit- logs
    • For best accuracy, use the @timestamp field for determining chronology