Logging and monitoring

This page explains how to use the logging and monitoring options that are available with Anthos private mode.

This page is for platform administrators.

Monitoring Dashboards

Sample dashboards for clusters and nodes metrics monitoring are available from the management center.

Monitoring Tab

The Grafana Monitoring Dashboard button opens the Grafana application hosted on the admin cluster. Grafana access is controlled through the management center. If you are logged in to management center, no additional login is required.

Dashboards

After logging in, a list of sample dashboards are available for demonstration:

Grafana Dashboards

Logs

The Grafana Loki Logs button opens the Grafana Explorer where logs, including Kubernetes system logs, can be queried. Logs are labeled as described in the following section.

Log labels

Log Type Labels
Container Logs
  • cluster=CLUSTER_NAME
  • resources=k8s_container
  • namespace=NAMESPACE_NAME
  • node=NODE_NAME
  • pod=POD_NAME
  • container=CONTAINER_NAME
  • Other Kubernetes Labels
Node Journald Logs
  • cluster=CLUSTER_NAME
  • resources=k8s_node
  • node=NODE_NAME
  • identifier=SYSLOG_IDENTIFIER
  • machine=MACHINE_ID
Audit Logs
  • cluster=CLUSTER_NAME
  • resource=k8s_audit
  • level=LEVEL
  • vert=VERB

All Kubernetes labels and a cluster label are included as log labels.

Sample queries

  • The following query selects container logs:

    {cluster="admin", namespace="kube-system", resources="k8s_container", container="kube-apiserver"}
    

    Grafana Query container logs

  • The following query selects node Journald logs:

    {cluster="admin", resources="k8s_node"}
    

    Grafana Query node Logs

  • The following query selects audit Logs:

    {cluster="admin", resources="k8s_audit"}
    

    Grafana Query audit Logs

Configuring a custom StorageClass

Anthos private mode uses PersistentVolumeClaims to provision space to store metrics, logs, dashboards, and alerts. When you create a cluster, the cluster automatically creates a LogMon custom resource. To override the default StorageClass, complete the following steps:

  1. Run the following command to open your LogMon custom resource in a command-line editor:

    kubectl --kubeconfig=ADMIN_OIDC_KUBECONFIG -n kube-system edit logmon logmon-default
    
  2. In the LogMon custom resource, update the storageClassName under the spec field:

    apiVersion: addons.gke.io/v1alpha1
    kind: Logmon
    metadata:
      # Don't change the name
      name: logmon-default
      # Don't change the namespace
      namespace: kube-system
    spec:
      system_logs:
        outputs:
          default_loki:
            deployment:
              storageClassName: anthos-system # configure the new storage class here 
      system_metrics:
        outputs:
          default_prometheus:
            deployment:
              storageClassName: anthos-system # configure the new storage class here
    
  3. To save the changes to the LogMon custom resource, save and exit your command-line editor.

Configuring storage size

When you create a cluster, the cluster automatically creates a LogMon custom resource. To override the storage size for metrics, logs, dashboards, and alerts, complete the following steps:

  1. Run the following command to open your LogMon custom resource in a command-line editor:

    kubectl --kubeconfig=ADMIN_OIDC_KUBECONFIG -n kube-system edit logmon logmon-default
    
  2. In the LogMon custom resource, update the storageSize under the spec field:

    apiVersion: addons.gke.io/v1alpha1
    kind: Logmon
    metadata:
      # Don't change the name
      name: logmon-default
      # Don't change the namespace
      namespace: kube-system
    spec:
      system_logs:
        outputs:
          default_loki:
            deployment:
              components:
                loki:
                   storageSize: 20Gi # configure the new storage size for logs here
      system_metrics:
        outputs:
          default_prometheus:
            deployment:
              components:
                alertmanager:
                   storageSize: 1Gi # configure the new storage size for alerts here
                grafana:
                   storageSize: 1Gi # configure the new storage size for dashboards here
                prometheus:
                   storageSize: 20Gi # configure the new storage size for metrics here
    ````
    
  3. To save the changes to the LogMon custom resource, save and exit your command-line editor.

Configuring retention time

To configure the retention time for logs in the Prometheus and Loki agents running in the cluster, complete the following steps:

  1. Run the following command to open your LogMon custom resource in a command-line editor:

    kubectl --kubeconfig=ADMIN_OIDC_KUBECONFIG -n kube-system edit logmon logmon-default
    
  2. In the LogMon custom resource, update the retentionTime under the spec field:

    apiVersion: addons.gke.io/v1alpha1
    kind: Logmon
    metadata:
      # Don't change the name
      name: logmon-default
      # Don't change the namespace
      namespace: kube-system
    spec:
      system_logs:
        outputs:
          default_loki:
            deployment:
              retentionPolicy:
                retentionTime: 720h # configure the new retention time for logs here
      system_metrics:
        outputs:
          default_prometheus:
            deployment:
              retentionPolicy:
                retentionTime: 720h # configure the new retention time for metrics here
    
  3. To save the changes to the LogMon custom resource, save and exit your command-line editor.

Configuring fluent-bit additional output

Exporting logs to other destinations in addition to the Loki running in the cluster is supported. See the list of supported destinations. To configure the additional logs output destination, complete the following steps:

  1. Create a configmap in the kube-system namespace with the logmon: system_logs label. The additional output configuration has the same syntax as fluent-bit output plugins, and should be added to the output.conf file under the data field. See the sample config files.

    apiVersion: v1
    kind: ConfigMap
    metadata:
      # The name should match the configmap name specified in step 3.
      name: <customized-system-logs-fluent-bit-output-config>
      # Don't change the namespace
      namespace: kube-system
      labels:
        # This label is required.
        logmon: system_logs
    data:
      # The file name must be output.conf
      output.conf: |
        # Add a customized fluent-bit output plugin configuration here
    
  2. Run the following command to open your LogMon custom resource in a command-line editor:

    kubectl --kubeconfig=ADMIN_OIDC_KUBECONFIG -n kube-system edit logmon logmon-default
    
  3. In the LogMon custom resource, add the additionalOutput field under the spec/system_logs/outputs field.

    apiVersion: addons.gke.io/v1alpha1
    kind: Logmon
    metadata:
      # Don't change the name
      name: logmon-default
      # Don't change the namespace
      namespace: kube-system
    spec:
      system_logs:
        outputs:
          additionalOutput:
            fluentbitConfigmaps:
            # The name should match the configmap name created in step 1.
            - "<customized-system-logs-fluent-bit-output-config>"
            # Scheme: []v1.VolumeMount. Add volumeMounts if necessary
            volumeMounts:
            - ...
            - ...
            # Scheme: []v1.Volume. Add volumes if necessary
            volumes:
            - ...
            - ...
    
  4. To save the changes to the LogMon custom resource, save and exit your command-line editor.