Checking cluster health

This page is for platform administrators.

This page describes how to check the health status of nodes, system Pods, and network connectivity in your cluster.

Check cluster health using actl

Run the following command to check the health of your cluster:

actl clusters baremetal check cluster CLUSTER_NAME --kubeconfig=ADMIN_KUBECONFIG

The command checks the following:

  • The health of the nodes in the cluster, such as kubelet running status, containerd status, disk capacity, and registry mirror reachability. etc.
  • The health of bare metal system Pods such anthos-cluster-operator if the cluster is an admin cluster.
  • Network connectivity between nodes, for example L2 connectivity between master nodes.

Here's an example of a successful health check:

Please check the logs at actl-workspace/user-1/log/check-cluster-20210616-215509/check-cluster.log
[2021-06-16 21:55:16+0000] Waiting for health check job to finish... OK
[2021-06-16 21:55:46+0000] - Validation Category: machines, network, add-ons and kubernetes
[2021-06-16 21:55:46+0000]  - [PASSED] add-ons
[2021-06-16 21:55:46+0000]  - [PASSED] kubernetes
[2021-06-16 21:55:46+0000]  - [PASSED] node-network
[2021-06-16 21:55:46+0000]  - [PASSED] 10.200.0.6
[2021-06-16 21:55:46+0000]  - [PASSED] 10.200.0.7
[2021-06-16 21:55:46+0000]  - [PASSED] 10.200.0.8
[2021-06-16 21:55:46+0000] Flushing logs... OK

Here's an example of a health check failure:

Please check the logs at actl-workspace/user-1/log/check-cluster-20210807-001826/check-cluster.log
[2021-08-07 00:18:32+0000] Waiting for health check job to finish... OK
[2021-08-07 00:20:52+0000] - Validation Category: machines, network, add-ons and kubernetes
[2021-08-07 00:20:52+0000]  - [FAILED] 10.200.0.6
      actl-workspace/user-1/log/check-cluster-20210807-001826/10.200.0.6
[2021-08-07 00:20:52+0000]  - [FAILED] 10.200.0.7
      actl-workspace/user-1/log/check-cluster-20210807-001826/10.200.0.7
[2021-08-07 00:20:52+0000]  - [FAILED] 10.200.0.8
      actl-workspace/user-1/log/check-cluster-20210807-001826/10.200.0.8
[2021-08-07 00:20:52+0000]  - [PASSED] add-ons
[2021-08-07 00:20:52+0000]  - [PASSED] kubernetes
[2021-08-07 00:20:52+0000]  - [PASSED] node-network
[2021-08-07 00:20:52+0000] Flushing logs... OK
[2021-08-07 00:20:52+0000] Error waiting for health check job: health check failed