Anthos private mode architecture

This page outlines the architecture for a proof of concept deployment of Anthos private mode. The cluster technology for this proof of concept is based on Anthos on bare metal.

Anthos private mode terminology

  • Admin workstation: A Linux machine used to create the admin cluster and perform other low-level administrative tasks.
  • Admin cluster: An Anthos cluster used to manage the user clusters. Only administrative workloads run on this cluster. The admin cluster is not used to deploy apps. It is usually smaller than user clusters. The admin cluster is where the local control plane and web interface are installed.
  • User cluster: An Anthos cluster where you deploy your apps. A single admin cluster can manage multiple user clusters.
  • Control plane node: This node runs the core components of a given cluster, such as the Kubernetes API server or etcd. For each cluster, you have either one control plane node (non-highly available setup) or three control plane nodes (highly available setup).
  • Worker node: A node where you deploy workloads in a given cluster. You can have many worker nodes in a cluster.
  • Load balancer node: A node where the load balancer for Anthos is running. For Anthos on bare metal (and Anthos private mode), this load balancer is MetalLB. In this proof of concept, you deploy the load balancers on the control plane nodes. Those nodes are both control plane nodes and load balancer nodes. In a production scenario, you can move the load balancers to worker nodes to protect the Kubernetes API server that is running on the control plane nodes from any interference. You can also dedicate worker nodes for load balancing. Google does not recommend running more than 10 load balancer nodes.

Architecture

Depending on the hardware that you have available, and what capabilities of Anthos private mode you want to test, we suggest one of the following options:

Minimal installation Highly available installation Multi-cluster installation
  • 1 admin workstation
  • 2 nodes for the admin cluster: 1 control plane node, and 1 worker node
  • 2 nodes for the user cluster: 1 control plane node, and 1 worker node
  • Total: 5 nodes
  • Minimal installation diagram
  • 1 admin workstation
  • 6 nodes for the admin cluster: 3 control plane nodes, and 3 worker nodes
  • 6 nodes for the user cluster: 3 control plane nodes, and 3 worker nodes
  • Total: 13 nodes
  • Highly available installation diagram
  • 1 admin workstation
  • 2 nodes for the admin cluster: 1 control plane node, and 1 worker node
  • 3 user clusters with 2 nodes each: 1 control plane node, and 1 worker node
  • Total: 9 nodes
  • Multi-cluster installation diagram
  • Advantages:
    • Lower hardware footprint and cost
    • Can test all the Anthos private mode features except high-availability and multi-cluster capabilities
    Advantages:
    • Run larger workloads
    • Can test all the Anthos private mode features, including high-availability, but no multi-cluster capabilities
    Advantages:
    • Can test all the Anthos private mode features including multi-cluster capabilities, but not the high-availability of a single cluster.

    For all options:

    • The load balancers are running on the control plane nodes of each cluster.
    • You must meet the requirements.

    You can mix these options, for example, you can have a small two-node admin cluster (one control plane node and one worker node), and a large six-node user cluster. You can also create two separate Anthos private mode environments (such as development and staging): one with a minimal installation, and the second with a highly available installation. A cluster should always have an odd number (usually one or three) of control plane nodes to avoid split brain scenarios where two nodes each think that they are the leader.

    By default, the nodes use their local hard drive or SSD to provision PersistentVolumes. Local drives are not highly available, and can't move from one node to another. After the clusters are created, we recommend that you:

    1. Install a Container Storage Interface (CSI) driver compatible with your storage provider (see the list of Anthos Ready Storage Partners),
    2. Create a StorageClass backed by this CSI driver.
    3. Make this StorageClass the default one.

    Diagrams

    These diagrams show the high-level architecture of each option. The routers and switches represented in the diagrams are only examples: you can use the network architecture of your choice as long as it complies with the requirements.

    Minimal installation

    drawing

    Highly available installation

    To achieve a highly available installation, your network must also be highly available. The implementation of this installation depends on your existing network environment.

    drawing

    Multi-cluster installation

    drawing

    What's next