Anthos private mode installation quickstart

Anthos private mode is a version of Anthos that can run fully isolated from the internet in an air-gapped environment. You can use Anthos private mode to run highly regulated workloads on your premises, while still benefiting from many of the advantages of infrastructure built for the cloud.

Anthos private mode quickstart scope

This quickstart guides you through the installation of Anthos private mode as an infrastructure operator -- responsible for the infrastructure layer such as bare metal machines, networking, storage, and operating system. In this quickstart you'll learn how to:

  • Install and configure Anthos private mode using the minimal installation approach.
  • Deploy an admin cluster.
  • Create a user cluster and deploy an application to the user cluster.

Anthos private mode has two types of clusters:

  • Admin clusters are used to create and manage user clusters.
  • User clusters run your workloads.

You also have an admin workstation which is a single machine containing the tools necessary to install and manage your Anthos private mode deployment.

Before you begin

Make sure that you meet the technical requirements for Anthos private mode and that you have five or more machines that are ready to use. Previous Kubernetes and system management experience is necessary for this quickstart.

Prepare the environment

Getting access to Anthos private mode

Google must grant you access to Anthos private mode releases. Provide your Google point of contact with an email address for each tester so that we can grant them access to the download repository. These email addresses must be Google accounts.

After your Google point of contact confirms that you have access to the Anthos private mode releases, check that you have the permissions to download the Anthos private mode releases by running the following command:

gsutil ls gs://anthos-private-mode-release/

If you get an error, verify that gsutil is using the same Google account that you have provided your Google point of contact by running the gcloud auth list command.

Alternatively, you can visit the latest release bucket to verify that you have the correct permissions. You must be logged in with the Google account that you provided earlier.

If you have issues accessing the Anthos private mode releases, contact your Google point of contact for help.

Download Anthos private mode

In this section, you download the Anthos private mode release, which is contained in multiple files of several gigabytes each. Depending on your Internet connection, it might take a long time to download.

Download Anthos private mode

On your admin workstation, run the following commands:

# Login with the account granted access to Anthos private mode
gcloud auth login

# Download the script which helps download all the latest components
export VERSION=1.8.2-gke.X

# Please use official 'INSTALLER_DIGEST' value from https://cloud.google.com/anthos/private-mode/docs/1.8/installing/infrastructure#download
export INSTALLER_DIGEST=
export INSTALLER=get-anthos-private-mode-$VERSION.sh
gsutil cp gs://anthos-private-mode-release/$VERSION/$INSTALLER .
if [[ -n "$INSTALLER_DIGEST" ]]; then echo "$INSTALLER_DIGEST $INSTALLER" | sha256sum -c; fi && chmod +x $INSTALLER && ./$INSTALLER

# If you are working on a workstation shared with other users,
# we recommend that you revoke your credentials after downloading the release.
gcloud auth revoke YOUR_EMAIL_ADDRESS

Install remaining dependencies

After the Anthos private mode download completes, run the following commands:

cd anthos-baremetal-private-mode

# Add actl command line tool and tools directory to the PATH
export PATH=$PWD/bin:$PATH

# Download Harbor offline installer
curl -SL \
  https://github.com/goharbor/harbor/releases/download/v2.2.0/harbor-offline-installer-v2.2.0.tgz \
  --output "local-registry/harbor-offline-installer.tgz"

# Install docker-compose
curl -SL \
  https://github.com/docker/compose/releases/download/1.28.5/docker-compose-Linux-x86_64 \
  --output "local-registry/docker-compose"

You do not need an internet connection again until you deploy the demo app.

Set up the local container registry

Anthos private mode works by storing the Anthos container images in a local container registry.

Export the following environment variables.

export REGISTRY_HOST=REGISTRY_HOST
export REGISTRY_PASSWORD=REGISTRY_PASSWORD

# By default, a library public project is created,
# and you can also create other private projects via the container registry portal.
export PRIVATE_REGISTRY=${REGISTRY_HOST}/library

Replace the following:

  • REGISTRY_HOST is your registry IP address. If you want to install and use the Anthos private mode container registry on the admin workstation, use your admin workstation IP address here.

  • REGISTRY_PASSWORD is the value you set for your registry password.

Set up the Anthos private mode container registry

In this section, you set up a private container registry on the admin workstation. Run all the commands below from the admin workstation.

  1. If you don't have your own private container registry, install the Anthos private mode container registry.

    cd ~/anthos-baremetal-private-mode
    
    # Move it to a path under $PATH
    chmod a+x local-registry/docker-compose
    sudo cp local-registry/docker-compose /usr/bin
    
    # Install local registry
    ./local-registry/install.sh
    
  2. Log into the registry to verify that you have access. You might need to wait a few seconds if you receive an error.

    docker login ${REGISTRY_HOST} -u admin -p ${REGISTRY_PASSWORD}
    

The container registry is available at https://REGISTRY_HOST/ after the service is started. The login credentials are username admin and REGISTRY_PASSWORD for the password.

Upload images into the container registry

In this section, you upload the Anthos private mode container images to your container registry.

Prepare and upload the Anthos private mode container images to your container registry. If prompted, choose the Use that credential option, or enter the new credentials.

actl images push --private-registry=${PRIVATE_REGISTRY} \
    --images ~/anthos-baremetal-private-mode

Note: If you use an HTTP proxy on your workstation, you may need to unset the following environment variables for the actl images push command to work:

unset http_proxy
unset https_proxy
unset HTTP_PROXY
unset HTTPS_PROXY

Install Management Center

In this section, you install an Anthos admin cluster on bare metal machines, and then install the Anthos private mode management center on top of the admin cluster.

Prepare the admin cluster config

Update the ./actl-workspace/admin/admin.yaml file with settings that match your environment.

vi ./actl-workspace/admin/admin.yaml

Complete the following fields:

  • The sshPrivateKeyPath is the key that's used during installation to access the other nodes that will be part of the cluster. This is the private counterpart of the public key you distributed to the nodes in the Prerequisites.

  • The registryMirrors identifies the location of the all registry mirror where the docker images are stored.

    • registryMirrors.endpoint is the endpoint of the mirror. The endpoint's format is URL_SCHEME://REGISTRY_HOST/v2/IMAGE_PREFIX . For example, if you created the registry on the workstation, then set the endpoint to https://REGISTRY_HOST/v2/library. Note that the /v2/ in between REGISTRY_HOST and IMAGE_PREFIX is necessary.

    • registryMirrors.pullCredentialConfigPath is the path for the credentials. Usually, this is something like /home/USER/.docker/config.json.

    • The registryMirrors.caCertPath is the location of the Certificate Authority that the registry will use to encrypt traffic. This certificate is distributed to the nodes during the installation process to allow them to pull images from the registry. The path is typically /etc/docker/certs.d/REGISTRY_HOST/ca.crt.

  • The controlPlane spec defines which nodes are part of the control plane.

    controlPlane:
      nodePoolSpec:
        nodes:
        # Control plane node pools. Typically, this is either a single machine
        # or 3 machines if using a high availability deployment.
        - address: 10.200.0.3
    
  • The loadBalancer spec defines which type of load balancer is used. The default configuration installs a load balancer onto the running cluster. In this case, at minimum, you need to set the controlPlaneVIP to identify a virtual IP (VIP) address pool to expose the Kubernetes API Server and the addressPools to allocate VIPs on demand to services requesting them. For example:

    loadBalancer:
      mode: bundled
      # There are two load balancer VIPs: one for the control plane and one for the
      # L7 Ingress service.
      # The VIPs must be in the same subnet as the load balancer nodes.
      vips:
        # ControlPlaneVIP specifies the VIP to connect to the Kubernetes API server.
        # This address must not be in the address pools below.
        controlPlaneVIP: 10.200.0.50
      # AddressPools is a list of non-overlapping IP ranges for the data plane load
      # balancer.
      # All addresses must be in the same subnet as the load balancer nodes.
      # Address pool configuration is only valid for 'bundled' LB mode in non-admin
      # clusters.
      addressPools:
      - name: pool1
        addresses:
        # Each address must be either in the CIDR form (1.2.3.0/24)
        # or range form (1.2.3.1-1.2.3.5).
        - 10.200.0.51-10.200.0.59
    
  • The last section of the Cluster spec specifies the loginUser. This login must have access to the SSH key specified below and is the user for which the key was distributed in the Prerequisites.

    nodeAccess:
      loginUser: LOGIN_USERNAME
    

    Replace LOGIN_USERNAME with the username used to log into the user cluster.

  • The nodePool spec identifies the nodes that will host Management Center Console, Prometheus, Grafana and other services on the admin cluster. These nodes are the worker nodes of the admin cluster. For example:

    apiVersion: baremetal.cluster.gke.io/v1
    kind: NodePool
    metadata:
      name: node-pool-1
      namespace: cluster-admin
    spec:
      # Cannot be changed, must be admin
      clusterName: admin
      nodes:
      - address: 10.200.0.6
    

Create the admin cluster

Run the following command to create the admin cluster. It takes approximately 30 minutes.

cd ~/anthos-baremetal-private-mode
actl clusters baremetal create admin

Once the admin cluster is created, check that you have access to it.

export ADMIN_KUBECONFIG=$(pwd)/bmctl-workspace/admin/admin-kubeconfig
KUBECONFIG=${ADMIN_KUBECONFIG} kubectl get nodes

Install Management Center

  1. Install Management Center onto the admin cluster with the default configuration.

    cd ~/anthos-baremetal-private-mode
    actl platform management-center create
    
  2. Access the Anthos private mode management center in your browser.

    actl platform management-center describe --kubeconfig=${ADMIN_KUBECONFIG}
    

The Anthos private mode management center URL is returned.

Register resources

Register inventory machines

To create a user cluster, Anthos needs a pool of idle machines available. In this section, you register inventory machines to the Anthos admin cluster to make them available.

kubectl apply -f <path/to/example-machine.yaml> --kubeconfig=${ADMIN_KUBECONFIG}

Example:

apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: InventoryMachine
metadata:
  name: IP_ADDRESS
  labels:
    "KEY": "VALUE"
spec:
  address: IP_ADDRESS

Replace the following:

  • IP_ADDRESS: the IP address of the machine, for example 10.200.0.10.
  • KEY:VALUE: a key:value pair, for example "rack": "r1020" to indicate a rack location.

The labels are free-form key-value pairs that are attached to the resource. These labels can be used later in Management Center to filter machines. For example, you can use the labels to identify rack location or special hardware configurations.

Create user clusters

In the Clusters page of Management Center, click Create to create a new Anthos user cluster. Once the cluster is created, it is automatically registered in the management center and shows up in the cluster dashboard immediately.

Create a Cluster

It might take up to 15 minutes to install and become ready. While you wait, you can check the status of the user cluster with the following command:

kubectl get Cluster -n USER_CLUSTER_NAME --kubeconfig=${ADMIN_KUBECONFIG}

Replace USER_CLUSTER_NAME with the name of the cluster you created.

Once the status is ready, the cluster is successfully created and registered to the admin cluster.

Get access to the user cluster

After the user cluster is created, you can download the kubeconfig via the Management Center.

Deploy demo app

You can explore Anthos private mode by trying out the Online Boutique sample application.

Download the image

The steps in this section require an internet connection.

  1. Download the sample repo:

    git clone https://github.com/GoogleCloudPlatform/microservices-demo.git/ online-boutique
    
  2. Download the images:

    IMAGES=$(cat online-boutique/release/kubernetes-manifests.yaml | grep image | sed "s/ *image: //")
    actl images save --output online-boutique-images ${IMAGES}
    

Deploy the sample

  1. Push the images to the private registry:

    actl images push --images online-boutique-images --private-registry=PRIVATE_REGISTRY
    

    Replace PRIVATE_REGISTRY with the name of your private registry.

  2. Create a namespace for the application.

    kubectl create namespace demo --kubeconfig=USER_CLUSTER_KUBECONFIG
    

    Replace USER_CLUSTER_KUBECONFIG with the path to the user cluster Kubeconfig file.

  3. Deploy the sample to the cluster:

    kubectl apply -n demo -f online-boutique/release/ --kubeconfig=USER_CLUSTER_KUBECONFIG
    

Online Boutique with Insufficient Resources

Clean up

Delete your user cluster by running the following command:

kubectl -n cluster-USER_CLUSTER_NAME \
  delete Cluster USER_CLUSTER_NAME --kubeconfig=ADMIN_KUBECONFIG

What's next