Anthos private mode error codes

Anthos private mode error messages consist of an error ID in the format APME1234 where 1234 is a unique number, followed by a description of the problem and a suggestion for how to fix it. This document lists each of those error messages.

APME1000 ImageAccessError

This error indicates that the component which produced the error cannot find one or more of the required images in the specified registry.

Please verify that:

  • The required images exist in the container registry.
  • The registry is reachable from the origin of the error.
  • The workstation has permission to READ and PULL from the registry.
  • The workstation has the file ${HOME}/.docker/config.json and it is correctly configured to access the registry.

APME1001 InvalidRegistryInputError

The --private-registry flag value needs to conform to the format: hostname[:port]/repository/location and cannot begin with a scheme (such as https://). The hostname cannot be localhost or 127.0.0.1. The value is used by Kubernetes to fetch images.

Example flag values:

  • fictional.registry.example/repository/location
  • fictional.registry.example:10443/repository/location
  • 10.200.0.2/library

APME2000 KubeContextNotFoundError

This error indicates that the specified kubecontext cannot be found.

Please verify if:

  • The KUBECONFIG file exists.
  • The $HOME/.kube/config file exists when the KUBECONFIG environment variable is not specified.
  • The context exists in the KUBECONFIG file.

APME2001 UserClusterKubeconfigError

This error indicates that the kubeconfig for the user cluster cannot be found.

Please verify if:

  • The namespace cluster-CLUSTER_NAME exists in the admin cluster.
  • The cluster objects exist under the namespace cluster-CLUSTER_NAME.
  • The secret CLUSTER_NAME-kubeconfig exists under the namespace cluster-CLUSTER_NAME.
  • You have all the necessary permissions to access the objects mentioned above.

APME2002 ManagementCenterNotSchedulableError

This error indicates that the management center cannot be scheduled to run on the nodes of a cluster.

This error usually can happen if the admin cluster doesn't have any worker nodes, and by default your cluster will not schedule workloads on the control-plane node for security reasons.

To fix this issue:

  1. [Preferred] Add a worker NodePool to the admin cluster.

    KUBECONFIG=${ADMIN_KUBECONFIG} kubectl apply -f <path/to/example-nodepool.yaml>
    

    Example YAML:

    apiVersion: baremetal.cluster.gke.io/v1
    kind: NodePool
    metadata:
      name: node-pool-1
      namespace: cluster-admin
    spec:
      clusterName: admin
      nodes:
      - address: <IP address of the worker node machine. e.g. 10.200.0.4>
    
  2. Remove the control plane taint from the control plane nodes, which will allow non-system workloads to run on the control plane nodes.

    NOTE: Consider the downsides of this solution:

    • Control plane availability: Running additional workloads increases entropy on control plane nodes. If the machine is already resource constrained, misconfigured, or the workload has unexpected behavior, it may have a negative impact on the Kubernetes API server.

    • Security: Containers are not a strong security boundary. If a malicious workload breaks out of the container, it may gain control of the cluster control plane. If the cluster is managing other user clusters, the attacker may access the credentials in the admin cluster and gain control of user clusters too.

    • Cost: Anthos private mode does not charge for control plane nodes as long as the node-role.kubernetes.io/master:NoSchedule taint is present. After you remove the taint and enable user workloads to run on control plane nodes, they are charged like any other nodes. See the Pricing page for detailed pricing information.

    Example command to allow workloads to run on all the control plane nodes:

    KUBECONFIG=${ADMIN_KUBECONFIG} kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-
    

APME2003 AdminClusterNotUpgradedError

This error indicates that the admin cluster must be upgraded before upgrading the Anthos private mode management center.

The management center in the new Anthos private mode release might require running on the new version of the admin cluster.

Update the admin cluster config file and run the actl clusters baremetal upgrade admin command to complete the upgrade. See Upgrading the admin cluster for detailed instructions.

APME3000 AISLoginClientValidationError

In order for users to access the clusters using their federated credentials (OIDC authentication), ADMIN_OIDC_KUBECONFIG needs to be generated by admins.

ADMIN_ACTL_AUTH_LOGIN_CONFIG=admin-actl-auth-login-config.yaml
ADMIN_OIDC_KUBECONFIG=oidc-kubeconfig
actl auth login --login-config=${ADMIN_ACTL_AUTH_LOGIN_CONFIG} --cluster=admin --kubeconfig=${ADMIN_OIDC_KUBECONFIG}

${KUBECTL_ANTHOS_CONFIG} is an input config file downloaded from the Management Center, whereas ${ADMIN_OIDC_KUBECONFIG} is the output kubeconfig file location. Do not share the output config file even with employees or admins.

In order to proceed, check your input values:

  • --login-config and --kubeconfig flag values are not empty.
  • --login-config and --kubeconfig flag values are not the same.
  • --login-config and --kubeconfig flag values have valid file paths.

Use the --skip-validation flag to skip the validation.

APME3001 PermissionDeniedError

The error indicates insufficient permissions to perform the action.

To troubleshoot further:

  • Identify the role that is assigned.

    • Method 1: Using the Management Center
      • Open the Identity and Access tab in the Management Center.
      • Go to the Access tab within the Identity and Access screen to view the role that is assigned.
    • Method 2: Using the kubectl command:
    kubectl get
        rolebinding,clusterrolebinding --all-namespaces -o jsonpath='{range
        .items[?(@.subjects[0].name=="{your
        account}")]}[{.roleRef.kind},{.roleRef.name}]{end}'
    
    Example: kubectl get rolebinding,clusterrolebinding --all-namespaces -o
    jsonpath='{range
    .items[?(@.subjects[0].name=="foo@bar.com")]}[{.roleRef.kind},{.roleRef.name}]{end}'
    [ClusterRole,anthos-platform-admin-read-only]
    
  • Refer to the preset authorization roles and the permissions to view the permissions available to the preset authorization roles.

  • If a lower privilege role is assigned and a higher privilege role is necessary to perform a certain action or if the role was incorrectly assigned, request the administrators to check if the access can be granted.

  • If the role has sufficient permissions for the action as per preset authorization roles and the permissions and APME3001-PermissionDeniedError is incorrectly seen, report a bug at anthos-private-mode-feedback@google.com.

APME3002 AuthMethodPreflightCheckError

The error indicates that the provided auth method setting cannot pass preflight check.

If you provided an OIDC setting:

  • Double check if the URL to the OIDC provider is correct.
  • Make sure that the OpenID config discovery page https://<your OIDC provider URL>/.well-known/openid-configuration is accessible and contains a valid config.
  • If your OIDC provider uses a self-signed SSL certificate, make sure to enter it into the OIDC provider certificate field.

APME3003 UserCredentialNotFoundError

This error indicates that the requested action cannot be performed without a user credential.

In order to perform the request, enable authentication or ask your administrator to enable authentication. Refer to Authenticating with OIDC to set up OIDC authentication.

What's next