Anthos private mode authentication issues

This page lists steps that help you troubleshoot authentication, authorization, and audit logging issues while using Anthos private mode.

Identify OpenID Connect issues

If you configure OpenID Connect (OIDC) but fail to log in to Management Center, the OIDC specification might be misconfigured.

To reduce the risk of a misconfigured infrastructure, the OIDC performs preflight checks for the following:

  • Connectivity to the OIDC provider over HTTPS
  • Access to /.well-known/openid-configuration

Check Anthos Identity Service logs

Anthos Identity Service is a service used to authenticate requests.

  1. Use the kubectl logs command to print Anthos Identity Service logs.

    kubectl --kubeconfig $ADMIN_KUBECONFIG logs deployment/ais --all-containers=true -n anthos-identity-service
    

    Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

  2. Review the logs for errors that help diagnose the OIDC issue.

    If you can't authenticate with OIDC, Anthos Identity Service logs provide the most relevant and useful information for debugging the problem.

    • For example, if the OIDC specification has a wrong certificate for the OIDC provider, logs contain the following entry:

      Failed fetching the Discovery URI "https://accounts.google.com/.well-known/openid-configuration" with error: Failed to load TLS certificates.
      
    • If the client ID or client secret is misconfigured in the OIDC specification, logs contain the following entry:

      Attempt at webflow login failed: Unable to fetch the ID Token: Received unexpected 401 status: {
         "error": "invalid_client",
         "error_description": "Unauthorized"
       }
      
    • If you see certificate verification or trust issues, verify that the entire certificate chain is in the configuration file, and that the certificates are in the correct order.

      The order must be as follows:

      1. identity provider certificate

      2. intermediate CA

      3. root CA

      E0618 12:37:40.186900      15 oidc_cache.cc:232] Failed to fetch the Discovery URI "https://sts.examplecorp.domain/adfs/.well-known/openid-configuration" with error: The HTTPS           server is unavailable or the certificate(s) do not match.
      E0618 12:37:40.186911      15 authentication_plugin.cc:306] OIDC (https://sts.examplecorp.domain/adfs) - Unable to fetch JWKs needed to validate OIDC ID token.
      I0618 12:37:40.186920      15 service.cc:169] Kubernetes webhook adapter received request from 10.10.24.12 and responded with 401.
      

  3. If you have identified a configuration issue in the logs, proceed to Reconfigure OIDC.

Check the OIDC configuration

If a username claim or a group claim does not exist in the requested OIDC scopes, the following error displays in the Management Center Console:

Logins with this OIDC provider appear to be misconfigured, please contact your system administrator.

The OpenID Connect (OIDC) configuration is located in the clientconfig.authentication.gke.io custom resource with the default name and kube-public namespace.

  1. Use kubectl get to print the OIDC configuration for your cluster.

    kubectl --kubeconfig $ADMIN_KUBECONFIG get clientconfig.authentication.gke.io default -n kube-public -o yaml
    

    Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

  2. This file can contain multiple OIDC configurations. Use kubectl get to identify the authentication method used for a specific domain.

    kubectl --kubeconfig $ADMIN_KUBECONFIG get domainconfig.managementcenter.anthos.cloud.google.com -o yaml
    

    Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

  3. Review the field values in clientconfig.authentication.gke.io to confirm that the configuration is correct.

  4. If you have identified a configuration issue in the specification, proceed to Reconfigure OIDC.

Reconfigure OIDC

If you identified problems in the OIDC configuration, you can reconfigure the configuration in your cluster to address them without re-creating the cluster. To reconfigure the OIDC, edit the ClientConfig object with the default name in the kube-public namespace.

KUBECONFIG=$ADMIN_KUBECONFIG kubectl edit ClientConfig -n kube-public

Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

Alternatively, use the following command to reset the authentication configuration and start the OIDC configuration from the initial state:

KUBECONFIG=$ADMIN_KUBECONFIG bin/actl auth reset

Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

Identify authorization issues

If you can sign in after OIDC configuration but still encounter RBAC access denied or Permission denied issues, check your authorization configuration. Anthos Service Mesh along with Kubernetes Role-Based Access Control (RBAC) enforce authorization. The following steps help check each of the configurations.

Check the RequestAuthentication configuration

The RequestAuthentication API defines which authentication methods are accepted. You must define only one RequestAuthentication resource. The resource verifies that the Authorization Bearer tokens are valid and originate from the configured issuers. The resource must have a jwtRule corresponding to the issuerURI in clientconfig.authentication.gke.io.

kubectl --kubeconfig $ADMIN_KUBECONFIG get RequestAuthentication istio-ingressgateway -n istio-system -o yaml

Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

Check the AuthorizationPolicy configuration

Anthos private mode creates authorization resources to grant permissions to subjects bound to preset authorization roles at a fine-grained level. AuthorizationPolicy is a resource that allows or denies access to web endpoints.

  1. Define AuthorizationPolicy as follows:

    kubectl --kubeconfig $ADMIN_KUBECONFIG get AuthorizationPolicy -n istio-system -o yaml
    

    Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

    Anthos private mode includes the following two authorization resources:

    1. istio-ingressgateway-open defines web endpoints that authorizations are not applied to.
    2. istio-ingressgateway-auth defines identity providers, users, and groups that are authorized to access the Management Center Console.
  2. Verify that istio-ingressgateway-auth has a rule for each OIDC provider with correct claims.

    For example, if an email claim is used, the rule would look as follows:

    apiVersion: security.istio.io/v1beta1
    kind: AuthorizationPolicy
    metadata:
      creationTimestamp: "2021-06-03T18:42:38Z"
      generation: 4
      name: istio-ingressgateway-auth
      namespace: istio-system
      resourceVersion: "443786"
      uid: e5eb0e91-a7b3-485c-8ba6-3cf8f4907bbe
    spec:
      rules:
      - from:
        - source:
            requestPrincipals:
            - <issuer-url>/*
        when:
        - key: request.auth.claims[email]
          values:
          - <email>
      selector:
        matchLabels:
          app: istio-ingressgateway
    

Check the EnvoyFilter configuration

EnvoyFilter routes authentication requests to the Application Interface Services web proxy. Verify that the istio-ingressgateway resource exists in the istio-system namespace.

kubectl --kubeconfig $ADMIN_KUBECONFIG get  EnvoyFilter istio-ingressgateway -n istio-system -o yaml

Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

Check the role assigned to the user

List all roles or cluster roles bound to a user and ensure the user is bound to one of the preset roles.

kubectl --kubeconfig $ADMIN_KUBECONFIG get rolebinding,clusterrolebinding --all-namespaces -o jsonpath='{range .items[?(@.subjects[0].name=="oidc-foo-johndoe@example.com")]}[{.roleRef.kind},{.roleRef.name}]{end}'

Replace ADMIN_KUBECONFIG with the path of the kubeconfig file for the admin cluster.

Include the OIDC provider userPrefix in the query. For example, when checking joedoe@example.com from an OIDC provider with userPrefix: oidc-foo- set, you need to put oidc-foo-joedoe@example.com in the query. Similarly, to check a group, append groupPrefix of the OIDC provider first.

Refresh token expiration

The identity provider issues two tokens:

  • an ID token that has a short lifespan
  • a refresh token that typically has a longer lifespan

When your ID token expires, use your refresh token to retrieve a new ID token without reauthentication. When both of your tokens expire, you must reauthenticate. If you frequently see refresh token expired, your identity provider - LDAP or otherwise - might be issuing refresh tokens with a short lifespan.