Exporting metrics

This page is for infrastructure operators.

This page describes how to export metrics from an in-cluster Prometheus monitoring system to external logging and monitoring tools.

Prerequisites

The admin cluster must have at least one spare IP address from the address pool of the load balancer.

Configuring the Prometheus federation

Anthos private mode metrics can be exported to external systems through Prometheus federation. These instructions assume that the external monitoring and logging system is a Prometheus-based system.

  1. To allow access for federation to Prometheus from external systems, apply the following config to the admin cluster:

    cat <<EOF | kubectl apply --kubeconfig=ADMIN_OIDC_KUBECONFIG -f -
    apiVersion: v1
    kind: Service
    metadata:
      # the name of the service must be anthos-prometheus-k8s-external
      name: anthos-prometheus-k8s-external
      namespace: kube-system
    spec:
      ports:
      - name: http
        port: 9090
        protocol: TCP
        targetPort: 9090
      selector:
        app: anthos-prometheus-k8s
      sessionAffinity: ClientIP
      type: LoadBalancer
    EOF
    
  2. Add the following configs to the scrape_configs section of the Prometheus configuration file in the external system:

    scrape_configs:
      - job_name: 'federate'
        scrape_interval: 1m
    
        honor_labels: true
        metrics_path: '/federate'
    
        params:
          'match[]':
            - '{job!=""}'
    
        static_configs:
          - targets:
            - '<loadbalancer-ip-of-anthos-prometheus-k8s-external>:9090'
    
  3. To get the load balancer IP of Prometheus, run the following command:

    kubectl get --kubeconfig=ADMIN_OIDC_KUBECONFIG -n kube-system svc anthos-prometheus-k8s-external -ojsonpath={.status.loadBalancer.ingress[*].ip}
    

What's next

Learn more about logging and monitoring.