How to stop gke-metadata-server from keep generating this log?

Question

I have created a deployment which meant to insert message from pubsub to bigquery with Workload identity enabled, cloud log keep sending this kind of log to me.

{
  "insertId": "test",
  "jsonPayload": {
    "message": "[rpc-id:test] \"/computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\" HTTP/200, started at 2022-06-24 13:40:43.261475517 +0000 UTC m=+39273.838829908",
    "pid": "1"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "gke-metadata-server",
      "pod_name": "gke-metadata-server-45thg",
      "project_id": "test",
      "location": "us-west2-a",
      "cluster_name": "test",
      "namespace_name": "kube-system"
    }
  },
  "timestamp": "2022-06-24T13:40:43.261643773Z",
  "severity": "INFO",
  "labels": {
    "k8s-pod/pod-template-generation": "1",
    "k8s-pod/k8s-app": "gke-metadata-server",
    "k8s-pod/addonmanager_kubernetes_io/mode": "Reconcile",
    "compute.googleapis.com/resource_name": "gke-test-pool-1-77a7892c-l5kl",
    "k8s-pod/controller-revision-hash": "test"
  },
  "logName": "projects/test/logs/stderr",
  "sourceLocation": {
    "file": "metadata.go",
    "line": "142"
  },
  "receiveTimestamp": "2022-06-24T13:40:46.939645996Z"
}

It looks like everytime when I received a message from pubsub or everytime I writed to bigquery,gke-metadata-server sent a request to authenticate the scope.

What should I do to stop the server from keep authenticating or keep generating these logs?

Answer 1

I don't think it's possible to modify that metadata server pod since it will get reconciled. So alternatively you could tell Cloud Logging to not log these by setting up on exclusion filter on the _Default log sink with the following query:

resource.type = ("k8s_container")
resource.labels.container_name = ("gke-metadata-server")

You can use gcloud to configure the exclusion filter in Cloud Logging like this:

gcloud logging sinks update "_Default" \
--add-exclusion=name="ignore-gke-metadata-server",filter="resource.type = ("k8s_container")
resource.labels.container_name = ("gke-metadata-server")"

Note: Thee is a new line after ("k8s_container")

I noticed that for some reason, the gke-metadata-server logs in my environment by default aren't being sent to Cloud Logging without having to setup the exclusion filter. Not sure why though.