Search code examples
google-kubernetes-engine

GKE not capturing any application logs

New GKE cluster has Cloud Operations for GKE set to System and workload logging and monitoring, however no application logs are showing up. When I inspect the installed fluentbit-gke in kube-system I'm noticing same error over and over again.

1server.go:47] Configured Fluent Bit GKE plugin server to run on the port 2021
server.go:139] Failed to process request: saw 1 errors; last: rpc error: code = PermissionDenied desc = The caller does not have permission
server.go:139] Failed to process request: saw 1 errors; last: rpc error: code = PermissionDenied desc = The caller does not have permission
server.go:139] Failed to process request: saw 1 errors; last: rpc error: code = PermissionDenied desc = The caller does not have permission
server.go:139] Failed to process request: saw 1 errors; last: rpc error: code = PermissionDenied desc = The caller does not have permission
Solution

  • You may want to check that your cluster complies with the following points:

    • its node-pools should have the Stackdriver Logging API access scopes set to "Write Only" or "Full".
    • the service account configured at node-pool level should have the necessary role/permissions to contact the Stackdriver API.