I'm running a Docker container on a GCE instance under the Container-optimizes OS. I've followed this manual https://cloud.google.com/container-optimized-os/docs/how-to/logging and added the google-logging-enabled metadata value to enable sending containers' logs to Cloud Logging:
But it doesn't work. I can't see containers' logs in the Cloud Logging:
While I have a lot of container logs in reality:
How to make it working and really send containers' logs to Cloud Logging?
Update
The log filter is the default one: (resource.type="gce_instance" AND resource.labels.instance_id="***") OR (resource.type="global" AND jsonPayload.instance.id="***") that means that all types of logs from the current instance should be shown.
My issue was that my service account associated with the instance had no the Logs Writer permission. I found that by running the sudo systemctl status stackdriver-logging command which shown the following:
● stackdriver-logging.service - Fluentd container for Stackdriver Logging
Loaded: loaded (/usr/lib/systemd/system/stackdriver-logging.service; static; vendor preset: disabled)
Active: active (running) since Fri 2021-03-05 01:18:55 UTC; 1h 12min ago
Main PID: 385 (docker)
Tasks: 9 (limit: 2382)
Memory: 31.9M
CPU: 751ms
CGroup: /system.slice/stackdriver-logging.service
└─385 /usr/bin/docker run --rm --name=stackdriver-logging-agent -v /etc/stackdriver/logging.config.d/:/etc/google-fluentd/config.d/ -v /var/log:/var/log -v /var/lib/docker/containers/:/v
Mar 05 02:30:11 jwp-gitlab-runner-m3l0 docker[385]: 2021-03-05 02:30:11 +0000 [warn]: #0 Dropping 1 log message(s) error="User unauthorized to access *** for resource ***
By googling the error message I found the answer in the GitHub issue https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/issues/295
The following answer also helped me to diagnose the problem: https://stackoverflow.com/a/65768214/8016720
Now I have all the logs I want in Cloud Logging.