Search code examples
dockerlogginggoogle-cloud-platformstackdrivergoogle-container-optimized-os

Injest logs as JSON in Container Optimized OS

I am able to injest logs to Google Log Viewer with the help of stackdriver logging agent from Container Optimized OS as JSON.

It injests logs as a value to message, but not as json payload with the default configuration

What I have tried?

I have changed the fluentd config in /etc/stackdriver/logging.config.d/fluentd-lakitu.conf to the following:

<source>
  @type tail
  format json
  path /var/lib/docker/containers/*/*.log
  <parse>
    @type json
  </parse>
  pos_file /var/log/google-fluentd/containers.log.pos
  tag reform_contain
  read_from_head true
</source>

But its unable to send logs to Log viewer

OS: Container Optimized OS cos-81-12871-1196-0

Solution

  • As @Kamelia Y mentioned about the https://issuetracker.google.com/issues/137517429

    There is a mention on workaround used

    <filter cos_containers.**>
@type parser
format json
key_name message
reserve_data false
emit_invalid_record_to_error false
</filter>

    The above snippet parses the logs into JSON and injest to Cloud Logging.

    In this discussion in Google Groups on Stackdriver, we have discussed on how to use it with startup-script.

    Here is the snippet for startup script.

    cp /etc/stackdriver/logging.config.d/fluentd-lakitu.conf /etc/stackdriver/logging.config.d/fluentd-lakitu.conf-save
# Shorter version of the above: cp /etc/stackdriver/logging.config.d/fluentd-lakitu.conf{,-save}
(
head -n -2 /etc/stackdriver/logging.config.d/fluentd-lakitu.conf-save; cat <<EOF
<filter cos_containers.**>
@type parser
format json
key_name message
reserve_data false
emit_invalid_record_to_error false
</filter>
EOF
) > /etc/stackdriver/logging.config.d/fluentd-lakitu.conf
sudo systemctl start stackdriver-logging

    This image can be used to generate random JSON logs. https://hub.docker.com/repository/docker/patelathreya/json-random-logger