How to parse redis log with fluentd

I want to show redis log at gcp logging explorer. To do that I installed fluentd at the gcp instance which has the redis log, by following https://www.fluentd.org. Then I configured td-agent.conf as follows:

<source>
  @type tail
  format none
  path /var/log/redis/*.log
  pos_file /var/log/td-agent/pos/redis-server-log.pos
  read_from_head true
  <parse>
      @type syslog
  </parse>
  tag redis
</source>

<filter **>
  @type add_insert_ids
  insert_id_key uniq_id # Optional.
</filter>

<match **>
  @type google_cloud
</match>

The format of /var/log/redis/redis.log is:

40165:M 28 Jan 2021 16:06:57.699 - 0 clients connected (0 replicas), 1449680 bytes in use
40165:M 28 Jan 2021 16:07:02.720 - 0 clients connected (0 replicas), 1449680 bytes in use
40165:M 28 Jan 2021 16:07:05.592 - Accepted 127.0.0.1:40696

somehow I do not see redis log at gcp logging. I doubt part td-agent.conf should not be syslog. but what parser should I use to parse the redis log?

Solution

Your example redis log does not have a format. It is unstructured text prefixed with a date string.

Check if you have log rotation. If yes, then you probably do not want to use *.log. Instead specify the log filename.

This is what I would use:

<source>
  @type tail

  # Parse the timestamp and collect the entire line as 'message'
  format /^(?<message>(?<time>[^ ]*\s*[^ ]* [^ ]*) .*)$/

  path /var/log/redis/*.log
  pos_file /var/lib/google-fluentd/pos/redis.log.pos
  read_from_head true
  tag redis
</source>