Why fluentd-cloudwatch Kube deamonSet cannot send logs from specified dir to cloudwatch?

Question

I applied a fluentd-cloudwatch DeamonSet in EKS using config file as this: fluentd.yml

Situation ONE: with line 302 - line 313 commented in fluentd.yml, everything works well. I can see the new created log group and log streams in CLOUDWATCH PANEL.

Situation TWO: with line 302 - line 313 uncommented in fluentd.yml, fluentd cannot send any log data into CLOUDWATCH. log group and log streams are not created automatically.

I check the file /logs/fun-practice-log/server.log.0 exists in fluentd-cloudwatch pod. The content in server.log.0 as below:

unknown - 2019-11-21 00:42:44,588 - WARNING - 404 GET /latest/dynamic/instance-identity/document (10.10.130.37) 0.38ms
unknown - 2019-11-21 00:42:44,818 - WARNING - 405 CONNECT 34.116.211.122:80 (10.10.130.217) 0.51ms
unknown - 2019-11-21 00:42:45,752 - WARNING - 405 CONNECT 34.116.211.122:80 (10.10.130.217) 0.64ms
f6c5a62f536a4444bbb12c0e45973141 - 2019-11-21 02:17:19,605 - INFO - 200 POST /fun_practice/v1.1/get_stat?user_token=seYBtp39Uko0lfsYoDBrQlZV8SWuF (10.0.15.188) 895.95ms

I want to stream the logs in folder /logs/fun-practice-log/ to CLOUDWATCH using source as below, I cannot figure out what I miss in config file.

    <source>
      @type tail
      @id in_tail_svc
      @label @hostlogs
      path /logs/fun-practice-log/server.log.0
      pos_file /logs/fun-practice-log/server.log.pos
      tag host.svc
      read_from_head true
      refresh_interval 5
      <parse>
        @type regexp
        expression /(?<request_id>[^ ]*) - (?<timestamp>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}) - (?<log_type>[^ ]*) - (?<message>.*)/
        time_key timestamp
        time_format %Y-%m-%d %H:%M:%S,%z
      </parse>
    </source>

Answer 1

I finally figure out what I did wrong in fluentd.yml. I should not set mountPath: /logs/account-svc-log and mountPath: /logs/fun-practice-log to readOnly: true. Just remove the args, everything works fine.