Fluentbit Cloudwatch templating with EKS and Fargate

Question

I've an EKS cluster purely on Fargate and I'm trying to setup the logging to Cloudwatch. I've a lot of [OUTPUT] sections that can be unified using some variables. I'd like to unify the logs of each deployment to a single log_stream and separate the log_stream by environment (name_space). Using a couple of variable I'd need just to write a single [OUTPUT] section.

For what I understand the new Fluentbit plugin: cloudwatch_logs doesn't support templating, but the old plugin cloudwatch does.

I've tried to setup a section like in the documentation example:

[OUTPUT]
      Name              cloudwatch
      Match             *container_name*
      region            us-east-1
      log_group_name    /eks/$(kubernetes['namespace_name'])
      log_stream_name   test_stream
      auto_create_group on

This generates a log_group called fluentbit-default that according to the README.md is the fallback name in case the variables are not parsed.

The old plugin cloudwatch is supported (but not mentioned in AWS documentation) because if I replace the variable $(kubernetes['namespace_name']) with any string it works perfectly.

Fluentbit in Fargate manages automatically the INPUT section so I don't really know which variables are sent to the OUTPUT section, I suppose the variable kubernetes is not there or it has a different name or a different array structure.

So my questions are:

• Is there a way to get the list of the variables (or input) that Fargate + Fluentbit are generating?
• Get I solve that in a different way? (I don't want to write more than 30 different OUTPUT one for each service/log_stream_name. It would be also difficult to maintain it)

Thanks!

Answer 1

After few days of tests, I've realised that you need to enable the kubernetes filter to receive the kubernetes variables to the cloudwatch plugin. This is the result, and now I can generate log_group depending on the environment label and log_stream depending of the container-namespace names.

filters.conf: |
    [FILTER]
        Name kubernetes
        Match *
        Merge_Log Off
        Buffer_Size 0
        Kube_Meta_Cache_TTL 300s
  output.conf: |
    [OUTPUT]
        Name cloudwatch
        Match *
        region eu-west-2
        log_group_name /aws/eks/cluster/$(kubernetes['labels']['app.environment'])
        log_stream_name $(kubernetes['namespace_name'])-$(kubernetes['container_name'])
        default_log_group_name /aws/eks/cluster/others
        auto_create_group true
        log_key log

Please note that the app.environment is not a "standard" value, I've added it to all my deployments. The default_log_group_name is necessary in case that value is not present.

Please note also that if you use log_retention_days and new_log_group_tags the system is not going to work. To be honest log_retention_days it never worked for me also using the new cloudwatch_logs plugin either.