Fluentbit not sending EKS logs to S3

Question

I've tried to send EKS logs to S3, but logs are not being pushed to S3. But, I can able to visualize logs in Elastic search. Below is the output section of my fb-configmap.conf file.

    output-elasticsearch.conf: |
      [OUTPUT]
          Name            es
          Match           *
          Host            ${FLUENT_ELASTICSEARCH_HOST}
          Port            ${FLUENT_ELASTICSEARCH_PORT}
          HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
          HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWD}
          Logstash_Format Off
          Index           dev
          Replace_Dots    On
          Retry_Limit     False
    output-s3.conf: |
      [OUTPUT]
          Name                  s3
          Match                 *
          bucket                eks-logs
          region                ap-south-1
          total_file_size       250M
          s3_key_format         /%Y/%m/%d/%H/%M/%S/$UUID.gz
          s3_key_format_tag_delimiters .-

As per my understanding, AWS access & secret key will be taken from the .aws/credentials path.
And do I need to install any plugin like fluent-bit-s3-plugin for this?

I've been trying to resolve this for days, any help is much appreciated.

Answer 1

Can able to push to S3 after checking the below things,

• output-s3.conf should be given properly
• your S3 bucket should have public access
• should have @INCLUDE output-*.conf in your conf
• fluent-bit image should be latest/stable/updated