AWS EKS Fargate logging to AWS Cloudwatch: log groups are not creating

Question

I used official procedure from AWS and this one to enable logging.

Here is yaml files I've applied:

---
kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  flb_log_cw: "true"
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region us-east-1
        log_group_name fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group true
        log_key log

  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  
  filters.conf: |
     [FILTER]
        Name parser
        Match *
        Key_name log
        Parser crio

Inside the pod I can see that logging was enabled:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    CapacityProvisioned: 2vCPU 4GB
    Logging: LoggingEnabled
    kubectl.kubernetes.io/restartedAt: "2023-01-17T19:31:20+01:00"
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2023-01-17T18:31:28Z"

Logs exists inside the container: kubectl logs dev-768647846c-hbmv7 -n dev-fargate But in AWS CloudWatch log groups are not created, even for fluent-bit itself From the pod cli I can create log groups in AWS Cloudwatch, so the permissions are ok I also tried cloudwatch instead of cloudwatch_logs plugin, but no luck

Answer 1

I've solved my issue. The tricky thing is: IAM policy must be attached to the default pod execution role which created automatically with the namespace and it has no relation to the service account \ custom pod execution role