Search code examples
fluent-bitazure-log-analytics-workspace

FluentBit into Azure Log Analytics Workspace memory buffer

I have a client application that runs in AKS with a pod that forwards the logs to Log Analytics Workspace. The software is from a third party that they purchased and we are spitting out Log4J2 logs using the JSON formatter (a custom one, not the default, but same basic principals).

The problem is I keep getting this error:

[ warn] [input] tail.0 paused (mem buf overlimit)

I tried a few different configurations, but went back to the drawing board (I tried switching to storage.type filesystem, etc.). The logs are being persisted in a Storage Account File Share and being accessed by the FluentBit pod via an SMB mount.

I settled into this (putting multiple files into one):

[SERVICE]
  Flush        1
  Log_Level    warn
  Daemon       off
  Parsers_File parsers.conf

  @INCLUDE inputs-cots-app.conf
  @INCLUDE output-law.conf

[PARSER]
  Name   cots-json
  Format json

[INPUT]
  Name             tail
  Tag              cots.json
  Path             /opt/logMount/*podType*/rolling.json
  Exclude_Path     /opt/logMount/*podType*/rolling.log
  Path_Key         filename
  Read_From_Head   True
  Mem_Buf_Limit    128MB
  Buffer_Max_Size  64MB
  Skip_Long_Lines  On
  Skip_Empty_Lines On
  Refresh_Interval 60
  Inotify_Watcher  False
  Parser           cots-json

[OUTPUT]
  name        azure
  match       cots.json
  Customer_ID ${LAW_WORKSPACE_ID}
  Shared_Key  ${LAW_ACCESS_KEY}
  Log_Type    cotsLogsJSON

Are there any tips on troubleshooting this? I can't seem to nail down just how best to resolve this issue.

Solution

  • What load are you expecting to handle with such a configuration? Perhaps your mem_buf_limitis just too small for such a load. See https://docs.fluentbit.io/manual/administration/backpressure