Search code examples
fluent-bit

how to enrich logs with cluster name fluentbit


I have been using fluentbit across multiple kubernetes clusters and sending data to a centralized elasticsearch cluster. Logs have a lot of metadata on them that helps up to pin point the origin of a given log. Its missing the one crucial information about logs metadata, cluster_name.

Is there a way to enrich logs with a cluster name using fluentbit?

I can have a filter which adds a custom static value like this,

custom-filter.conf: |
    [FILTER]
        Name                modify
        Match               *
        Add                 cluster_name robust_cluster_1

but this approach seems error prone.

I am looking for an approach where cluster name is dynamically fetched by fluentbit.


Solution

  • You can interpolate environment variables, as shown in the Record Modifier example:

    [FILTER]
        Name    record_modifier
        Match *
        Record hostname ${HOSTNAME}
    

    Usage with the modify filter and cluster name would be pretty much the same

    [FILTER]
        Name modify
        Match *
        Add cluster_name ${CLUSTER_NAME}