Dynamic index for opensearch output in fluent bit

Question

I want to generate a index name based on information that is within the log. I have the following fluent-bit.conf

[SERVICE]
    log_level   debug
    Parsers_File /fluent-bit/etc/parsers.conf

[INPUT]
    Name tail
    Tag haproxy

    Path /var/log/haproxy/*.log

    Parser json

[OUTPUT]

    Name    opensearch
    Match   haproxy

    Host ...
    Port ...
    Path ...

    HTTP_User ...
    HTTP_Passwd ...

    #Index my-index-$community-%Y%m
    
    # Create index in the style my-index-$comunity-%Y%m
    Logstash_Format On
    Logstash_Prefix my-index-$community
    Logstash_Prefix_Separator -
    Logstash_DateFormat %Y%m

    tls on
    tls.ca_file ...

    Retry_Limit False

    # With this _doc type is removed
    Suppress_Type_Name On

    # Enable 8 threads for this output
    Workers 4
    
[OUTPUT]
    Match       haproxy
    Name        stdout

Within my json logs, each one is something like this:

{
   "community": "a",
   "some_other_info": 1,
   "some_other_info2": 2,
    ....
    ....
}

For this example, this log should be pushed to index my-index-a-202408, but instead is pushed to my-index-$community-202408. Is there a way to do this? Thanks in advance.

I've tried using Logstash to retrieve data from the record accessor but my index comes like my-index-$community-202408 instead of my-index-a-202408.

Answer 1

Two things:

• Apparently fluentbit opensearch output plugin,CURRENTLY, doesn't support the creation of an index based on information extracted from the record plus some static prefix or suffix such as what I was trying my-index-$comunnity.
• Then, in my code I was using the wrong key to extract data from the record. The correct one is Logstash_Prefix_Key instead of Logstash_Prefix. This key will allow to extract information from the record but you can't appended anything else.

I fixed my problem by adding the static and dynamic parts of the index name in the source of the record generation, in my case HAProxy, and then I used the Logstash_Prefix_Key to extract that value and create the index.