My lua script is returning record["key"] as nil

Question

I am trying to fliter/mask logs by using fluent bit to change logs using lua. The filter contains a simple substitution of digits to # using gsub. I am entering logs in JSON format bit lua is not able to access key value.

test.lua

        function modifyRecord(tag, timestamp, record)
        local new_record = record
        old_message = record["message"]
        digits = "[%d]+"
        new_message = old_message:gsub(digits, "#")
        new_record["message"] = new_message
        return 1, timestamp, new_record
        end

Log file – test.log

 echo '{"Test": "Logs","message":"This is a message 123432"}' >> test.log

fluent-bit.conf

    [SERVICE]
        # This is the main configuration block for fluent bit.
        # Ensure the follow line exists somewhere in the SERVICE block
        Plugins_File plugins.conf
    
    [INPUT]
        Name tail
        #Path /export/home/dummy/newrelic-infra/var/log/newrelic-infra/newrelic-infra.log
        Path /export/home/dummy/nrfb/test.log
   
    [OUTPUT]
        Name newrelic
        Match *
        licenseKey <key>
        proxy <proxy>
        endpoint <endpoint>
    
    [FILTER]
        Name modify
        Match *
        Add hostname <hostname>
        Add service_name newrelic-infra-fluent-bit
    
    [FILTER]
        Name    lua
        Match   *
        script  /export/home/dummy/nrfb/test.lua
        call    modifyRecord

Fluent-bit logs

        -bash-4.2$ ./fluent-bit -c ./fluent-bit.conf
        Fluent Bit v1.6.3
        * Copyright (C) 2019-2020 The Fluent Bit Authors
        * Copyright (C) 2015-2018 Treasure Data
        * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
        * https://fluentbit.io
        
        [2022/04/15 11:09:20] [ info] [engine] started (pid=11454)
        [2022/04/15 11:09:20] [ info] [storage] version=1.0.6, initializing...
        [2022/04/15 11:09:20] [ info] [storage] in-memory
        [2022/04/15 11:09:20] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
        [2022/04/15 11:09:20] [ info] [sp] stream processor started
        [2022/04/15 11:09:20] [ info] [input:tail:tail.0] inotify_fs_add(): inode=194 watch_fd=1 name=/export/home/dummy/nrfb/test.log
        [2022/04/15 11:09:49] [error] [filter:lua:lua.1] error code 2: /export/home/dummy/nrfb/test.lua:5: attempt to index global 'old_message' (a nil value)

Answer 1

I had mistake in config file

[INPUT]
    Name tail
    #Path /export/home/dummy/newrelic-infra/var/log/newrelic-infra/newrelic-infra.log
    Path /export/home/dummy/nrfb/test.log

It should have parser set to look/parse for json. Changes in input and addition of parser did the trick.

[INPUT]
    Name tail
    #Path /export/home/dummy/newrelic-infra/var/log/newrelic-infra/newrelic-infra.log
    Path /export/home/dummy/nrfb/test.log
    Parser docker
[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S %z