Fluentd parser plugin pattern not matched with data

Question

Hi I'm writing a custom plugin for fluentD, I need to use this plugin to parse one field from syslog5424 format, the ruby code works just fine when you run it without fluentd but not when it runs as a plugin

error

#<Fluent::Plugin::Parser::ParserError: pattern not matched with data '[kubernetes@47450 app="api-mma" pod-template-hash="9849d88d6" namespace_name="qa" object_name="api-mma-9849d88d6-59mzd" container_name="api-mma" vm_id="91cdf0e0-b1fb-45bc-8a3f-c85b027af505"]'>

The custom plugin:

require 'fluent/plugin/parser'
require 'json'


module Fluent::Plugin
  class TKGIMetadataParser < Parser
    # Register this parser as 'tkgi_metadata'
    Fluent::Plugin.register_parser('tkgi_metadata', self)

    # `delimiter` is configurable with ' ' as default
    config_param :delimiter, :string, default: ' '

    def configure(conf)
      super

      if @delimiter.length != 1
        raise ConfigError, "delimiter must be a single character. #{@delimiter} is not."
      end
    end

    def parse(text)
        left_braket_pos = text.index('['.freeze)
        right_braket_pos = text.index(']'.freeze)
        data = text.slice(left_braket_pos + 1, right_braket_pos - 1)
        source, key_values = data.split(' ', 2)
        record = {}
        #key_values.gsub!(/[\"]/,'')
        key_values.split(' ').each do |kv|
            k, v = kv.split('=', 2)
            record[k] = v
        end
        record.merge!(source: source)
        yield record.to_json
    end
  end
end

fluentd.conf

<filter k8s>
    @type parser
    key_name syslog5424_sd
    reserve_data true
    reserve_time true
    <parse>
      @type tkgi_metadata
    </parse>
</filter>

sample log:

[kubernetes@47450 app="api-mma" pod-template-hash="9849d88d6" namespace_name="qa" object_name="api-mma-9849d88d6-59mzd" container_name="api-mma" vm_id="91cdf0e0-b1fb-45bc-8a3f-c85b027af505"]

Answer 1

You need to yield time and a record as a hash:

def parse(text)
  # ...
  yield convert_values(parse_time(record), record)
end

Example: https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/parser_ltsv.rb#L46-L47