Search code examples
logstashelastic-stacklogstash-groklogstash-configuration

Logstash 6.2.4 crashes when adding an ID to plugin (Expected one of #)

I am trying to add ID field to my Logstash 6.2.4 config. What i want to do is to debug "http://localhost:9600/_node/stats/pipelines" so i need some names (id is random UUIDs without id field in configs). I found documentation about plugin id. It works for me like this:

input {
    http {
        port => "${HTTP_PORT_FOR_EVENTS:8089}"
        additional_codecs => {"application/json"=>"json"}
        id => "http_events"
        tags => [ "test" ]
    }
}

filter {
    if "test" in [tags]  {
        mutate {
            remove_field => [ "headers", "host" ]   
        }
    }
}

But it crashes like this:

input {
    http {
        port => "${HTTP_PORT_FOR_EVENTS:808}"
        additional_codecs => {"application/json"=>"json"}
        id => "http_events"
        tags => [ "test" ]
    }
}

filter {
    id => "test2"
    if "test" in [tags]  {
        mutate {
            remove_field => [ "headers", "host" ]   
        }
    }
}

With this error (i guess, there is two erros with shutdown between because of Docker container restart or something):

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 32, column 8 (byte 676) after filter {\r\n id ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

SIGTERM received. Shutting down. Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} Ignoring the 'pipelines.yml' file because modules or command line options are specified Starting Logstash {"logstash.version"=>"6.2.4"} Successfully started Logstash API endpoint {:port=>9600} SIGTERM received. Shutting down.

Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} Ignoring the 'pipelines.yml' file because modules or command line options are specified Starting Logstash {"logstash.version"=>"6.2.4"} Successfully started Logstash API endpoint {:port=>9600}

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 24, column 8 (byte 534) after filter {\n id ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

Those configs are also break my logstash:

input {
    rabbitmq {
        id => "test1"
        type => "event"
        exchange => "event"
        exclusive => true
    }
}


input {
    id => "test1"
    rabbitmq {
        type => "event"
        exchange => "event"
        exclusive => true
    }
}
Solution

  • I found a solution.

    ID must be placed in filter plugin field (such as grok, mutate or json), not just in filter field:

    filter {
    if "test" in [tags]  {
        mutate {
            id => "test2"
            remove_field => [ "headers", "host" ]   
        }
    }
}

    Also not every version of plugins support IP field. Update plugins if necessary:

    logstash-plugin update logstash-filter-mutate