Search code examples
elasticsearchlogstash

Need assistance with Elasticsearch Logstash filter


I want to enrich hostname from another index, as in some cases that value is missing in my main index. Hence, I use Elasticsearch Logstash filter to query the hostname {as shown in the attachment}.

However, when I test the pipeline using command {manually}, I got error as mentioned below -

[ERROR] 2023-04-24 10:02:58.784 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] >, :backtrace=>["/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/base.rb:341:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/http/manticore.rb:91:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:197:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-7.17.1/lib/elasticsearch.rb:41:in `method_missing'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-api-7.17.1/lib/elasticsearch/api/actions/ping.rb:38:in `ping'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:330:in `test_connection!'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:118:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/data/logstash/pipelines/sendEmailAlerts_updated.conf"], :thread=>"#<Thread:0x1d8d4a8@/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:130 run>"}
[INFO ] 2023-04-24 10:02:58.785 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2023-04-24 10:02:58.793 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

Note: Other pipelines are working fine without any error ; hence looks like Logstash config file are okay.

logstash conf file

input {
        elasticsearch
        {
        hosts => "localhost:9200"
        user => "reader"
        password => "*******************"
        index => "*-testalert"
        query => '{ "query": {
                             "bool": {
                                      "must": [{"terms": { "kibana.alert.severity": [ "high", "critical"] }} ],
                                      "filter": [ {"range": {"@timestamp": { "gte": "now-2d"}}}]
                                     }
                             }
                  }'
        schedule => "/5 * * * *"
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
        codec => "json"
        }
    }



filter {
if [host][hostname] != ""
{
mutate {
        add_field => {
            "alertHostName" => "%{[host][hostname]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }
}
if [host][hostname] == ""
{
elasticsearch {
              hosts => "localhost:9200"
              index => ".fleet-agents"
              query => "{[local_metadata][host][id]}:%{[host][id]}"
              fields => {
                       "[local_metadata][host][id]" => "host_name"
                      }
              }

mutate {
        add_field => {
            "alertHostName" => "%{[host_name]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }
}
}

output {
stdout {
 codec => "json"
}
}


enter image description here


Solution

  • On the first log line you can see this error:

    Unauthorized: [401]

    So it looks like you're just missing some authentication in your elasticsearch filter which queries the .fleet-agents index.

    You probably need to add user => "reader" and the appropriate password the same as you have in the input.