Logstash(6.5.4) elasticsearch output

Question

My filebeat is on two servers. let's say app1 and app2

My logs are in the format

jobID status data

Example: app1 log

5hgsxyt3838 SCHEDULED data

app2 log

5hgsxyt3838 COMPLETE data

Here both these logs have the same jobID. I am using this jobID as the document _id on elasticsearch

elasticsearch {
            hosts => [ "localhost:9200" ]
            index => "import-export-logger-%{index-name}"
            document_id => "%{jobID}"
        }

Now I want my final status in elasticsearch as COMPLETE but sometimes there is a high load on app1 so app2 logs are processed first then app1. So the final status becomes SCHEDULED

Is there a way to prevent this from happening i.e I want my document to be updated only when the status is not COMPLETE? When the status is COMPLETE document must not update itself

Answer 1

I'm not a huge fan of overwriting events like this (It would be better to just record all the events and then filter in your queries).

But if you want to go that path anyways you can do this:

output {
    if [status] == "COMPLETE" {
        elasticsearch {
            hosts => [ "localhost:9200" ]
            index => "import-export-logger-%{index-name}"
            document_id => "%{jobID}"
            action => "update"
            doc_as_upsert => true
        }
    }
    else {
        elasticsearch {
            hosts => [ "localhost:9200" ]
            index => "import-export-logger-%{index-name}"
            document_id => "%{jobID}"
            action => "create"
        }
    }
}

So if status is scheduled will create the document if doesnt already exists, complete will update or create.