Why ElasticSearch sometimes doesn't show latest log that Logstash sent?

Question

My Logstash sends log to ElasticSearch, in Logstash output file I see that new logs are sent but they are not shown when click refresh in Kibana Discover page. The Discover page shows logs ~1 hours ago.

What could cause this issue? Is there any way to check if a record is in ElasticSearch rather than using Kibana?

The timestamp in my log file is like this: [2020-09-02 13:53:07,392Z]

Here is the pipeline.yml my logstash uses:

input {
    #stdin {}
    beats {
        port => "5055"
    }
    # file {
}
filter {
    grok {
        patterns_dir => "C:\logstash-7.4.2\patterns"
        match => { "message" => "^\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{LOGLEVEL:level}\]\[%{Thread:thread}\]\[%{JAVACLASS:class}\](\[\d*\])? %{GREEDYDATA:msg}" }
   }
   if "_grokparsefailure" in [tags] {
       drop{}
   }
}
output {
    file {
        path => "C:\logstash-7.4.2\logstash_output.txt"
    }
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "ts_services-%{+YYYY.MM.dd}"
    }
}

I see the latest logs after I deleted the existing index pattern and created a new one. But I don't think this is solution.

Answer 1

To follow the discussion in the comments. The issue here is a very common one.

The index pattern in Kibana was to specific:

ts_services-2020-08*

So data in September where ignored by Kibana, even if they are in Elasticsearch.

Changing the Kibana index pattern will fix the issue:

ts_services*

But you will have to correct all custom visualizations.