In my logs, some messages should appear in the same order. But for some reason, the order is swapped in failure case. Is it somehow possible in Kibana to set up monitoring or dashboard for swapped messages? E.g. when an Alarm-Clear event arrives before an Alarm-Set event.
Between Set and Clear, there are usually less than two seconds. So, a solution would also be to wait two seconds after the Set event for the Clear event.
More specific, there are messages send over a message bus and received by ELK. The sending system sends the messages in correct order but looking at Kibana Discover, the order is swapped. This I want to detect in Kibana.
Some example from the sending system:
2023-05-03 07:40:09,535 [Queue.Package:28] INFO Send=<<AlarmID>135</AlarmID><AlarmText>AlarmSet</AlarmText>>
2023-05-03 07:40:09,535 [Queue.Package:28] INFO Send=<<AlarmID>135</AlarmID><AlarmText>AlarmClear</AlarmText>>
In ELK Discover, AlarmClear before AlarmSet visible.
There are a couple of solutions for that. the first solution will help you to solve the problem. The second solution will help you to detect the problem.
Use the original timestamp for the data before queueing and/or indexing data into elasticsearch. For logstash, you can use the date filter plugin.
Use transform and aggregate the data according to AlarmID. Transform API will create a new index and you can calculate the diff between AlarmSet - AlarmClear.