does anyone have any experience with entity centric indexing with elasticsearch by using python and groovy scripts for reindex the event centric index into entity centric indexes where every log message has own index or so?
I've got a lot of following messages:
Jul 23 09:24:16 msda msda-core[5147]: 1563866656876839.mt
Jul 23 09:24:18 msda msda-core[5210]: 1563866656876839.0.dn
where I have a lot of the same id numbers with .mt suffix and .dn suffix.
I always need to find the message with the same id number and appropriate dn suffix if a message with .dn suffix appears within one hour.
Any idea would be appreciated!
If you are running v7.2 or later, I would recommend using Elasticsearch data frame transforms for creating an event centric index grouped by the id numbers.. https://www.elastic.co/guide/en/elastic-stack-overview/current/ml-dataframes.html
Use a scripted metric on the min and max timestamp to calculate duration. There is a good example in the Elastic documentation - https://www.elastic.co/guide/en/elastic-stack-overview/7.3/example-clientips.html