I am new to elk stack. Let me explain what i am trying to do. I have a application that is running separately for different users i.e. 5 different users will have 5 independent instance of the same application. I am using filebeats to send the logs from the application to logstash where it will be processed first before being sent to the elasticsearch. What i want is to write the application which enables the users to view the logs of theirs instance of application only. Now what i tried to do is creating the logstash pipeline for each the user with different port which will process the log and send it to elasticsearch with the different index name. Can you suggest me if this is the best practice or i am doing it wrong? Is there a more better way to do it without having separate pipeline for individual users with separate port? I think the way I am doing it is wrong and it will be harder for me to manage when the instances will grow in numbers.
Thank You
I would suggest if there's no skinning , validation and enrichment involved then skip logstash altogether. You can straight away pass filebeat logs to ES. Now there are two ways from here. Filebeat can additionally send a parameter (any fixed string) along with the scanned message to ES or you can store the meta (like ip) source which filebeat will send along with message. This string can then be used to identify the source of the log message and then on kibana you can configure to show dashboard based on that fixed string / user / meta. This simply the process and avoid unnecessary hops.