Pull logs from remote server into elasticsearch

Question

The short question is: Is it possible to pull logs (within logfiles) from a remote server and ingest them into the ELK stack.

The long story is the following:

• We have a setup with a DMZ which is publically facing
• We have an intranet environment which hosts alot of internal systems, including the ELK stack
• Due to security regulations we cannot establish connections (on IP level) from the DMZ towards the intranet.
• We can however establish connections from the intranet towards the DMZ

Due to this setup, we cannot follow the normal route of installing a Filebeat on the server where the logs are stored and push the messages towards our logstash installation.

What we would like to do is something that looks somewhat like the following:

• Filebeat or any other process gathers the logs on the server inside the DMZ
• On this server there is some process (Filebeat, logstash, another elasticsearch instance?) which keeps those information in a local store
• This tool (whatever it might be in the end) listens on a port which is available from the intranet
• Another tool inside the intranet connects to the DMZ tool and pulls all gathered logs for further processing.

Our investigations sofar only resulted in solutions which push the log information to either logstash or elasticsearch.

One thing we do not want to do is to use fileshares to make the logfiles available directly from the intranet.

Our question is whether what we have in mind is possible at all and if so, what tools and with which setup we would accomplish this.

Answer 1

You can try the following using Kafka as a message broker

On your DMZ server you will have filebeat collecting logs and sending to a logstash instance, this logstash instace will then output your logs to kafka.

It is a simple pipeline with a beats input, your fitlers and a kafka output, if you don't want to do any enrichments on your data, you can send your logs direct to kafka from filebeat.

Your kafka broker will then listen on a port and wait for any consumer to connect and consumes the messages.

On your intranet you will need a logstash instance with a pipeline using the kafka input, this pipeline will act as a kafka consumer and will pull your messages, you can then use the elasticsearch output to store then in your intranet elasticsearch cluster.

For more information read the kafka documentation, and the documentation for the kafka input and kafka output in the logstash documentation.