I apologize in advanced as I am fairly new to using stack overflow. Please let me know if I need to change anything with my question.
I am currently trying to set up a log aggregator for my work. My plan is to use the elastic stack and filebeat to gather the logs and display them. I used this tutorial to setup the initial server:
I was able to read the syslogs from the system using the filebeat system module just like in the tutorial. I then disabled it so that I could try to log the syslogs from a juniper switch without the interference of having to read through all the server syslogs.
Right now I am just trying to get the logs to come in from a juniper switch to my virtual machine. I have the switch set up to send syslogs to the virtual machine's ip on port 9513. I can see the logs coming in when I use tcpdump to monitor the port.
user@logserver:~$ sudo tcpdump -i ens160 port 9513
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
16:42:20.558768 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 124
16:45:00.571980 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 79
16:45:00.576370 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
16:47:45.944277 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 124
16:48:20.568840 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 64
16:50:00.653082 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
16:50:55.769884 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 106
16:50:55.777058 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.notice, length: 112
16:50:57.704914 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 101
16:50:57.817315 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.notice, length: 112
16:55:00.701165 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
17:00:00.766060 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
17:00:00.777780 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 79
I know there are few different ways I can use filebeat to read this input data. One is using the default juniper module, which I have enabled. I could also setup a syslog type input in the filebeat.yml config file. I have tried both of these methods using both port 9513 and port 514.
filebeat.inputs:
- type: syslog
protocol.udp:
host: "localhost:514"
- type: syslog
protocol.udp:
host: "localhost:9513"
- module: juniper
junos:
enabled: true
# Set which input to use between udp (default), tcp or file.
var.input: udp
var.syslog_host: localhost
var.syslog_port: 514
- module: juniper
junos:
enabled: true
# Set which input to use between udp (default), tcp or file.
var.input: udp
var.syslog_host: localhost
var.syslog_port: 9513
I am guessing that there is something in the filebeat configuration that I am not understanding, since filebeat isn't sending any of the syslog info to logstash. I also used tcpdump on port 5044 and the lo interface, the port that the tutorial uses to setup the output of filebeat and the input of logstash. When I run it and receive a new log from the switch on either port 514 or 9513 nothing gets sent through 5044 as shown here:
user@logserver:~$ sudo tcpdump -i lo port 5044
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
Any help understanding filebeat would be much appreciated. Thank you in advanced
I am not sure what I did exactly, but it appears to be working now. I deleted the juniper configuration of "set system syslog host ..." and redid it without the port assignment. This meant all my syslog messages were now going to the 514 port, so I checked that port on the server. It was receiving all the logs, so I then reset the logstash and filebeat config files to the default settings and disabled all modules on filebeat. I disabled and stopped both filebeat and logstash, re-enabled, and restarted. After that I re-enabled the system and juniper modules in filebeat. This somehow worked and I can now see the syslogs coming in to elasticsearch from the juniper switch.