I am working on passing nginx access logs to Fluentd to aws kinesis to AWS S3 via kinesis firehose. nginx logs will be pushed to AWS Glacier during log rotation.
I am at the initial steps where I need need to pass nginx access logs to fluentd via syslog/rsyslog.
Nginx version used is 1.14.
My OS is :
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
There are many ways to push nginx logs to rsyslog.
Way 1 : Using module imfile , and updating rsyslog.conf as shown in https://petermolnar.net/log-nginx-to-rsyslog/
Way 2 : Updating nginx.conf to write directly in syslog. I am worried this will slow nginx down during high traffix. https://docs.nginx.com/nginx/admin-guide/monitoring/logging/#logging-to-syslog
Way 3: Skip rsyslog/syslog and directly pass data from nginx to fluentd via module https://github.com/fluent/nginx-fluentd-module . Here i am worried that this module is reliable or no and also if nginx should not become less responsive due to this module being used.
What is the best way to achieve the goal , the goal is for realtime data flow from nginx access logs to fluentd , without blocking nginx to serve requests.
Since there are no replies for over a month, Sharing the way that I chose.
Using module imfile , and updating rsyslog.conf as shown in https://petermolnar.net/log-nginx-to-rsyslog/